43 commits

Author	SHA1	Message	Date
武本龍門	43be1c2ae7	fix(router)!: keep RPM, region, and param checks when token counting fails (#44544) ... * fix(router): stop skipping rate-limit checks when token counting fails Router._pre_call_checks() caught any token_counter() exception and returned the full, unfiltered deployment list, bypassing the RPM, region, and invalid-params checks for every deployment - not just the context-window check that actually depends on the count. A content block token_counter() cannot parse (e.g. an invalid image_url detail value) silently disabled rate-limit enforcement for that request, with only a log line and no error surfaced to the caller. Now a counting failure only disables the context-window check; the other checks still run for every deployment as before. Input-token resolution is pulled into its own method and computed once before the loop, so nothing needs to track "did counting fail" with a mutable flag across iterations. * test(router): cover the async region check when token counting fails Replace the unit test of the private input-token resolver with a regression test through async_get_healthy_deployments, the path the proxy uses: an EU-pinned request whose content cannot be token-counted must not get the US deployment * test(router): cover the param filter when token counting fails and satisfy router coverage * fix(router): answer 400 when no deployment takes the requested param, honor allowlists and drop_params The pre-call param filter now reuses the router's param-acceptance helper, so allowed_openai_params, drop_params, and additional_drop_params keep a deployment instead of filtering it out. When every available deployment lacks a requested param, the router answers 400 UnsupportedParamsError right away instead of a retried 429. Deployment context windows are resolved once per call. * fix(router): keep the retried 429 when a cooling-down deployment takes the param * fix(router): steer to a deployment that takes the param before dropping it, keep context-window fallbacks * fix(router): steer by deployment capability, apply drop settings after region and RPM checks A deployment-level allowed_openai_params no longer counts as taking the param when another deployment does, the drop and allow settings now apply only when no deployment that passed the region, RPM, and context-window checks takes it, and an authenticating provider is checked by its declared model instead of always passing * test(router): make the github_copilot pre-call filter test hermetic in CI * test(router): cover pre-call region and response_format filters through the proxy * fix(router): judge the 400 against the request's own deployments, not a name lookup * fix(router): judge the 400 against the passed deployments when the group name lists none * fix(router): let a request's drop and allow settings replace the deployment's in the param check --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 09:20:34 +00:00
Michelangelo Setaro	c288dc7da0	fix(router): let an exact base_model widen a deployment's context window (#46197) ... * fix(router): honor explicit bedrock base model capacity * fix(router): honor an exact base_model's capacity for every non-Azure provider Extends the Bedrock-only check to every non-Azure provider and only uses the base_model when it has its own exact cost-map entry, so an unmapped or wildcard base_model falls back to the route id instead of raising in the router. * test(router): call _maps_to_exact_model_info directly in its cost-map tests * fix(router): let an exact base_model only widen a deployment's context window A base_model used as a pricing stand-in with a smaller window no longer narrows the route's limit, and a route with no window never takes one from base_model * test(router): count the base_model window helpers as covered through get_router_model_info * fix(router): widen only from a model_info base_model naming the same model A stand-in base_model for a different, larger model widened a smaller deployment's window, so pre-call checks sent it prompts the provider rejects with 400. The window now widens only when model_info.base_model names the same provider model id as the route, which is also the field /model/info reads. * fix(router): match a base_model's model id across Bedrock region prefixes and @ qualifiers * fix(router): read litellm_params.base_model too and look the base row up as written * test(router): cover a same-model base_model window on chat, messages, responses, and complexity escalation --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 06:37:40 +00:00
yuneng-jiang	b602209ca6	test(integration): hand every harness server a port from one per-worker block (#46761) ... * test(integration): hand every harness server a port from one per-worker block About 45 places in the integration harness picked a port by binding port 0, reading the number back, closing the socket and starting a server on that number later. Another process could take the port in between, which surfaced as "Address already in use" flakes. Earlier fixes retried around the race in single call sites without removing it. tests/integration/_support/ports.py now owns every port the harness picks ahead of a server. Each pytest process claims a 200-port block between 10000 and 32767 by taking an exclusive flock on a per-block lock file in the system temp directory, so parallel runs on one machine never share a block and the lock goes away with the process. The range is above the fixed service ports and below the Linux and macOS ephemeral ranges. Ports go out in order, are never reused in a run, skip anything an outside process already holds, and a process that exhausts its block claims the next free one. The three _free_port copies, worker_endpoints' port-0 binds, free_loopback_port, the owned proxy's lost-port-race retry and the owned Redis retry loop are gone. Tests that restart a server on the same port now allocate that port up front instead of reading it back from the first server. TQ012 in scripts/check_test_quality.py rejects a port-0 bind read back with getsockname anywhere under tests/integration. * ci(integration): record listening sockets and the ephemeral port range * Revert "ci(integration): record listening sockets and the ephemeral port range" This reverts commit d3fafc3da2.	2026-10-15 16:56:40 -07:00
yuneng-jiang	46913be92b	test(integration): wait on the right signal in five flaky integration contracts (#46737) ... * test(integration): wait on the right signal in five flaky integration contracts The Prometheus shared-label tests waited for the latency metric and then read the request counter, which the logger writes after an await. They now wait for the counter The auto-router session poll returned on the first 200 while the caller expected two turns, so it now waits for the expected turn count The S3 queue cap test let its failing sink recover on a 15s clock that could expire mid-flush. The sink now stays down until a flush fails after every event is queued K3 normally takes 168-175s against a 180s limit, so the limit is now 300s The MCP env var browser spec reused an idle keep-alive socket that the proxy closes after 5s, so its API calls now send Connection: close * test(integration): add until_asserted so a wait polls the exact asserts it guards eventually() takes a predicate separate from the asserts that follow it, which lets a test wait on one value and then assert on another written later. until_asserted reruns a block of asserts until they all pass and re-raises the last failure on timeout The Prometheus shared-label tests and the auto-router session check now use it	2026-10-15 13:48:51 -07:00
devin-ai-integration[bot]	e344dbd8db	feat(proxy): split Azure PTU deployments across teams with per-team PTU ceilings and PTU-hour reporting (#43043) ... * feat(proxy): split Azure PTU deployments across teams by share A PTU deployment's model_info takes ptu_shares (team_id -> whole PTUs adding up to ptu_count) in place of a single team_id. With LITELLM_ENABLE_PTU_COST_ATTRIBUTION on, routing hides the deployment from teams holding no share, the v3 limiter enforces each share as a per-team TPM ceiling from Azure's per-model TPM-per-PTU sizing, the daily flat-cost rollup splits the hourly cost by share, and /team/daily/activity plus the Team Usage page report PTU-hours next to tokens. * fix(proxy): weight cached input in PTU ceilings and size team-scoped deployments by public name The PTU flag is read from the process environment on every check instead of through the secret-manager path, so a proxy with a hosted secret manager does not pay a round trip per request. A team's ceiling now carries the model's cached-input ratio and settlement charges cache reads at that ratio, matching how Azure sizes a PTU. A deployment registered for one team is found by its team_public_model_name for PTU-hours and the ceiling, since its model_name is rewritten to a unique internal name. ptu_shares are StrictInt on ModelInfo and parsed_ptu_shares refuses booleans, strings, and non-string team ids, a fractional ptu_count reserves nothing, and models_processed counts distinct deployments rather than holdings. * fix(proxy): reserve PTU ceilings in normalized tokens, keep share routing flag-independent, and warn only shared or Azure reservations * fix(proxy): size the uncapped output floor against the PTU share in raw output tokens and escape the exported team id * fix(ptu): refuse ptu_shares naming a team that does not exist * refactor(ptu): keep Azure detection under llms/azure and check share teams with one query * fix(proxy): drop the reverted TeamIdSearchFilter import after merging main Main reverted the team key search change that added TeamIdSearchFilter, so the PTU share team check types its where clause locally * fix(proxy): chunk the ptu_shares team lookup under the IN-list bound * fix(proxy): count a request naming a shared PTU deployment directly against its group's ceiling * fix(proxy): resolve every name a shared PTU deployment is served under to its group's ceiling * fix(ptu): refuse ptu_shares declared without ptu_count and cost_per_ptu_per_hour * fix(ui): seed total_ptu_hours in the empty daily activity metadata * fix(ptu): size usage rows by the name the request used * fix(ptu): size a team's usage rows by the deployment it is served from * fix(ptu): never size a team's usage row by another team's reservation * fix(ptu): reserve an uncapped request against the cap the proxy writes, not its prompt length * fix(ptu): read a share map only through the terms registration requires A ptu_shares map stored without ptu_count, cost_per_ptu_per_hour and ptu_effective_from, or one that does not add up to ptu_count, is refused at registration but was enforced once loaded: every other team got the 400 and the share holder a ceiling while no flat cost accrued. _deployment_shares now reads the shares through ptu_terms, so such a row reserves nothing, the same as before the map was read. * fix(ptu): keep a declared split as the access rule and check a config split at boot with the flag off 6fb24d1d read ptu_shares through the pricing terms so a termless map reserved nothing, but with the flag off the router skips ptu_config_error for config.yaml entries, so such an entry loaded and served every team, against the rule that a declared split restricts access whether or not cost attribution is on. _deployment_shares reads the declared map again, and the router now raises ptu_config_error for a config-sourced entry declaring ptu_shares regardless of the flag, so a malformed split is refused at boot instead of being honoured or ignored silently. * fix(ptu): charge wildcard-served names, pass-through usage, and team-less settlements against the team's PTU ceiling A name served by a wildcard route now resolves to the shared deployment behind it, so the limiter's ceiling and the team daily activity sizing both find it. Pass-through responses settle the PTU counter from combined_usage_object in Azure-normalized tokens, and a reservation taken at admission is settled even when the success metadata lost the team. * fix(ptu): resolve a wildcard-served name to the one pattern the router picks, and a deployment id before any wildcard A team's wildcard reservation behind an open proxy-wide * route was charged and sized for traffic the * route served, and naming a shared deployment by id next to a catch-all wildcard escaped its per-team ceiling. Router.wildcard_route_deployments now owns the most-specific-proxy-wide-else-team precedence, and both the limiter and the consumption page resolve through it after listed rows and deployment ids. * test(router): pin the wildcard route precedence that PTU sizing and the router share * fix(proxy): accept whole-number PTU shares written as floats, as ptu_count already does A share of 2.0 now reads as 2 on POST /model/new and in config.yaml, where it was refused with a 422 and dropped at load. Fractional, infinite, boolean and string shares stay refused. Adds unit tests for the malformed-rate, provider-over-prefix, zero-token Responses usage, fractional-count and sizing-warning paths. * fix(proxy): settle team PTU reservations in their own minute and keep billed PTU days when shares move * fix(proxy): write each PTU deployment day's team charges in one transaction The backfill counts a deployment day as priced once any team's row exists for it, so a rollup that wrote team A's share and lost team C's left team C unbilled for good. Each deployment day's team rows now go through one Prisma batch with the retry around the whole batch, so a day is fully written or not written at all and the next backfill prices it for every team. * fix(proxy): read the PTU flag from a secret manager, serve share holders from their share, and refuse unfittable calls and share-holding team deletes * test(router): list the cached PTU ceiling helpers in the router coverage check * fix(proxy): read the PTU flag only for share holders, guard org and SCIM team deletes, keep share holders on their shared deployment through retries and the scheduler, and date PTU-hours by the terms in force * fix(proxy): bill a rerun PTU day to the current share holders alone * fix(proxy): replace a PTU day's rows by day and deployment, not a negated team list * fix(proxy): read the PTU flag from the secret manager first, like get_secret * fix(router): scope retry health lookups to the caller only for PTU share holders * fix(router): move the share-holder retry check into ptu_shares Adds ptu_hours and total_ptu_hours to the daily activity integration goldens and fixtures, since every route now reports them * test(integration): cover PTU shares across routing, delete guards, cost attribution and config regex timing --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-15 18:44:30 +00:00
Mateo Wang	18831b818c	fix: keep model capability flags out of provider requests (#46647) ... * fix: keep model capability flags out of provider requests A supports_* flag or another model map field set on a deployment's litellm_params was forwarded to the provider as a request field, so Bedrock answered 400 Extra inputs are not permitted on chat traffic and on the background health check alike. The shared owned-kwarg predicate now treats every ProviderSpecificModelInfo field as litellm-owned. Builds on the health-check-only approach in #40952 by leilei3167. * fix: treat every supports_ model map flag as a capability field and warn when one sits in litellm_params * fix(router): skip declared litellm params in the capability flag warning and warn for runtime-added deployments * fix(router): strip model capability flags from the router's deployments instead of every request * fix(router): keep capability flags out of health probes and read them from the loaded catalog * perf(router): read the catalog's supports_ field set once per cost map load Cache the loaded catalog's field names when the cost map snapshot is taken so the capability-flag check stops walking every catalog entry per deployment, and add integration cells that drive the flagged and control deployments through chat, streaming, messages, responses, embeddings, health and model management against a vendor that rejects the flags	2026-10-15 14:30:10 +00:00
Wang Yunzhuo	a2f792634e	fix: preserve explicit provider for async image, audio, and realtime routes (#36346) ... * fix(images): preserve explicit provider for async generation * test(images): prove a bare model name reaches its explicit provider on the async path * fix(images): validate async provider hints instead of reading untyped kwargs Read custom_llm_provider and api_base in aimage_generation through a TypeAdapter over a small TypedDict, so the provider lookup gets typed values and the basedpyright gate stays within its merge-base ceiling * fix: answer bad provider hints with a 400 and keep the explicit provider for async speech and transcription * fix: keep the explicit provider for realtime and cover the audio bad-hint 400 * fix: keep the explicit provider on the realtime HTTP routes and read base_url in async audio * fix: stop async image generation from rejecting a base_url it never reads * fix(audio): skip base_url validation when api_base is set on async speech and transcription * test(integration): cover a declared provider with a bare model on image, audio, and realtime routes --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-15 05:21:43 -07:00
devin-ai-integration[bot]	e136e91273	fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl (#46084) ... * fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl A prefix marked cache_control ttl 1h lost its pinned deployment after the fixed 300 s, so the next request went to another deployment and paid a cold cache. The pin TTL now follows the longest ephemeral breakpoint in the cacheable prefix, tools included: 3600 s for a 1h breakpoint and 300 s otherwise. * fix(router): give prompt cache pins their own tier and honour any declared ttl Pins now live in a DualCache of their own instead of the router's 200-slot in-memory tier, so an hour-long pin no longer evicts cooldowns and rate windows. The pin lifetime follows any <n>s|m|h cache_control ttl, floored at 300 s, not only the literal "1h". * fix(router): hash the pin ttl off the event loop and flush prompt cache pins with the router cache Compute the cache_control ttl scan through offload_token_count beside the prefix hash, clear the dedicated pin cache in Router.flush_cache, and point the order-fallback and priority-group pin tests at the pin cache they read * fix(router): pin 1h prefixes only on 1h-capable models, cap the pin at 1h, and give pins 10k in-memory slots * fix(router): match the 1h pin gate to what Bedrock forwards Bedrock deployments now decide the 1h pin with is_claude_4_5_on_bedrock, the same check that decides whether the 1h ttl reaches Bedrock, so an application inference profile ARN keeps a 5 minute pin whatever its base_model prices * test(router): give the anthropic_messages pin fixture its provider * fix(router): let each pin write set its own lifetime and require a positive 1h price In a group mixing 1h and 5m deployments the in-memory pin kept the lifetime of its first write, so a 5m deployment could hold an hour-long pin. Each write now clears the in-memory entry first, matching Redis. The Bedrock gate uses supports_bedrock_prompt_cache_ttl after #46058, and a zero 1h price no longer counts as 1h support. * test(router): integration cells for the one-hour prompt cache pin lifetime Cover every surface the pin lifetime touches: chat and /v1/messages through both SDKs, the responses bridge, response-cache hits, each provider family's 1h gate, malformed and duplicated ttl values, rewrites of one prefix, mixed and cross-group routing, concurrency, every Redis attach path, the in-memory slot pressure without Redis reads, a Redis outage and a killed worker * fix(caching): bound the in-memory expiry heap and count prompt cache pins in the memory debug routes * test(integration): cover prompt cache pins in the memory debug routes and the bounded expiration heap * fix(caching): serialize in-memory cache writes so a heap rebuild cannot lose another thread's expiry entry * fix(caching): keep the in-memory cache bounded when entries have no usable expiry * test(caching): cover the response cache bound when cached entries have no usable ttl --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-14 21:10:09 -07:00
Kerry Lu	7d7e28829f	test(integration): delete flaky team model TPM concurrency test (#46541) ... Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-14 16:24:08 -07:00
yuneng-jiang	f95e466bb7	test(integration): fix three more CircleCI integration reds from test setup (#46510) ... OpenRouter basic translation expected the pre-#44788 response shape. Read refusal, logprobs and system_fingerprint from the fixture at their new positions. The team model TPM test required exactly one of three concurrent requests to be served. The pipelined limiter can reject all of them, which is an accepted trade-off, so assert no over-admission and that a lone retry is served when the burst got nothing. The Voyage host tests count every host their proxy dials. The responses cost poller could dial api.openai.com for background-response rows that other tests left in the shared worker database, so turn polling off in those proxies.	2026-10-14 15:23:14 -07:00
devin-ai-integration[bot]	2d4472efc1	fix(vllm): write spend log rows for /vllm passthrough calls to router models (#45733) ... * fix(vllm): write spend log rows for /vllm passthrough calls to router models The vLLM passthrough config never built a logged response for relayed calls, so stream and non-stream calls through /vllm with a router model wrote no spend row and never charged the key. vLLM now shares the OpenAI-shaped relay logging that Azure already had, moved into an openai_like helper both configs call. * ci: run the vllm passthrough unit tests in the llms-providers shard * fix(proxy): log /vllm text completions and ask router models for stream usage A chat stream relayed to a router model without stream_options now gets include_usage added on the way out, and the usage chunk vLLM appends is removed before the client sees it, so the spend row carries vLLM's own token counts instead of a local estimate. /vllm/completions calls now write a row too, stream and non-stream. * fix(vllm): keep usage, fingerprints, and disconnect rows correct on /vllm streams Inject and strip stream usage on /vllm/completions too, hold every finish frame for the fingerprint when n > 1, pass a forced-usage server's stream through untouched, stop turning continuous_usage_stats alone into per-chunk usage, close the upstream when the client hangs up so the row is written, and assemble text choices with one sort instead of a scan per choice * fix(vllm): keep n>1 stream order when holding finish chunks for the fingerprint A held finish chunk now goes out as soon as any other chunk arrives, so a choice that finishes while another still streams keeps its place. Only the finish chunks right before the usage chunk get vLLM's system_fingerprint. * fix(vllm): keep caller stream_options and log both success handlers Forward a stream_options the proxy cannot read, or a non-boolean include_usage, to vLLM as the caller sent it. Only a false always_include_stream_usage changes the /vllm relay, so clients get the same stream bytes under every setting. Record the passthrough origin when logging starts so the sync and async success handlers both parse the relayed body, hand callbacks a parsed text completion, and read relayed messages and prompt separately so a stray field no longer drops the local count. Add the audit's integration tests for the relay, the settings, and the Azure spend row. * test(vllm): expect the caller's stream under always_include_stream_usage true * fix(vllm): relay long SSE frames in linear time * test(vllm): time the relay's usage and finish_reason prefilter on one large frame --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-14 12:40:48 -07:00
yuneng-jiang	0915691d32	test(integration): remove six setup causes of CircleCI integration reds (#46490) ... Drain the worker's fake-upstream request log when a test's gateway fixture starts, so traffic left by earlier tests on the same xdist worker no longer shows up in a test's first observation read Generate the fine-tuning rig's provider file and job ids per fixture build, so a rebuilt module fixture in the same worker no longer collides on the managed object unique constraint Wait for every wedged team_id and team_alias invalidation publish to fail before taking the recovery marker, now that team updates publish both keys Pin the cooled-down priority scheduler request to the connection that observed the cooldown, since a sibling worker reads cooldowns from Redis on an interval Simulate upstream outages and the held budget peer on ports the test keeps owning, instead of releasing a port another worker can bind Start a peer proxy for the accounting group, which has a test that needs one	2026-10-14 12:00:35 -07:00
Rick	4f3cb48dc1	feat(dashscope): add Wan 3.0, Wan 2.7 and HappyHorse video generation via /v1/videos (#43706) ... * feat(dashscope): add Wan 3.0, Wan 2.7 and HappyHorse video generation via /v1/videos DashScopeVideoConfig maps litellm's OpenAI Videos API onto Alibaba Model Studio's async task API: create hits /api/v1/services/aigc/video-generation/video-synthesis and status and content poll /api/v1/tasks/{task_id}. It is registered for dashscope and for the qwencloud and qwen_ai_platform brand aliases, following the same get_dashscope_family_*_config pattern image generation uses. Unlike the Sora-shaped providers the request body is nested, so prompt and media go under input and everything else goes under parameters. X-DashScope-Async is always sent, because DashScope rejects the call outright without it OpenAI params translate to DashScope's contract. seconds becomes duration, and size yields both a ratio and a named resolution tier (1280x720 gives 16:9 plus 720P), since sending only the ratio leaves the request on the default 1080P and silently bills at that rate. input_reference becomes a single media entry, base64 encoded into a data URI for file inputs. Which media type a model takes (first_frame or, on the reference-to-video models, reference_image) is read from provider_specific_entry.dashscope_video_reference_input via get_model_info rather than hardcoded; models without it predate the media array and get the flat img_url field. A caller-supplied media array always wins, as that is the only way to express multi-role combinations The create call is the only billed call, so it bills what DashScope will deliver: the requested duration and tier, or the documented 5 second and 1080P defaults when they are omitted. Smart duration (duration -1) lets the service pick the length after that call, so it is rejected rather than recorded at zero, with a 400. Task creation and task lookup both report failures as a 200 carrying top-level code and message with no output, so that shape is raised instead of surfacing as a queued video with an empty id. Status responses re-encode the deployment model from the polled id, so follow-up polls and downloads route to the same deployment. Timestamps parse as UTC+8. DashScope publishes no list, delete or remix endpoint, so those raise rather than guess at a URL The end-to-end tests drive the real avideo_generation() entrypoint through an httpx MockTransport and assert on the JSON body httpx actually encoded. That is what catches a mapper that drops input_reference before the request is built, or a body fragment json cannot serialize, neither of which a test calling the transform directly would see Both model_prices maps gain wan3.0-video, wan3.0-video-prime, wan2.7-t2v, wan2.7-i2v, wan2.7-r2v and the HappyHorse 1.0 and 1.1 t2v, i2v and r2v models, for dashscope and the qwencloud and qwen_ai_platform aliases, priced per output second by resolution tier from Model Studio's USD rate cards. dashscope and qwen_ai_platform default to the Beijing host and use its rate card; qwencloud defaults to the international host and uses Singapore's. The three *_API_BASE_VIDEO overrides are documented in BerriAI/litellm-docs#1920 Co-authored-by: AaronHowell <237895480@qq.com> * fix(dashscope): fail loudly on video responses that cannot be read Addresses the review on the video generation PR. A body that is not the documented task shape, an output block missing task_id or task_status, an undocumented task_status and a missing output all raise DashScopeVideoError now, instead of defaulting into a queued video with an empty id that callers poll forever. A JSONDecodeError from a gateway response no longer escapes as a generic connection error. task_id and task_status are required on _TaskOutput, and a missing output raises even when the body carries no code or message. Download failures move into the 4xx range, since a 5xx has the proxy retry them: 409 while the task is still running, 400 for a task that failed, 404 for an id that expired. The status a provider reported is kept when it reported one, so an in-body failure that arrives on a 200 is billed as the 400 it is rather than as a retryable connection error. characters and image are only dropped when drop_params is set; the params that are rebuilt into the body or consumed by the caller (model, prompt, user, extra_*) stay silently skipped, since the proxy stamps user from the caller's end-user header. Co-authored-by: AaronHowell <237895480@qq.com> * fix(dashscope): clear the lint gates after merging main * fix(dashscope): accept file tuples and paths as video input_reference video_generation() types input_reference as FileTypes, which includes (filename, content[, content_type[, headers]]) tuples and pathlib paths, but the DashScope encoder only read bytes and file objects, so those image-to-video calls raised before reaching DashScope. Bare strings stay URLs and are never opened as paths. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dashscope): snap size to a supported ratio and read file tuples and paths * test(dashscope): assert whole mapped values and bill against hand-written rates The tier pricing test read its expected rate from the same map the cost calculator uses, so a wrong rate passed. Billing is now checked against fixed rates driven through the status response usage, while the map test only asserts every supported tier has its own rate. Size mapping tests compare the full mapped dict, and the single-test wiring class is a plain function. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(videos): download content through the caller's client and stop retrying dead tasks Two Cursor Bugbot findings on the DashScope video PR. The DashScope artifact lives on OSS, so transform_video_content_response makes a second request. It built its own cached client, discarding an injected client and its ssl_verify. Thread the client through the content response the way transform_video_status_retrieve_response already does, and use it in the three providers that fetch from a second URL (DashScope, RunwayML, fal.ai). The remaining providers accept it and read the response body, matching how they already accept an unused client on status retrieve. A FAILED or CANCELED task reported without a code or message fell past the in-flight check into the 502 branch, so the proxy retried a task that can never produce a video. Raise 400 for a terminal status instead. * fix(dashscope): answer 404, not 409, when a Wan video is downloaded before it finishes A download that lands while the task is still PENDING or RUNNING answered 409. That reads as a client error, which was the intent, but litellm's retry predicate counts 409 as retryable, so the proxy polled DashScope twice more before giving up and then counted the caller's own early download against the model's error rate 404 is what the OpenAI videos API documents for content that is not ready, and it is the one status in that family the retry predicate leaves alone. The rest of the mapping is unchanged: 400 for a task the provider reported as failed, 404 for an expired task id, 502 when a succeeded task carries no artifact The download tests now drive avideo_content and assert the caller sees NotFoundError rather than an internal server error * fix(dashscope): route a video poll back to the model group that created the task The create response encoded the upstream model name into the returned id. Both the dashscope and qwen_ai_platform prefixes serve the same model names from their own hosts, so a proxy with a deployment per region had deployments that only the model group told apart. Encoding the bare model name resolved a poll to whichever deployment the router lists first, which sent it to the other region's host with that region's key and a task id that region never issued. That is the reroute the earlier proof records as the dashscope and qwen_ai_platform poll ids coming back tagged qwencloud The id now carries the model group the create was routed to, which is what the proxy resolves a poll back to, and falls back to the model name for a direct SDK call that has no router The regression test drives a real router with one deployment per region, under a different prefix and under the same one, and asserts the poll carries the key of the region that created the task * fix(dashscope): read the routed model group from litellm_metadata The router stamps the model group it picked into litellm_metadata. metadata only carries a copy when the caller sent no metadata of its own, and every proxy request sends some, so a proxied create still encoded the bare model name and the poll resolved to whichever region's deployment the router lists first The regression test now also creates the video with caller metadata, which fails without this change * fix(dashscope): reject unbillable video requests and keep fal and runway on their own clients * fix(dashscope): show the caller why a video task is gone or not ready instead of the bare upstream body * fix(dashscope): check video request values before the billed create and honor per-call drop_params Type check documented DashScope parameters so a bad seed answers 400 instead of a billed failed task, pass the call's and deployment's drop_params into the video mapping, answer 400 for an expired task id so the deployment is not cooled down, read DASHSCOPE_API_BASE for video, keep polling SUSPENDED tasks, and refuse to map size onto models litellm has no price for. * fix(dashscope): carry the requested model name in video ids so alias and wildcard keys can poll A key scoped to an alias or to a name a wildcard serves could create a video and then got a 403 on the poll and the download, because the id carried the group that served the call (wan3, qwencloud/*) and the key was never given that name. The id now carries the name the router was asked for on that hop, and the router resolves an alias, routing group or wildcard name in an id back to itself when no deployment name or provider model matches it first * fix(dashscope): keep a routing group's video polls on the member that created the task A routing group spans several model groups, so an id carrying the group's name load-balanced its polls, and some left for another region's host and key with a task id that region never issued. The id now carries the member that served the task when the request named a routing group, and keeps the requested name for a model group, an alias, or a wildcard match. * fix(dashscope): carry the requested and served model in video ids so group-scoped keys poll in the creating region * fix(proxy): pin a video's served group only after key and team aliases resolve * test(integration): audit DashScope video create, poll, download, routing and chaos paths * fix(proxy): keep catch-all video polls on their provider and count served polls against the requested group * test(routing): expect a forged served group to be dropped from the polled video id * test(integration): cover rate-limited routing group polls and catch-all polls of DashScope videos --------- Co-authored-by: AaronHowell <237895480@qq.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 10:10:07 -07:00
devin-ai-integration[bot]	e226016aee	test(integration): cover health_routing endpoint contracts (#44744) ... * test(integration): cover health_routing endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tighten health_routing contracts per review Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): close health_routing review gaps Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(integration): format observed routing fallback cells Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): close health_routing final review findings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): resolve health_routing review findings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-14 06:34:22 +00:00
yuneng-jiang	ec58b889cc	test(integration): isolate shared proxy state behind six order-dependent flakes (#46172) ... Each test below passed or failed on the same commit depending on test order or port timing in the CircleCI integration workflow - readiness probe only polls a port the launched proxy owns, so a stolen port no longer sends GET /health/integration-readiness to another test's wire server - team update tests seed fresh member users, since a proxy admin relocation adds team members to the destination org and leaked org_b membership into the shared INTERNAL_USER - realtime whisper tests wait until every proxy worker serves only the new gpt-realtime-whisper deployment, via a per-worker deployment id route - configured fallbacks test runs on its own proxy and database, because /config/update persists router settings and cannot unset fallbacks, which left [] and changed the no-fallback error text other tests assert - otel excluded services tests each get a private Redis so a sibling proxy cannot warm the budget registry cache the tenant span depends on - Langfuse audit upstream uses unique response ids instead of len(targets), which could repeat under concurrency and drop SpendLogs rows	2026-10-13 09:55:11 -07:00
devin-ai-integration[bot]	7efce7f8e0	test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) ... * test(integration): give chaos and timing tests a harness API and gate raw signals Chaos tests killed and suspended proxy workers with raw os.kill and psutil calls and bounded elapsed time with fixed literals, so they raced the uvicorn worker healthcheck and failed on slow CI boxes OwnedProxy now offers kill_worker and replacing_worker, which return only once a replacement answers readiness with its own pid, and suspended, which is refused unless the proxy was launched with a lenient worker healthcheck. timing.py bounds durations against a measured control or the injected delay. run.py measures the proxy import time once and derives readiness deadlines and the per-test timeout from it, writing boot-calibration.json check_test_quality.py adds TQ010 (raw process signals) and TQ011 (elapsed time asserted under a numeric literal) for tests/integration outside _support, and every call site is migrated so both report zero with no suppressions * test(integration): steer the assistants wire burst onto both workers The mixed-status burst assumed the OS would spread 48 connections across both workers and failed when one worker took them all. Each half is now sent while the other worker is suspended, and the test asserts an exact 24 and 24 split * test(integration): restore product-enforced timing bounds lost in the migration Some TQ011 migrations dropped or loosened a bound whose time is the behaviour under test. The websocket session cap only kept its floor, so a cap closing at 300 s would have passed, and several timeouts were bounded by the injected stall instead of the configured product value assert_enforced_within bounds an elapsed time by the configured value plus explicit slack, with an optional floor. The websocket cap, the MCP validation timeout, the guardrail litellm_params timeout, the prisma command timeouts, the baseline projection lock timeout and the key metadata statement timeout use it. The device code refusal keeps its 15 s margin, and the langfuse retried prompt fetch is now bounded by a cold fetch of a steady prompt as well as by the ignored retry-after * fix(tests): wait for the supervisor to reap killed workers and fix real-kill migrations The lenient worker healthcheck was the whole case budget, so uvicorn's supervisor sat in a ping on the killed worker for up to that long before respawning it. It now uses the graceful stop budget, and a worker kill returns only after the victim is reaped, so pooled connections no longer race the dead worker's socket. Several tests on main picked the multiprocessing resource tracker as their victim and never killed a worker. Now that they kill a real one, presidio counts streams cut by the death, delete_model waits for the config reload that removes a deleted model from a replacement worker, and the wedged coordination Redis test kills before the wedge and waits for every parked publish to fail before resuming Redis. The redis cluster test imports the harness through the tests package so it collects in the unit job, and the per-model fallback rig exposes the owned proxy it now needs. * test(observability): only require S3 deliveries the killed worker could not have buffered A real SIGKILL drops the victim's unflushed batch, so requests it already answered can't be expected to land. The test now requires every request served after the replacement to land, some of the surviving worker's burst share to land, and nothing unsent to land --------- Co-authored-by: Yuneng Jiang <yuneng@berri.ai>	2026-10-12 05:27:33 -07:00
devin-ai-integration[bot]	ca0ff24726	fix(databricks): call the ai_decide AI Function route for decisions and the auto-router (#45722) ... * fix(databricks): call the ai_decide AI Function route for decisions and the auto-router Databricks serves ai_decide at POST <host>/api/2.0/ai-functions/ai-decide, not on a serving endpoint's /invocations route. The decisions provider and the auto-router classifier now accept only the ai_decide model, post to that route without a model field, unwrap the response envelope, and map a noul answer's probability to the System One noul field. * test(databricks): bound the megabyte model name test by scaling, not wall time * test(databricks): assert the whole OpenAI-format decision and floor the liveliness bound * test(decisions): move the Databricks OpenAI-format wire row to the ai_decide route * fix(databricks): keep serving endpoint decisions beside the ai_decide route * test(databricks): assert the whole classifier request body in the test routing case * test(databricks): correct System One spend call type Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-10 23:58:16 -07:00
Shivam Rawat	15154a40a8	fix(proxy): let admins make per-model key/team rate limits a hard 429 instead of falling back (#45459) ... * fix(proxy): let admins make per-model key/team rate limits a hard 429 instead of falling back Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): sync per-model rate limit API types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): type per-model fallback setting access Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep the per-model hard cap inside the fallback loop and test it through the real limiter * fix(proxy): list the per-model hard cap setting on the settings page and cover every per-model descriptor * fix(proxy): read disable_fallbacks_on_per_model_rate_limits as a boolean value and cover the team-wide per-model cap * test(proxy): cover per-model rate limit 429s skipping fallbacks on a live proxy --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-09 23:10:23 -07:00
Mateo Wang	883e3210ee	feat(decisions): add Databricks ai_decide as a /v1/decisions provider and auto-router decider (#45200) ... * feat(decisions): add Databricks ai_decide as a /v1/decisions provider and auto-router decider * fix(decisions): reject Databricks endpoint names carrying URL separators and keep the provider wiring under llms/ * fix(decisions): reject dot-segment Databricks endpoint names and add Databricks to the dashboard's OSS classifier editor * fix(ui): name every OSS classifier provider in the classifier radio copy * feat(decisions): register the Databricks OpenJev decider as an evaluation-mode cost-map entry * fix(cost-map): declare zero cache rates on the databricks-openjev-qwen35-4b entry * fix(proxy): reconcile every DB deployment when a registry miss lands on an auto router A /model/new for an auto router lands on one replica. A request routed to that auto router on a sibling replica missed it, the model read-through loaded only the auto router's own row, and the pre-routing hook then picked a tier the sibling had not loaded, so the caller got a 400 "no healthy deployments for <tier>" until the next periodic reload. An auto router miss now runs the full add_deployment reconcile, tiers included. Found by the audit's two-worker rig on this PR's routing cells; CircleCI's one-worker proxy never opens the window. * test(integration): audit cells for Databricks decisions and the Databricks auto-router classifier Wire cells for /v1/decisions with databricks/<endpoint> deployments, routing cells for the complexity router's databricks classifier on all three endpoints with the Postgres rows read back, management cells for the classifier's spend writes and for the evaluation-mode health check of a Databricks serving endpoint.	2026-10-09 16:19:23 -07:00
yuneng-jiang	cddc7cde97	test: fix stale and flaky tests across CircleCI, GHA and Buildkite (#45530) ... * test(integration): poll the partition lock witness off the event loop The witness poll ran a blocking psycopg connect and query on the test's event loop, so the partition DDL task only progressed during the 20ms sleeps and missed the 3s deadline on loaded runners. Polling through asyncio.to_thread lets the DDL reach the held lock concurrently, and the deadline is 10s since it only bounds how long the DDL takes to start waiting * test(e2e): retry the reasoning turn until the model emits a reasoning item Whether gpt-5.4-mini emits a reasoning item next to a forced function call is up to the model, and it skipped it on four scheduled runs while a same-SHA rerun passed. The first turn now retries up to three times before the unchanged reasoning-item assertion * test(e2e): retry a Bedrock Converse model once when it reports no cache tokens Opus on Bedrock Converse reported zero cache tokens on half of the scheduled runs while a same-SHA rerun passed. A model that comes back uncached is asked once more inside the 5 minute window, and the non-zero cache token assertion is unchanged * test(e2e): poll until a deleted stored response stops being retrievable Azure kept serving a just-deleted streamed response for a moment, so a single retrieve did not raise. Both delete tests now poll retrieve until it returns an error and still require a 4xx * test(e2e): give the Together prompt cache five primed attempts Together documents its prefix cache as best-effort, and three primed attempts came back with zero cached tokens once while a same-SHA rerun passed * test(e2e): allow 30s for the idle RSS reading at session start A loaded router replica took longer than 10s to answer the session-start memory read, which pytest reruns cannot recover. The RSS budget assertion is unchanged * test(e2e): scope the MCP submission check to its own card The register call timed out at 15s on a loaded stack and leaked a submitted server, so the retries matched two "3 passing, 1 failing" cards. Registration gets 60s and the check reads only the submitted server's card * test(router): wait for the primary success record before counting it The shadow fan-out test waited only for the two shadow success events and then asserted exactly one primary success, which the logging worker sometimes had not delivered yet. It needed a rerun on 5 of 59 main runs * test(proxy): give the spend-log cleanup run a 1s budget A 0.25s budget could expire before the first batch on a loaded xdist worker, leaving rows_deleted at 0. One second still stops the 50-batch, 5-second loop on the deadline, which is what the test proves * test(autorouter): release the slow token count at 2.05s instead of 2.3s The count only has to finish after a quarter of the 8s worker budget, and the planner gives it 3s, so releasing at 2.3s left 0.7s of slack that a loaded runner used up. 2.05s is still past the quarter mark with almost 1s of slack * test(xai): run the reasoning-effort tests on grok-4.6 with a prompt that needs reasoning grok-4.7 now returns reasoning tokens without any reasoning text, so reasoning_content was never set and both tests failed on every scheduled run. Called directly, grok-4.6 returned reasoning text 6 of 6 times on a step-by-step arithmetic prompt but only some of the time on a bare greeting * test(integration): give the two worker-kill chaos tests the owned-proxy time budget Each test spends about 50s on the burst and then up to graceful_stop_seconds() stopping its owned proxy, which overran the 90s default pytest timeout on loaded runners. They now use the same 2 * graceful_stop_seconds() + 120 budget as other owned-proxy tests * test(integration): compare regional image responses without their created timestamp Two identical generations that straddle a second boundary differ only in created, which failed the byte-for-byte comparison. Every other field and both spend rows are still compared * test(integration): count only the S3 logger's flush task The set of asyncio tasks created while the logger starts also picked up client close finalizers left by earlier tests, so the count ranged from 1 to 8. The test now counts the periodic_flush task it owns and cancels * test(integration): send guardrail timeout probes eight at a time All 44 probes went at once to a two-worker proxy, so on a loaded runner some guardrails used their whole 1s timeout before their request left the proxy, and the sink never saw them. A different set of providers failed on each run. Eight in flight still overlaps the waits * test(integration): count a killed Arize chaos worker as gone once it is a zombie psutil reports an unreaped zombie as still running, so the 10s death check failed whenever the supervisor was slow to reap the SIGKILLed worker, and the stop then overran the 90s default timeout. The test now accepts a zombie or a missing process and has the owned-proxy time budget * test(integration): send the Typesafe connection test its real model and endpoint The test passed the proxy alias as litellm_params.model, and /health/test_connection lays the request over the stored deployment, so the alias replaced the real typesafe/ model and the check failed with "LLM Provider NOT provided" on every run since it landed in #45481. It now sends the provider model, api_base and api_key directly * test(integration): keep DB-stored models out of the usage-routing Redis read test The owned proxy inherited store_model_in_db and the job's shared database, so a model another test left behind joined the router and added its cooldown key to the MGET the test compares exactly. Reproduced locally with one /model/new model present (3 failed), and green with model loading from the database turned off * test(vertex_ai): prove the batch upload streams by laziness instead of peak memory The two tracemalloc ratio tests flaked on unrelated PRs because peak memory on a shared xdist worker includes other threads' allocations and garbage from earlier tests. They are replaced by deterministic checks of the same property: the upload stream parses and maps a row only when it is pulled, so a body whose tail is not JSON yields its valid rows first, and a Path source reflects a row rewritten on disk after the upload started. Making the parse, the output, or the file read eager fails these tests * test(integration): keep the alias test_connection call as a known bug * test: type the counting mapper and the Bedrock rerun helper * test: annotate the new test locals as Final and build them as tuples * test(vertex_ai): prove pull-driven transforms with the garbage tail alone	2026-10-09 11:40:14 -07:00
yuneng-jiang	72ab736863	test(routing): wait on deployment registration instead of boot model-info traffic (#45267)	2026-10-08 21:31:53 -07:00
devin-ai-integration[bot]	50ab9a3211	fix(token_counter): price a base64 PDF document per page instead of as one image (#45301) ... * fix(token_counter): price a base64 PDF document per page instead of as one image A `document` or `file` block carrying inline PDF bytes was priced like a single image (85 tokens), so when a provider's count-tokens endpoint rejected the model (Bedrock Opus) the local fallback answered 116 for a 12-page PDF the provider then billed at 35941 input tokens. The counter now reads the PDF with pypdf and prices each page as its extracted text plus the image Anthropic renders it to (1568 px long edge, 1.15 MP, 750 pixels per token), falling back to the old image pricing when pypdf is missing or the bytes are not a readable PDF. * refactor(token_counter): count PDF pages as they are read Sum each page's text and rendered-image tokens straight from the pypdf reader instead of materializing a page list first, keep the fallback to image pricing atomic when a page cannot be read, and annotate the new tests' locals as Final * test(integration): audit cells for page-priced PDF documents in count_tokens, pre-call checks and spend * test(integration): release the held peer when the concurrent budget wait times out --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-08 21:20:46 -07:00
devin-ai-integration[bot]	8a176c0f0a	fix(router): drop the encrypted reasoning a fallback hop's target cannot decrypt (#45393) ... * fix(router): drop the encrypted reasoning a fallback hop's target cannot decrypt An order-based or configured fallback hop replayed the failed provider's encrypted reasoning items to the next deployment, which answered 400 (Bedrock Mantle: invalid encrypted reasoning; OpenAI: invalid_encrypted_content), so every multi-turn Responses fallback for Codex-style clients failed. The hop now drops the encrypted reasoning its target cannot decrypt and keeps each item's readable summary. With encrypted_content_affinity on, the pin narrows to the hop's target order instead of emptying it, so the hop reaches the next order instead of failing with no deployments available. * fix(router): keep encrypted reasoning a same-boundary fallback hop can decrypt Unmarked encrypted reasoning on a hop is attributed to the deployment that just failed, read from the retry breadcrumb, so a hop to a deployment on the same api_base and api_key keeps it and a cross-provider hop still drops it. The hop tests script the upstream at the httpx boundary instead of doubling the handler, and the router coverage script lists the two hop helpers with their tests * fix(router): read the hop's failed deployment from its own metadata bucket and carry it into the Responses mid-stream snapshot * test(router): use a real Router without the origin deployment in the hop strip test * test(router): cover the hop strip when no failed deployment is known * fix(proxy): drop the router's fallback hop state keys from the client body * fix(proxy): keep a request's max_fallbacks cap, drop only the hop state keys * test(integration): audit cells for the fallback hop encrypted reasoning strip Forty-five checked-in cells under tests/integration/routing prove the hop strips the previous deployment's encrypted reasoning on /v1/responses, /v1/chat/completions and /v1/messages (httpx, OpenAI and Anthropic SDKs, sync and async, streaming and not), that the affinity pin yields to the hop, that a client-sent fallback_depth, _target_order and attempted_targets never move or strip a request, and that a concurrent burst, an order-1 outage and a killed worker keep every request stripped and logged once. Every call goes through a lane pinned to one worker that already lists the deployments it needs, because the peer worker learns a /model/new row through the config-sync resync up to sixteen seconds later --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-08 17:30:01 -07:00
devin-ai-integration[bot]	d5173e3d70	test: move 76 live top-level tests to offline unit and integration coverage (#45352) ... * test: move 76 live top-level tests to offline unit and integration coverage * test: keep the mock-param opt-in guard valid once the top-level senders are gone * test: address review on offline replacements * docs(test): drop the deleted harness test file from the harness check command * test: make the key rebind, team member delete and routes integration tests exercise the legacy paths * test: make fallback, rpm and spend integration contracts deterministic and clean up their rows, assert the rpm limit in usage-based routing * test: expect the no-deployments error at the rpm limit and cover the strategy check without pre-call checks * test: hand member cleanups to the scenario instead of growing a budget list, flatten callback kinds * test: scope the admin health check to the test's own deployment --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-08 10:33:35 -07:00
devin-ai-integration[bot]	9df3d4c96b	test(integration): boot owned proxies with a readiness-sized worker healthcheck budget (#45292) ... * test(integration): boot owned proxies with a readiness-sized worker healthcheck budget * test(integration): keep owned_proxy_process's extra_arguments for cells that pass their own flags --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-07 23:40:26 -07:00
Mateo Wang	a7ee038592	fix(router): keep include_fallback_errors off the provider call on the sync Router path (#45218) ... * fix(router): keep include_fallback_errors off the provider call on the sync Router path Router.completion spread include_fallback_errors into litellm.completion, so OpenAI answered 400 Unknown parameter and a fallback walk lost the backup's reply. Only Router._acompletion popped the router-only flags. Both paths now build the provider call through one shared helper that drops them, and a wire-level respx test covers each path. * test(router): pin include_fallback_errors off the wire across the sync, async, and proxy paths	2026-10-07 22:25:12 -07:00
RachelHuangZW	bf9f59d76b	fix(scheduler): remove a request's queue entry once it stops waiting (#43061) ... * fix(scheduler): remove a request's queue entry once it stops waiting Requests admitted while a healthy deployment existed never left the priority queue, and neither did requests cancelled while waiting. The stale entries blocked later requests during cooldown and, with Redis, made add_request raise a TypeError on queues read back as JSON lists Both scheduling paths now share one polling helper that removes the entry in a finally block, whether the request was admitted, timed out or cancelled Related to #43059 * test(router): allowlist _wait_for_scheduler_turn in the router coverage check The coverage script only counts direct calls in test files. The helper is exercised through prioritized acompletion and atext_completion in test_router.py, like the other allowlisted entries * fix(scheduler): admit healthy requests before reading the queue, and clean up cancelled enqueues poll() raised on an empty queue before checking for healthy deployments. With the cleanup now rewriting the queue after every admission, a concurrent write from another replica can erase a waiting request's entry, and that request then failed while a deployment was healthy. poll() now admits as soon as a deployment is healthy and only reads the queue during cooldown add_request also moved inside the try block, so a request cancelled while its queue write is in flight still has its entry removed * refactor(scheduler): move the wait loop into Scheduler.wait_for_turn The router passes a healthy-deployments callable into the scheduler, so tests inject a Scheduler directly instead of replacing the router's scheduler attribute. The cancelled-mid-enqueue test moves to test_scheduler.py with the other scheduler tests, and poll() takes the deployments as a Sequence since it only checks whether any are healthy * test(scheduler): move scheduler tests into tests/unit * fix(scheduler): finish the queue removal when cancellation is delivered again during cleanup A second cancel, or an anyio cancel scope that re-cancels on every await, interrupted remove_request mid-write and left the entry in Redis. * fix(scheduler): read queue entries back from redis as tuples * fix(scheduler): admit a request whose queue entry vanished during cooldown * fix(scheduler): re-enqueue a request whose queue entry vanished during cooldown * test(integration): cover priority scheduler queue cleanup across instances Adds tests/integration/routing/test_priority_scheduler_queue_cleanup.py: 27 cells against the real proxy (two workers, Postgres, Redis, a scripted upstream) for every prioritized surface (/v1/chat/completions, /v1/completions, /queue/chat/completions, streaming and not, OpenAI SDK sync and async, raw httpx), the non-integer priority pass-through, the in-memory queue on one proxy, two proxies sharing a Redis queue (served requests leave no entry, a dead replica's entry is skipped or expires, a waiter that times out or disconnects removes only itself), a Redis outage mid burst and a SIGKILLed worker. Each cell asserts the caller's response, the upstream's requests by marker and the Redis queue contents. On the merge base the cross-instance cells fail with the list-of-lists TypeError and the in-memory cell with 408s behind the leaked entry; on this branch every cell passes --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-07 17:34:38 -07:00
devin-ai-integration[bot]	e98bbc2f8e	fix(exceptions): map upstream 402 to PaymentRequiredError and cool down 402 deployments (#44879) ... * fix(exceptions): keep upstream 402 status and cool down 402 deployments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(exceptions): map 402 to PaymentRequiredError subclass of BadRequestError Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(exceptions): annotate PaymentRequiredError methods Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): skip 402 cooldown on single-deployment model groups Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(exceptions): single prefix and 402 fallback response for PaymentRequiredError Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): map billing_error to PaymentRequiredError regardless of status Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): honor explicit allowed-fails policy for single-deployment 402s Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * revert(anthropic): drop billing_error body mapping to PaymentRequiredError Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(exceptions): type the PaymentRequiredError constructor parameters * test(integration): cover 402 PaymentRequiredError mapping and cooldown --------- Co-authored-by: Mubashir Osmani <mubashir@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-06 20:35:13 -07:00
devin-ai-integration[bot]	68d9b8bbb8	fix(router): retry a /v1/messages stream the provider drops before the first content chunk (#44276) ... * fix(router): retry a /v1/messages stream the provider drops before the first content chunk A /v1/messages stream that the upstream closed before any content reached the client answered an error event after a single attempt, so the router's num_retries never applied to that drop. The pre-content failure is now retried within the model group before the fallback chain runs, with the budget resolved the way a failure raised before the stream opened resolves it: a retry policy that names the error class, then the request's num_retries, then the deployment's, then the router's. A drop after content reached the client keeps surfacing the provider's error after one attempt. Fixes #44238 * fix(router): hand a retry's non-retriable error to the fallback chain and type the retry helpers A retry that failed before its stream opened with an error no retry covers raised straight to the client, skipping a fallback the first attempt would have used. assert_never now comes from typing_extensions so the router imports on Python 3.10, and the retry helpers read their kwargs through typed narrowing instead of Mapping[str, Any] * fix(router): cast the untyped router fallback defaults the stream retry gate reads The retry gate passed the router's fallback attributes, declared without element types, to the typed request override helper, which basedpyright counted as new unknown-argument errors * fix(router): consult context_window_fallbacks when a retried /v1/messages stream overflows A retry attempt raising ContextWindowExceededError reached the fallback chain inside its mid-stream envelope, so only the regular fallbacks list matched. The fallback attempt now unwraps it the way it unwraps a content policy error. The new router helpers are covered for the router code coverage check with two direct-call tests and named covering tests * fix(router): retry a 408 raised by a /v1/messages retry and honor deployment num_retries before the stream opens * fix(router): attribute a retried /v1/messages stream to the deployment that served it and bound the retry-policy hold * fix(router): retry /v1/messages error frames under their retry-policy class and keep the first drop's committed budget An `event: error` frame that arrives before the first content delta now raises the exception class the pre-stream mapping gives an HTTP answer with the same status (429 RateLimitError, 500 and 529 InternalServerError, 503 ServiceUnavailableError, 504 Timeout), so a retry policy's per-class budget governs it the way it governs the error before the stream opened. The status the client sees is unchanged A retry that lands on a sibling deployment keeps the budget the first drop committed to, read back from the request's attempted_retries and max_retries, instead of recomputing it from the new deployment's num_retries, matching the pre-stream retry loop * refactor(anthropic): keep the error-frame exception mapping under llms and type the retry test helper The status-to-exception mapping an `event: error` frame gets before the retry policy is consulted now lives next to the Anthropic error status map in llms/anthropic/common_utils.py, with its own unit test, and the two-deployment retry test helper takes explicit typed parameters instead of a bare dict and untyped kwargs * refactor(anthropic): map an error frame's status with explicit returns on every path * fix(router): map stream error frames through the pre-stream exception mapping An overloaded `event: error` frame on a /v1/messages stream now raises the InternalServerError a 529 answer maps to, built by exception_type from the frame's own body, so one retry policy class governs the error before and after the first byte; a failed fallback after such a frame answers 500 like every other litellm path instead of the frame map's 503 A model_group_retry_policy that does not parse (a non-integer budget, an entry that is not a mapping) no longer fails every healthy stream of that group before its first attempt: the stream runs with no policy and the plain num_retries budget, with a warning naming the group * fix(router): forward an error frame nothing can take over for as the provider sent it A pre-content error frame whose class the retry policy grants no retry, with no fallback configured, raised an HTTP error only on the first attempt while the same frame after exhausted retries reached the client verbatim. Both now pass through as sent, the way the merge base forwarded every frame. * test(integration): audit /v1/messages pre-content retry across routes and budgets Adds the /audit cells for the pre-content stream retry: the native Anthropic route (drops and error frames before content, HTTP rejections before the stream opens, SDK sync and async, after-content and non-retriable controls, budget exhaustion, cache twin, spend row and headers), the chat and responses bridges, the generic routes (responses, chat, vllm pass-through, Gemini generateContent, fine-tuning jobs list), owned two-worker proxies for router-level budgets, retry policies and fallbacks, and two chaos cells (a worker killed mid burst, an outage on every first attempt). Shared helpers for scripted Anthropic SSE upstreams and OpenAI-compatible wire replies live in tests/integration/_support --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-05 22:53:39 +00:00
devin-ai-integration[bot]	3286782dea	fix(caching): key response cache by router model group in litellm_metadata (#44542) ... * test: align integration fixtures with current behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): wait for the spend flush before asserting its trace placement Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): key response-cache entries by model group from litellm_metadata /v1/responses routes through the router with model_group in litellm_metadata, which the cache key ignored, so identical requests to different model groups sharing one underlying model hit each other's cached responses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-05 17:06:21 +00:00
devin-ai-integration[bot]	9d16412341	fix(caching): count tool_call cache_control marks in the injection census (#43556) ... * fix(caching): count tool_call cache_control marks in the injection census Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): remove cache census casts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): only skip injection on message or content marks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): skip injection on messages whose tool calls carry marks Reverts 9f08d8aef8. A default 5m mark injected on assistant text lands before the client's 1h tool_use mark, which Anthropic rejects with a 400 because a 1h breakpoint must not follow a 5m one. Keeping the full census in the skip check leaves the client's tool_call breakpoint as the only one on that message. * fix(caching): count every client tool_call cache mark in the breakpoint census The census gated tool_call marks on type function and dict shape, so a client mark on a call without a type or with a string cache_control slipped past the count and injection overflowed the 4 breakpoint cap. Count any non-None tool_call mark, keep the server tool exclusion, and add integration cells for the capped surfaces, the yaml stand-down, Bedrock and Gemini, and router affinity * test(integration): hold the upstream so the worker kill lands mid-burst * refactor(caching): reuse the transform's server tool lookup in the breakpoint census The census now calls the same helper the Anthropic transform uses to decide whether a marked tool call becomes a server tool block, so the two cannot drift apart. The owned-proxy burst test waits up to 90 seconds for the burst to reach the wire before it kills a worker * refactor(anthropic): move the server tool rebuild check under llms/anthropic * test(integration): audit the tool call mark census across chat, messages, responses, and chaos Adds the /audit cells for the breakpoint census on assistant tool_calls marks: the Responses stream bridge, the OpenAI and Anthropic SDK clients, in-process Pydantic messages, response cache twins, request-level points, a provider 401 on a capped request, malformed provider_specific_fields and tool_call ids, null or empty points, a points update mid-burst, a proxy restart mid-burst, and a worker SIGKILL that picks the worker holding the burst's upstream connections * test(integration): close the SDK clients the cache census cells open --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-03 17:33:48 -07:00
devin-ai-integration[bot]	e9bf2cfd01	refactor(types): replace Any with proven types in 9 files (#44389) ... * refactor(types): replace Any with proven types in 13 files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(types): keep provider error paths for malformed prefetch and poll JSON Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(types): keep main's login body parsing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(types): keep main's Copilot auth and budget alert typing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): audit cells for Any sweep 20261003_2 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): load audit video deployments at proxy start Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(types): keep main's video-edit prefetch handling Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): fix audit chaos Responses cells Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): probe every route after audit worker kill Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-03 21:03:16 +00:00
devin-ai-integration[bot]	fe9b6fd603	refactor(types): replace Any with proven types in 7 files (#43844) ... * refactor(types): replace Any with proven types in 9 files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin xecguard-adjacent guardrail retry, mcp mixed tools, jwt routing and complexity router paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(types): keep only live-provable Any removals Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(types): type cache_hit as bool | None on the sync success path Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-02 02:47:47 -07:00
yuneng-jiang	a93c396a5c	test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128) ... * test(integration): move legacy proxy, router and Redis tests into tests/integration Port 39 legacy tests to the integration tier that owns them, running against the scripted upstream, local Postgres and Redis, test-owned wire peers and owned proxies. Delete 5 legacy tests whose contract is already owned by an existing integration test, and remove the legacy functions, files and helpers left unused. * test(integration): cover recovery of a spent key after its budget is raised	2026-10-01 23:00:54 -07:00
yuneng-jiang	65a316fc92	ci(circleci): test Redis behavior against local Redis and print short tracebacks (#44062) ... * ci(circleci): test Redis behavior against local Redis and print short tracebacks The redis_caching_unit_tests job ran three legacy files against the shared remote Redis. The DualCache and batch-read logic that never needed a server now lives in tests/unit/caching/test_dual_cache.py with a mocked RedisCache, and the behavior that does need one (the increment-with-floor Lua script, read-through, deletes, batch reads) moved to tests/integration, which starts a local Redis. test_returned_settings only read REDIS_PORT and is replaced by a unit test of Router.get_settings CircleCI pytest runs now use --tb=short so failure output stays readable in the test results tab * ci(integration): print short tracebacks from run.py and allow Redis in the sdk shard	2026-10-01 13:00:54 -07:00
yuneng-jiang	bba85f0b6c	chore(lint): remove the LIT002 mutable-construction rule (#43971) ... * chore(lint): remove the LIT002 mutable-construction rule Drop LIT002 from scripts/check_type_discipline.py along with its helpers, its budget entry, its unit tests, and the AGENTS.md and gate docstring mentions. `# mutable-ok` now only suppresses LIT001, so the markers that only existed to silence LIT002 became LIT013 stale suppressions and are removed. The files whose layout depended on those trailing comments are reformatted with ruff format. Every other LIT rule count is unchanged and the ASTs of all touched litellm/ files match main apart from one docstring. * chore(lint): keep the leftover mutable-ok markers for a follow-up Restore the ~1.4k `# mutable-ok` markers stripped in the previous commit so this PR only touches the checker, its tests, the budget, and docs. Those markers no longer suppress anything, so `# mutable-ok` is exempt from LIT013 until a follow-up strips them. * Revert "chore(lint): keep the leftover mutable-ok markers for a follow-up" This reverts commit c35bc0b84e. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-01 12:24:02 -07:00
devin-ai-integration[bot]	6bc17f98d7	refactor: clean up fresh tech debt from 2026-09-29 (#43830) ... * refactor: clean up fresh tech debt from 2026-09-29 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(routing): pin usage-based routing Redis reads through the proxy and SDK Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-30 02:40:37 -07:00
devin-ai-integration[bot]	e450f2d54c	fix(router): serve Responses turns from a sibling when the encrypted content origin has no boundary peer (#43015) ... * test(integration): reproduce encrypted_content_affinity 503 when origin has no boundary peer Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): keep encrypted content affinity turn one out of the response cache Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): degrade encrypted_content_affinity when the origin has no encryption-boundary peer A routed-group candidate that is currently unavailable and shares its (api_base, api_key) with no healthy deployment used to raise a proxy-level 503/429 from _unavailable_origin_error, even though healthy siblings in the same model group could still serve the turn. Strip the encrypted reasoning and dispatch to the healthy pool instead, matching the existing cross-group behavior, and log a warning naming the origin model_id and routed group Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): bound the degraded-affinity log marker and assert the strip on the wire Address review findings: restore num_retries alongside optional_pre_call_checks in the integration test teardown, record scenario request bodies on the scripted upstream so the tests can assert no encrypted reasoning reaches the sibling, and truncate the client-supplied model_id in the degraded-dispatch warning Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tests): only record JSON bodies on the scripted upstream Multipart uploads to scripted POST routes have no JSON body, so gate the observation recording on the request content-type Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): make encrypted_content_affinity runtime-toggleable so /config/update can turn it off --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-09-24 17:31:43 -07:00
devin-ai-integration[bot]	b72a030501	test: take keys out of the legacy proxy, enterprise and mcp unit tests before moving them (#42901) ... * ci: fix the litellm-tests unit job with sysmon coverage, an env allowlist and coverage upload on failure * test: replace key-dependent proxy, enterprise and mcp unit tests with synthetic values and integration and e2e coverage * test: drop key reads at the legacy proxy, enterprise and mcp paths and wire the gemini pass-through split * ci: fail the unit shard when circleci tests split errors * test: drop restating comments from the gemini pass-through split * ci: exit the unit shard cleanly when circleci tests split assigns it no files --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-09-24 14:56:52 -07:00
devin-ai-integration[bot]	3fb6f8740b	test(integration): add read-replica routing harness to the CircleCI integration suite (#42692) ... * test(integration): add read-replica routing harness * refactor(integration): hoist the maintenance url imports * fix(integration): keep per-test databases and the witness sequence readable under replica roles * fix(integration): opt bespoke database and pool tests out of the injected read replica * test(integration): commit recorded replica routing expectations * fix(integration): judge routing by role containment so shrinking role sets do not fail * fix(integration): run the pool-limit shutdown choreography on the superuser database url * ci(integration): add the mcp group to the replica matrix * fix(integration): judge routing by exact role sets with a named either-role allowlist * test(integration): drop containment-era routing expectations for re-recording * chore(integration): drop docstrings from the replica harness scripts * docs(integration): describe exact routing matching and the either-role list * test(integration): record exact replica routing expectations * test(integration): allow the SELECT 1 health probe on either role * test(integration): replace committed routing expectations with an on-demand base-vs-head parity run * test(integration): fix parity env scope, readme wording, and seed-deterministic serialization test * test(integration): make the sorted-role serialization test deterministic in-process * test(integration): swap all product code in parity runs and pin role gains * ci(integration): force tracked-file removal before parity checkout --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-09-24 00:25:03 -07:00
devin-ai-integration[bot]	a286ebf42e	test(integration): regression tests for July provider translation, routing and streaming bugs (#42693) ... * test(integration): optional Anthropic tool properties stay optional on the OpenAI Responses wire (Pylon #6619) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Bedrock InvokeModel count-suffixed cache usage fields are reported and charged (Pylon #6708) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): anthropic messages honors the deployment request timeout (Pylon #6505) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop client_metadata before the Bedrock Converse body reaches the provider (Pylon #6645) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): repeat Bedrock requests under one session name assume the role once (Pylon #6681) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): messages stream keeps include_usage off the Responses wire with always_include_stream_usage (Pylon #6466) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): clamp sub-16 max_tokens to the Responses API floor instead of 400 (Pylon #6539) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): forwarded client x- headers reach the provider on /v1/responses (Pylon #6565) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Anthropic messages stop_sequences reach OpenAI-compatible providers as stop (Pylon #6536) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Codex namespace tools reach a chat upstream flattened and round-trip through /v1/responses (Pylon #6409) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): keep Claude 4.6 legacy thinking budget_tokens on /v1/messages (Pylon #6727) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): nvidia nim ranking keeps image passages and applies top_n without sending top_k (Pylon #6401) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tpm-only model rejects priority traffic once recorded tokens reach the model tpm (Pylon #6344) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): reasoning-only chunks open an Anthropic thinking block at index zero on /v1/messages streams (Pylon #6337) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): file content streams to the client before the upstream finishes sending (Pylon #6315) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): agent whose card lives only at agentCard/v1.0 is reached with bearer auth (Pylon #6249) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): vertex batch create returns a batch when outputInfo is null (Pylon #6374) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): fireworks session id is sent as x-session-affinity and cached tokens land in spend log metadata (Pylon #6220) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): advisor sub-call failure does not cool down the executor deployment (Pylon #6212) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Gemini /v1/messages cache_control creates cachedContent with Anthropic ttl (Pylon #6221) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bedrock_mantle max_output_tokens below 16 is clamped before reaching Mantle (Pylon #6262) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): missing thinking signature 400 on /v1/messages retries without thinking blocks (Pylon #6222) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): format Gemini messages cache_control wire test (Pylon #6221) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): rebuilt shared aiohttp session keeps the configured keepalive timeout (Pylon #6387) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): sagemaker_chat signs the inference component header and sends hf_model_name as the body model (Pylon #6187) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Bedrock Converse DeepSeek drops Anthropic thinking and sends V3 reasoning_effort raw (Pylon #6149) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): concurrent team model TPM requests are reserved before the provider call (Pylon #6075) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): /v1/messages honors the configured timeout against a stalled upstream (Pylon #6025) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Codex additional_tools input items reach Bedrock Mantle as top-level tools (Pylon #6012) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): advisor api_base without api_key never sends the proxy Anthropic key to the caller host (Pylon #6226) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): rerank responses carry call id, latency and cost headers (Pylon #5981) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): parse the outbound Anthropic body with the typed JSON adapter (Pylon #6025) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Bedrock Knowledge Base search forwards userContext to the Retrieve body (Pylon #5991) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Marengo 3.0 text embeddings reach Bedrock nested under inputType (Pylon #5949) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): sub-16 max_tokens over a responses deployment reaches OpenAI as 16 (Pylon #6008) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): midturn system correction reaches the OpenAI Responses wire via /v1/messages (Pylon #6449) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): concurrent requests over a key tpm limit are rejected before reaching the provider (Pylon #5737) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): chat to responses bridge keeps deployment AWS credentials for Bedrock Mantle SigV4 (Pylon #5870) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): vertex gemini stream split across many fragments completes without stalling the proxy (Pylon #5838) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bedrock mantle /v1/messages stream keeps stream true and relays SSE events (Pylon #5596) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): large chat payloads are released from worker memory after the request ends (Pylon #5920) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): streaming success logs v3 rate limit remaining values for callbacks (Pylon #5767) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): a database created search tool backs Anthropic web search interception (Pylon #5669) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): register july provider regression contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): make july provider regression tests deterministic under cache and worker sharing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop order-fragile worker memory probe pending a real retention regression check Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): apply ruff import sorting and formatting Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop stale contract entry and pass question to advisor executor Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): use tiktoken-backed executor model in advisor tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-23 09:51:06 -07:00
devin-ai-integration[bot]	5c0b374f0a	test(integration): regression tests for August provider translation and streaming bugs (#42621) ... * test(integration): Bedrock batch files upload completions and responses records as user messages (Pylon #6882) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): client Anthropic OAuth token never replaces Bedrock SigV4 authorization (Pylon #6888) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bridge /v1/messages and /v1/responses streams through empty-choices chunks (Pylon #6992) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prepend azure content-filter metadata chunk to the messages stream (Pylon #6992) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): fireworks routers/ slug reaches the provider as accounts/fireworks/routers/<id> (Pylon #7030) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bedrock hidden thinking tokens are not reported as text tokens (Pylon #7067) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): azure_ai FLUX.2-flex image generation targets the flex provider path with the BFL body (Pylon #7092) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): keep Databricks streaming usage and cache reads in the client stream and spend log (Pylon #7094) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): openai-compatible image edits forward provider-specific form fields to the backend (Pylon #7122) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): replayed intercepted web search turn reaches Bedrock as text through /v1/messages (Pylon #7181) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): streamed web search turn capped by max_agentic_loops ends the turn with snippets and ordered blocks (Pylon #7230) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bedrock rerank keeps forwarded client headers out of the SigV4 signature (Pylon #7284) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): azure_ai rerank authenticates with an Entra token when no api key is set (Pylon #7303) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): perplexity stream with cost breakdown object completes and bills total_cost (Pylon #7331) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): azure_ai strips Anthropic message fields before the Foundry request (Pylon #7336) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): capped intercepted web search ends the turn without an internal tool_use block (Pylon #7378) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Bedrock passthrough converse-stream keeps event-stream content-type (Pylon #7482) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): v1/messages success exposes v3 priority rate limit headers (Pylon #7532) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): config deployment dropped by a stale boot cost map is restored after reload (Pylon #7564) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Bedrock Mantle project id reaches the provider as anthropic-workspace-id (Pylon #7583) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prefixed Opus 4.8 reasoning_effort reaches Bedrock as adaptive thinking (Pylon #7586) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): dashscope chat forwards reasoning_effort to the provider (Pylon #7606) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): failing stream logging callback still releases the max_parallel_requests slot (Pylon #7608) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): gen 5 Claude Bedrock Invoke tool search sends the Bedrock beta field (Pylon #7642) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): databricks ai gateway api_base requests OAuth token from workspace origin (Pylon #7724) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): deepseek vision image content list reaches the provider unchanged (Pylon #7729) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Bedrock Mantle context overflow surfaces as 400 prompt is too long (Pylon #7732) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): Codex history items reach Bedrock Mantle as supported Responses input types (Pylon #7783) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): chat over responses deployment returns finish_reason length when output tokens run out (Pylon #7784) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bedrock_mantle rewrites Codex history items before the Responses wire (Pylon #7812) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): advisor sub-call on /v1/messages uses the configured advisor deployment (Pylon #7828) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tencent thinking reaches the provider body instead of failing the request (Pylon #7834) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): xAI chat web search reaches /v1/responses with instructions and nested filters (Pylon #7835) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): send Bedrock Converse config blocks once at top level (Pylon #7839) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bedrock converse sends gpt-5 reasoning_effort as reasoning.effort (Pylon #7850) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bedrock cohere.embed-english-v3 embeddings accept encoding_format and dimensions (Pylon #7963) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): streamed chat completions emit SSE keepalive pings while the upstream is silent before its first token (Pylon #7987) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(integration): format the TTFT keepalive regression test (Pylon #7987) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): openai chat drops tool_choice when the request has no tools (Pylon #8022) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): stream whose first chunk has no choices falls back and bills the fallback (Pylon #8006) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Revert "test(integration): Bedrock Mantle project id reaches the provider as anthropic-workspace-id (Pylon #7583)" This reverts commit 864b65811f. * test(integration): reconcile contracts manifest for bundled regression tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): isolate proxy config writes in bundled regression tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): address review feedback on keepalive, cost map reload and websearch order Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-22 20:52:00 -07:00
Yuneng Jiang	7679e42736	test: cover provider wire contracts, streaming and recovery ... Adds owned local TCP peers, a Redis process helper and SigV4 helpers to tests/integration, and integration contracts for Anthropic and Bedrock wire shapes, Bedrock role configuration, S3 wire, observed routing, Redis recovery and stream contracts. Consolidates the four commits previously stacked on litellm_integration_accounting onto its main-merged tip	2026-09-16 12:48:30 -07:00