mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-19 04:54:37 +00:00
17 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yucheng-berri
|
9f115be88d
|
fix(bedrock)!: scan Claude invoke-with-response-stream passthrough output with post_call guardrails (#46524)
* fix(bedrock): scan Claude invoke-with-response-stream passthrough frames with post_call guardrails (LIT-9618) Invoke streams of Claude models now go through the same frame-by-frame relay as converse-stream. Each chunk frame's Anthropic event is scanned by the /v1/messages stream translation, a block ends the stream with an AWS exception frame, and a held stream gets its masks written back into re-encoded chunk frames. Claude is told by its model id or, behind an application inference profile, by the anthropic_version its request body carries. Streams of other model families stay as they were * test(bedrock): cover invoke stream vendor outage, upstream exception and release on scan (LIT-9618) * fix(bedrock): hold Claude invoke streams for guardrails the relay cannot run and mark each scan as streaming (LIT-9618) A guardrail released on scan or run in parallel left a Claude invoke-with-response-stream unscanned, while converse-stream holds the whole stream and scans it once. Claude invoke streams now take the same buffered path, with the rewrites written back into the chunk frames that carried each text. Each invoke scan also sends stream true and the events relayed so far as responses, as /v1/messages does, so a vendor can tie the scans of one streamed reply together. * refactor(bedrock): mark the request dicts the buffered stream paths hand to post_call hooks (LIT-9618) * fix(bedrock): scan a Claude invoke stream the upstream breaks off as a finished stream (LIT-9618) An invoke stream that ended on an upstream exception, without a message_delta stop_reason, was scanned as still in flight, so its tool input never reached the guardrail and a held stream released it unscanned. The scan and its scan key now close such a stream the way converse-stream treats the end of its upstream. * test(bedrock): pass the converse-stream endpoint where the tool call without input test frames its stream (LIT-9618) * test(bedrock): expect the ValidationException main now returns when a guardrail blocks a Claude invoke stream before its first frame (LIT-9618) * test(bedrock): type the Claude invoke event helpers in the passthrough guardrail tests * test(bedrock): type the remaining parameters of the Claude invoke stream tests * refactor(bedrock): bind the buffered invoke rewrite in its condition --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
ef1cb19351
|
fix(bedrock)!: stream Converse passthrough frames through post_call guardrails instead of buffering (#46307)
* test(integration): Converse stream post_call guardrail relays frames as they arrive (LIT-9515) * fix(bedrock): stream Converse passthrough frames through post_call guardrails instead of buffering (LIT-9515) A post_call guardrail that scans through apply_guardrail now runs on /bedrock/.../converse-stream the way it runs on /v1/messages: frames reach the caller as the upstream sends them, every fifth frame and the end of the stream are scanned, and a block after the first frame ends the stream with an AWS exception frame. streaming_buffer_until_moderated still holds the stream and applies masks per content block. Guardrails that need the whole response (own async_post_call_success_hook, mask_response_content, native hooks, policy pipelines) keep the buffered path. * fix(bedrock): apply held Converse stream masks when the upstream ends with an exception frame (LIT-9515) * fix(proxy): log a relayed passthrough stream's spend before its mid-stream failure hook (LIT-9515) * fix(bedrock): close and report every way a relayed Converse stream ends (LIT-9515) - The framer ends the stream with an empty item once the upstream closes, so held masks are applied whether the upstream ends with messageStop, an exception or error frame, or a dropped connection - A guardrail HTTPException after the first frame reaches the relay, which logs the stream's spend, runs the failure hook and ends the stream with the exception frame, instead of ending inside the guardrail with no failure row - A failure before the first frame waits for the stream's spend to be logged, as a mid-stream failure does - Closing the response before its body starts closes the guardrail layers and the upstream - Each stacked guardrail layer gets its own copy of every frame, so an earlier layer's end-of-stream rewrite never reaches frames a later layer still holds - Deltas whose text ends in half a surrogate pair are decoded with the standard JSON parser * fix(bedrock): relay a Converse frame whose payload is not UTF-8 instead of ending the stream (LIT-9515) * test(integration): cover Converse stream relay failure, end-of-stream-only, stacked, disconnect and burst cells (LIT-9515) * fix(bedrock): estimate prompt tokens for a passthrough stream cut before Bedrock reports usage (LIT-9515) Bedrock sends usage only in the last frame of a stream, so a stream ended by a guardrail block or a client disconnect logged 0 prompt tokens while AWS still charges for the prompt. The stream collector now estimates them from the request the way chat completions streams do * fix(bedrock): read tool result blocks without recursion in the passthrough prompt estimate (LIT-9515) * fix(passthrough): keep the spend flush running when the wait for it is cancelled (LIT-9515) * fix(bedrock): keep a converse-stream buffered when its guardrail releases a held stream on scan (LIT-9515) A held post_call guardrail with streaming_buffer_release_on_scan releases each window as received once its scan passes, so the relay delivered the original text instead of the guardrail's masks. Such a stream keeps the buffered path, which applies masks per content block as on main. * fix(bedrock): keep a converse-stream buffered when a post_call guardrail runs in parallel (LIT-9515) The relay ran every guardrail in config order, so a run_in_parallel guardrail listed before a masking guardrail scanned the unmasked text. Main's buffered path runs parallel guardrails after the sequential ones and discards their rewrites, which the relay cannot reproduce, so these streams stay buffered * fix(bedrock): mark Noma v2 relay scans like chat and stop reading Bedrock when a held stream's client leaves - Noma v2 scans on the converse-stream relay carry stream, and the last one carries the finished Converse response with its stopReason, as chat sends them, so Noma keeps one record per reply. Other guardrails get the request unchanged. When the last sampled scan already covered the whole answer, the end-of-stream scan still runs for Noma so it gets the finished answer - A held relay stream whose client leaves before the first frame stops reading Bedrock and returns 499, like chat. When Bedrock sent nothing yet, the budget reservation and the parallel-request slot are released - A cancelled wait for the first streamed chunk now stops the upstream read instead of leaving it running - A Noma v2 scan cancelled before Noma answers records not_run instead of success with an empty response * fix(bedrock): send Noma the finished Converse reply's tool calls and reasoning as Converse blocks * fix(bedrock): give Noma the ended scan of a Converse reply that only calls a tool without input * test(bedrock): expect the ValidationException main now returns for a held block before the first frame Main's AWS-shaped passthrough errors (#46522) answer a guardrail block before the first frame with x-amzn-errortype ValidationException, so boto3 reads the code and the guardrail message instead of a bare 400. * fix(bedrock): keep held masks when an earlier sampled guardrail ends the Converse stream with its message A sampled guardrail that blocks with disable_exception_on_block ends the stream with its message as raw event-stream bytes. A held guardrail after it on the same key only read frame objects, so it never saw the messageStop, skipped its write-back and released the held text without its masks. The Converse handler now reads those bytes back into frames, so the held guardrail sees the end of the message and masks the text * fix(proxy): free the parallel slot when a caller leaves a Converse relay holding part of the first frame The relay skipped releasing the slot and budget reservation whenever a spend flush was scheduled, but a flush of a partial first frame builds no response and logs nothing, so the slot stayed held until it expired. The flush now says whether it logged the stream. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
jesus-berri
|
7f603da9f1
|
fix(openai_like)!: surface provider response headers on non-streaming chat completions (#46928)
* fix(openai_like): surface provider response headers on non-streaming chat completions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(openai_like): mark the header test locals Final * fix(proxy): leave out response header values HTTP cannot carry Non-streaming OpenAI-like chat now forwards every provider header, so on the httpx transport a provider header value outside latin-1 made the proxy answer 500 after the provider had served the call. get_custom_headers now leaves such a value out and keeps the other headers, which also ends the same 500 on streaming Adds integration cells for the provider request id on non-streaming chat and the Responses bridge, LiteLLM headers an upstream cannot overwrite, spend parity under an upstream cost header, the latin-1 case, and the logging callback payload * fix(cost): ignore an upstream response cost header that is negative or not a number * test(integration): cover invalid upstream cost headers and non-latin-1 headers on every endpoint * fix(prometheus): keep deployment metrics when a provider rate-limit header is not a number * test(prometheus): type the rate-limit header test and drop its clock reads * test(integration): cover deployment metrics under an unparseable provider rate-limit header --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
38f3f8a1a9
|
fix(proxy): cooldown errors carry 429, Sentry starts at boot without request bodies, tracing failures log at ERROR, exporter counters exported (#46344)
* fix(router): carry a 429 status on cooldown and no-deployment errors RouterRateLimitError and RouterRateLimitErrorBasic had no status_code, so the proxy answered an all-deployments-cooled-down request as a generic 500 class error and logged the bug report notice for every cooldown. Both now carry status_code 429, which the proxy maps to a rate_limit_error and which the retry policy already treats as retryable * fix(proxy): start Sentry at config load and never attach request bodies Sentry only came up inside set_callbacks on the first LLM call, so boot and background errors were never reported, and general_settings.alerting: [sentry] always raised Missing SENTRY_DSN because the alerting path read a dead litellm.utils pin instead of the live SDK. The proxy now initializes Sentry once while loading the config whenever sentry is a configured callback or alerting target, the alerting and failure handlers read the live instance, and max_request_body_size is pinned to never so prompts and completions stay out of Sentry events * fix(tracing): log Lens failures at ERROR and export row counters to Prometheus A tracing store that fails to start, or a Lens service that rejects a batch, was only logged at WARNING, so operators running at the default ERROR level never saw tracing silently go dark. Both now log at ERROR and name the lens store. The exporter's rows_written and rows_dropped tallies were never exported, so the two counts are now Prometheus counters litellm_lens_export_rows_written_total and litellm_lens_export_rows_dropped_total, registered lazily on the default registry like the other proxy-level counters and charted on the all-metrics Grafana dashboard * test(proxy): drop the litellm import the alerting tests no longer use * fix(proxy): start Sentry once a DSN is set, from any config source Boot-time init used to lock Sentry in even with no SENTRY_DSN, so a DSN loaded later from DB environment_variables was never picked up. initialize_sentry now skips when no DSN is set and re-inits when the DSN changes. alerting_handler calls it too, so alerting: [sentry] set through the DB works. Drops the litellm_settings.callbacks check since that config fails to load with sentry in it * fix(sentry): keep stack frame locals out of events unless PII is opted in Frame locals carried prompts (messages, request bodies, provider options) into every error event. include_local_variables now follows SENTRY_SEND_DEFAULT_PII * test(router): drop the test that only pinned the 429 class attribute * fix(proxy): start Sentry at boot when it is only an input callback * fix(proxy): keep Sentry init and alerting within the strict ruff and basedpyright ceilings --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai> |
||
|
Kerry Lu
|
11e9fcc033
|
fix(cost): price bridged /v1/messages reasoning tokens from the chat completion, not the served Anthropic body (#46726)
* fix(cost): price bridged /v1/messages reasoning tokens from the chat completion, not the served Anthropic body * fix(cost): keep the type gate flat for the bridged /v1/messages billing path --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
17f621e116
|
fix(vertex_ai): route Model Garden models through the generateContent adapter (#41836)
* fix(vertex_ai): route Model Garden models through the generateContent adapter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(vertex_ai): annotate generateContent route test parameters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(vertex_ai): type litellm_params in get_vertex_ai_model_route * fix(vertex_ai): bill and log Model Garden generateContent calls like chat completions Streamed Model Garden calls through the generateContent adapter now request stream usage where the deployment supports it, keep cached-token details, and look up the provider-relative cost key, so they bill like the non-stream call. The adapter forwards router metadata, passes upstream errors through with their status, countTokens falls back to local counting for non-Gemini routes, and the native Gemini config is limited to the Gemini routes. * fix(google_genai): keep adapter stream usage, admin opt-out, single failure count, and base_model routing Streamed generateContent through the chat adapter now ends with the upstream usage, honors always_include_stream_usage false and the caller's stream_options, counts one failure per failed call, and routes Vertex deployments by their base_model like chat does * fix(google_genai): ask for stream usage on every adapter stream so Claude streams end with usage * test(google_genai): expect the trailing usage chunk on bridged OpenAI streams The OpenAI bridge test now sends the usage chunk OpenAI returns for include_usage and expects it translated into usageMetadata. The response_format support helper goes back to its single-purpose form since its only other caller is gone. * fix(google_genai): end adapter streams with one STOP event carrying usage, and keep upstream usage after the finish chunk * fix(google_genai): send the held STOP event when no usage arrives, keep a caller's stream_options, and route countTokens like generateContent * test(integration): cover the moved Vertex routes, adapter stream usage, the shared iterator and usage settings over the wire * test(integration): add cache-hit twins for the stream usage setting and non-stream cells for every iterator member --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
2d4472efc1
|
fix(vllm): write spend log rows for /vllm passthrough calls to router models (#45733)
* fix(vllm): write spend log rows for /vllm passthrough calls to router models The vLLM passthrough config never built a logged response for relayed calls, so stream and non-stream calls through /vllm with a router model wrote no spend row and never charged the key. vLLM now shares the OpenAI-shaped relay logging that Azure already had, moved into an openai_like helper both configs call. * ci: run the vllm passthrough unit tests in the llms-providers shard * fix(proxy): log /vllm text completions and ask router models for stream usage A chat stream relayed to a router model without stream_options now gets include_usage added on the way out, and the usage chunk vLLM appends is removed before the client sees it, so the spend row carries vLLM's own token counts instead of a local estimate. /vllm/completions calls now write a row too, stream and non-stream. * fix(vllm): keep usage, fingerprints, and disconnect rows correct on /vllm streams Inject and strip stream usage on /vllm/completions too, hold every finish frame for the fingerprint when n > 1, pass a forced-usage server's stream through untouched, stop turning continuous_usage_stats alone into per-chunk usage, close the upstream when the client hangs up so the row is written, and assemble text choices with one sort instead of a scan per choice * fix(vllm): keep n>1 stream order when holding finish chunks for the fingerprint A held finish chunk now goes out as soon as any other chunk arrives, so a choice that finishes while another still streams keeps its place. Only the finish chunks right before the usage chunk get vLLM's system_fingerprint. * fix(vllm): keep caller stream_options and log both success handlers Forward a stream_options the proxy cannot read, or a non-boolean include_usage, to vLLM as the caller sent it. Only a false always_include_stream_usage changes the /vllm relay, so clients get the same stream bytes under every setting. Record the passthrough origin when logging starts so the sync and async success handlers both parse the relayed body, hand callbacks a parsed text completion, and read relayed messages and prompt separately so a stray field no longer drops the local count. Add the audit's integration tests for the relay, the settings, and the Azure spend row. * test(vllm): expect the caller's stream under always_include_stream_usage true * fix(vllm): relay long SSE frames in linear time * test(vllm): time the relay's usage and finish_reason prefilter on one large frame --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Rick
|
4f3cb48dc1
|
feat(dashscope): add Wan 3.0, Wan 2.7 and HappyHorse video generation via /v1/videos (#43706)
* feat(dashscope): add Wan 3.0, Wan 2.7 and HappyHorse video generation via /v1/videos
DashScopeVideoConfig maps litellm's OpenAI Videos API onto Alibaba Model
Studio's async task API: create hits
/api/v1/services/aigc/video-generation/video-synthesis and status and content
poll /api/v1/tasks/{task_id}. It is registered for dashscope and for the
qwencloud and qwen_ai_platform brand aliases, following the same
get_dashscope_family_*_config pattern image generation uses. Unlike the
Sora-shaped providers the request body is nested, so prompt and media go under
input and everything else goes under parameters. X-DashScope-Async is always
sent, because DashScope rejects the call outright without it
OpenAI params translate to DashScope's contract. seconds becomes duration, and
size yields both a ratio and a named resolution tier (1280x720 gives 16:9 plus
720P), since sending only the ratio leaves the request on the default 1080P and
silently bills at that rate. input_reference becomes a single media entry,
base64 encoded into a data URI for file inputs. Which media type a model takes
(first_frame or, on the reference-to-video models, reference_image) is read
from provider_specific_entry.dashscope_video_reference_input via
get_model_info rather than hardcoded; models without it predate the media
array and get the flat img_url field. A caller-supplied media array always
wins, as that is the only way to express multi-role combinations
The create call is the only billed call, so it bills what DashScope will
deliver: the requested duration and tier, or the documented 5 second and 1080P
defaults when they are omitted. Smart duration (duration -1) lets the service
pick the length after that call, so it is rejected rather than recorded at
zero, with a 400. Task creation and task lookup both report failures as a 200 carrying
top-level code and message with no output, so that shape is raised instead of
surfacing as a queued video with an empty id. Status responses re-encode the
deployment model from the polled id, so follow-up polls and downloads route to
the same deployment. Timestamps parse as UTC+8. DashScope publishes no list,
delete or remix endpoint, so those raise rather than guess at a URL
The end-to-end tests drive the real avideo_generation() entrypoint through an
httpx MockTransport and assert on the JSON body httpx actually encoded. That is
what catches a mapper that drops input_reference before the request is built,
or a body fragment json cannot serialize, neither of which a test calling the
transform directly would see
Both model_prices maps gain wan3.0-video, wan3.0-video-prime, wan2.7-t2v,
wan2.7-i2v, wan2.7-r2v and the HappyHorse 1.0 and 1.1 t2v, i2v and r2v models,
for dashscope and the qwencloud and qwen_ai_platform aliases, priced per output
second by resolution tier from Model Studio's USD rate cards. dashscope and
qwen_ai_platform default to the Beijing host and use its rate card; qwencloud
defaults to the international host and uses Singapore's. The three
*_API_BASE_VIDEO overrides are documented in BerriAI/litellm-docs#1920
Co-authored-by: AaronHowell <237895480@qq.com>
* fix(dashscope): fail loudly on video responses that cannot be read
Addresses the review on the video generation PR. A body that is not the
documented task shape, an output block missing task_id or task_status, an
undocumented task_status and a missing output all raise DashScopeVideoError
now, instead of defaulting into a queued video with an empty id that callers
poll forever. A JSONDecodeError from a gateway response no longer escapes as
a generic connection error.
task_id and task_status are required on _TaskOutput, and a missing output
raises even when the body carries no code or message.
Download failures move into the 4xx range, since a 5xx has the proxy retry
them: 409 while the task is still running, 400 for a task that failed, 404
for an id that expired. The status a provider reported is kept when it
reported one, so an in-body failure that arrives on a 200 is billed as the
400 it is rather than as a retryable connection error.
characters and image are only dropped when drop_params is set; the params
that are rebuilt into the body or consumed by the caller (model, prompt,
user, extra_*) stay silently skipped, since the proxy stamps user from the
caller's end-user header.
Co-authored-by: AaronHowell <237895480@qq.com>
* fix(dashscope): clear the lint gates after merging main
* fix(dashscope): accept file tuples and paths as video input_reference
video_generation() types input_reference as FileTypes, which includes
(filename, content[, content_type[, headers]]) tuples and pathlib paths,
but the DashScope encoder only read bytes and file objects, so those
image-to-video calls raised before reaching DashScope. Bare strings stay
URLs and are never opened as paths.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(dashscope): snap size to a supported ratio and read file tuples and paths
* test(dashscope): assert whole mapped values and bill against hand-written rates
The tier pricing test read its expected rate from the same map the cost
calculator uses, so a wrong rate passed. Billing is now checked against
fixed rates driven through the status response usage, while the map test
only asserts every supported tier has its own rate. Size mapping tests
compare the full mapped dict, and the single-test wiring class is a
plain function.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(videos): download content through the caller's client and stop retrying dead tasks
Two Cursor Bugbot findings on the DashScope video PR.
The DashScope artifact lives on OSS, so transform_video_content_response makes a
second request. It built its own cached client, discarding an injected client and
its ssl_verify. Thread the client through the content response the way
transform_video_status_retrieve_response already does, and use it in the three
providers that fetch from a second URL (DashScope, RunwayML, fal.ai). The
remaining providers accept it and read the response body, matching how they
already accept an unused client on status retrieve.
A FAILED or CANCELED task reported without a code or message fell past the
in-flight check into the 502 branch, so the proxy retried a task that can never
produce a video. Raise 400 for a terminal status instead.
* fix(dashscope): answer 404, not 409, when a Wan video is downloaded before it finishes
A download that lands while the task is still PENDING or RUNNING answered 409. That
reads as a client error, which was the intent, but litellm's retry predicate counts 409
as retryable, so the proxy polled DashScope twice more before giving up and then counted
the caller's own early download against the model's error rate
404 is what the OpenAI videos API documents for content that is not ready, and it is the
one status in that family the retry predicate leaves alone. The rest of the mapping is
unchanged: 400 for a task the provider reported as failed, 404 for an expired task id,
502 when a succeeded task carries no artifact
The download tests now drive avideo_content and assert the caller sees NotFoundError
rather than an internal server error
* fix(dashscope): route a video poll back to the model group that created the task
The create response encoded the upstream model name into the returned id. Both the
dashscope and qwen_ai_platform prefixes serve the same model names from their own hosts,
so a proxy with a deployment per region had deployments that only the model group told
apart. Encoding the bare model name resolved a poll to whichever deployment the router
lists first, which sent it to the other region's host with that region's key and a task
id that region never issued. That is the reroute the earlier proof records as the
dashscope and qwen_ai_platform poll ids coming back tagged qwencloud
The id now carries the model group the create was routed to, which is what the proxy
resolves a poll back to, and falls back to the model name for a direct SDK call that has
no router
The regression test drives a real router with one deployment per region, under a
different prefix and under the same one, and asserts the poll carries the key of the
region that created the task
* fix(dashscope): read the routed model group from litellm_metadata
The router stamps the model group it picked into litellm_metadata. metadata only
carries a copy when the caller sent no metadata of its own, and every proxy request
sends some, so a proxied create still encoded the bare model name and the poll
resolved to whichever region's deployment the router lists first
The regression test now also creates the video with caller metadata, which fails
without this change
* fix(dashscope): reject unbillable video requests and keep fal and runway on their own clients
* fix(dashscope): show the caller why a video task is gone or not ready instead of the bare upstream body
* fix(dashscope): check video request values before the billed create and honor per-call drop_params
Type check documented DashScope parameters so a bad seed answers 400 instead of a billed failed task,
pass the call's and deployment's drop_params into the video mapping, answer 400 for an expired task id so
the deployment is not cooled down, read DASHSCOPE_API_BASE for video, keep polling SUSPENDED tasks, and
refuse to map size onto models litellm has no price for.
* fix(dashscope): carry the requested model name in video ids so alias and wildcard keys can poll
A key scoped to an alias or to a name a wildcard serves could create a
video and then got a 403 on the poll and the download, because the id
carried the group that served the call (wan3, qwencloud/*) and the key
was never given that name. The id now carries the name the router was
asked for on that hop, and the router resolves an alias, routing group
or wildcard name in an id back to itself when no deployment name or
provider model matches it first
* fix(dashscope): keep a routing group's video polls on the member that created the task
A routing group spans several model groups, so an id carrying the group's name
load-balanced its polls, and some left for another region's host and key with a
task id that region never issued. The id now carries the member that served the
task when the request named a routing group, and keeps the requested name for a
model group, an alias, or a wildcard match.
* fix(dashscope): carry the requested and served model in video ids so group-scoped keys poll in the creating region
* fix(proxy): pin a video's served group only after key and team aliases resolve
* test(integration): audit DashScope video create, poll, download, routing and chaos paths
* fix(proxy): keep catch-all video polls on their provider and count served polls against the requested group
* test(routing): expect a forged served group to be dropped from the polled video id
* test(integration): cover rate-limited routing group polls and catch-all polls of DashScope videos
---------
Co-authored-by: AaronHowell <237895480@qq.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
devin-ai-integration[bot]
|
31e2ce1427
|
feat: forward-port dd user_email span tag and model hub description from internal staging (#42749)
* feat(dd_span_tagger): emit litellm_user_email span tag for JWT-authenticated requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> (cherry picked from commit |
||
|
devin-ai-integration[bot]
|
5325c4a236
|
feat(proxy): per-model team member budgets with router fallback (#43887)
Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Shivam Rawat
|
15154a40a8
|
fix(proxy): let admins make per-model key/team rate limits a hard 429 instead of falling back (#45459)
* fix(proxy): let admins make per-model key/team rate limits a hard 429 instead of falling back Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): sync per-model rate limit API types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): type per-model fallback setting access Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep the per-model hard cap inside the fallback loop and test it through the real limiter * fix(proxy): list the per-model hard cap setting on the settings page and cover every per-model descriptor * fix(proxy): read disable_fallbacks_on_per_model_rate_limits as a boolean value and cover the team-wide per-model cap * test(proxy): cover per-model rate limit 429s skipping fallbacks on a live proxy --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
141dba2372
|
fix(proxy): keep the mapped status on assistants and threads route errors (#44994)
* fix(proxy): keep the mapped status on assistants and threads route errors * test(proxy): type and annotate the assistants and threads route error tests * fix(proxy): redact internal details from assistants and threads route error messages * test(proxy): integration cells for assistants route error mapping * test(proxy): drop malformed JSON cases that never reach the assistants route handler --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
ee47da334d
|
test(proxy): inject the clock into the v1 parallel request limiter and pin it in its tests (#45522)
Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
Mateo Wang
|
58258409c9
|
refactor(proxy): expose public names for private proxy helpers (#45170)
* refactor(proxy): expose public names for private proxy helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep original class names behind public aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve internal callback filtering Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep _PROXY_ class names for managed files hooks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names in package exports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve recursive auth helper name Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): align MCP limiter tests with server enforcement Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): match main's MCP limiter tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names bound in importing modules Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve compatibility imports through strict lint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): use exact pyright suppression in password helper test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): add reasons to compatibility import noqa comments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): update IN-list baseline for renamed helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c3a23fe499
|
refactor: expose core private helpers under public names (#44871)
* refactor: expose core private symbols with compatibility aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * types: narrow core migration diagnostics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve runtime behavior in core symbol migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve core private usage migration behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve private value rebinding compatibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore optional imports and cover public helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: exempt router property from call coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: align recursive detector ignore names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
7a7d27c550
|
fix(guardrails): run the end-of-stream post_call scan when the client disconnects mid-stream (#43839)
* fix(guardrails): run end-of-stream post_call scan when the client disconnects mid-stream Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): close the guardrail stream chain in async_data_generator on client disconnect Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): leave the raw upstream response to the shielded finalizer on client disconnect Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep disconnect cleanup going when a streaming callback cleanup raises Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): assert the refund through a recorder instead of the mock Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): inspect tool calls released before a disconnect under incremental_diff and record a failed scan marker The incremental_diff transform stream now scans tool calls it already released when the client disconnects, and a disconnect scan whose translation raises after the guardrail recorded success also records guardrail_failed_to_respond Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): pin that a text-only disconnect scan is not handed a tool_calls finish Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): cover disconnect scans on every streaming endpoint and client, plus outage, worker-kill and cache-hit cells Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): prove the cache-hit twin is served from the cache Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): type the disconnect-close streams so basedpyright stops reporting unknown arguments * fix(guardrails): give the guardrail metadata cast a reason so the type discipline gate accepts it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): scan released Messages and Responses tool calls on disconnect Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(guardrails): format the disconnect scan unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(guardrails): drop mutable-ok markers that no longer suppress a rule Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): scan released Responses output after a finished item and end only in-flight Chat choices Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): type the disconnect scan test helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(guardrails): type request_data in the disconnect scan helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): type request_data in the disconnect scan test doubles Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): pin that chat streams with no tool call in flight end as released Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): scan only released chunks on disconnect and skip it once a block owns the verdict The disconnect scan now uses the chunks actually yielded to the client, copies them before scanning, skips when a mid-stream block or HTTP error already settled the verdict, and the iterator wrapper only closes hooks that are async generators so plain async iterator hooks keep working Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): pin that a delivered guardrail error or final chunk settles the disconnect verdict Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): close any hook iterator that exposes aclose when the stream ends early Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): accept a synchronous aclose on custom streaming hook iterators Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): swallow callback aclose errors at end of stream A custom callback whose async_post_call_streaming_iterator_hook returns a non-generator async iterator with a raising aclose() failed the finished stream: content plus usage reached the client and then the stream surfaced an error SSE with no [DONE], or aborted a post_call pipeline's buffering loop into a 500 with an empty body. Wrap the aclose invocation in _wrap_streaming_iterator_with_enrichment in try/except and log a warning naming the callback and the cleanup error, matching close_guarded_stream and _close_guarded_layers. Iteration-time hook exceptions still propagate. * fix(proxy): log only the error type when a callback aclose raises Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
a76b59db9f
|
test(proxy): move middleware, spend_tracking, pass_through, common_utils and root proxy tests into tests/unit/proxy (#44015)
Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
Renamed from tests/test_litellm/proxy/test_common_request_processing.py (Browse further)