* fix(router): schedule default_priority requests once and reject a non-integer priority
A router configured with default_priority failed every request that carried no
priority with a 500 naming a missing 'priority' argument, and an explicit
priority re-entered acompletion and hit the same error. acompletion now resolves
the priority once (the request's integer, else the default, with 0 kept as a
real priority) and schedules through _schedule_factory directly, while
schedule_acompletion delegates to acompletion so the two paths share one queue
pass. A non-integer priority answers 400 naming the param, or is dropped under
drop_params, instead of leaking to the provider or crashing the scheduler.
* fix(router): honor router-level drop_params and schedule prompt-management models under a priority
* fix(router): drop a non-integer priority when the model's deployments set drop_params
* fix(router): resolve the drop_params deployment scope for team routes and deployment ids
* fix(router): read the request and router drop_params flags without a loop-assigned Final
* refactor(router): type the prompt-management wrapper's return and mock the provider in its test
* perf(router): consult drop_params for the priority only when it is not an integer
* test(integration): assert the new priority contract for non-integer and zero priorities
* docs(router): describe default_priority as the priority every request without one takes
* fix(router): parse default_priority once and apply it to text completions and the queue route
A digit string default is taken and any other non-integer default is ignored with a warning instead of failing every request. atext_completion and /queue/chat/completions resolve the priority the way acompletion does: a request without one takes the default, an integer queues once, and a non-integer answers 400 or takes the default under drop_params. Integration cells cover every default-priority surface on an owned two-instance rig.
* fix(router): type the priority dispatch so the lint budgets hold
* fix(router): read priority like the scheduler did, require a priority on the queue route, and keep litellm's drop_params order
* fix(router): drop a priority named in additional_drop_params and report the queued priority on streamed queue responses
* fix(router): serve deployment-id requests without waiting behind the queue and ignore a boolean default_priority
A request whose model is a deployment id now counts that deployment as healthy in the
scheduler, since routing by id already ignores cooldown. Before, the scheduler found no
healthy deployment for an id, so the request waited behind every entry ahead of it.
A boolean default_priority (true, or YAML off) is ignored at startup with a warning,
like any other value that is not a whole number.
The scheduler integration tests now share worker state for the defaulted pair, cool a
group through /v1/responses so no stale in-memory queue is left behind, and use "high"
instead of "1" for the non-integer cases. The concurrent burst test no longer asserts
that the queue drains, matching the explicit-priority burst test, since the scheduler's
read-modify-write queue can race across workers.
* test(router): keep the queue drain check as a BUG-skipped integration test
* fix(router): queue default_priority chat requests once instead of failing them
* test(router): open with the default_priority regression tests alone
* fix(router): honor deployment drop_params for a malformed priority on a request routed by provider model name
* fix(proxy): end a chat completions stream with an error event when a guardrail blocks mid-stream
A streaming guardrail that raised an HTTPException after the first chunk re-raised it inside the SSE generator, so the 200 body was cut off with no finish_reason, no error event and no [DONE]. Build the error frame through sse_error_payload instead, keeping the status, message and structured detail, and encode values JSON cannot represent as strings so the frame always goes out.
* fix(proxy): keep early guardrail blocks on their HTTP status and end mid-stream errors with [DONE]
A guardrail block before the first chunk is raised again, so its response stays the
400 event-stream body it is on main. A mid-stream error event now carries the failure
hook's replacement error when one is returned, is followed by data: [DONE] on
OpenAI-format streams, and leaves the Gemini route's raise unchanged. Error details
JSON cannot encode are written as strings instead of breaking the event.
* fix(proxy): keep a per-chunk guardrail block on the first chunk on its event-stream 400
* fix(proxy): treat a keepalive ping as a started stream so a later guardrail block ends in-band
* fix(proxy): encode non-JSON guardrail details in early stream error frames
An early guardrail block whose detail holds a value JSON cannot encode sent
400 headers and then cut the body. The error frame now writes such a value as
its string form, the same as the mid-stream event. The keepalive test now runs
on a virtual clock instead of real sleeps.
* fix(proxy): show an in-band stream error event as an error in the CLI, Python client, and dashboard playgrounds
ChatClient.completions_stream raises StreamError on a data frame carrying an error, so litellm-proxy chat prints the block reason and keeps the partial reply out of history. The prompt playground and the Interactions playground read the same frame through a shared streamErrorMessage helper. Adds the audit's integration cells for chat stream error events, and makes the endpoint tests' failure-hook mocks return None so they check the real error payload.
* fix(proxy): print stream error text verbatim in the CLI and read split error events in the prompt playground
The CLI printed the server's error message as Rich markup, so a closing tag like [/pii] in a guardrail message crashed lite chat. The prompt playground now buffers partial lines across reads, ChatClient raises only on a non-empty error, StreamError redacts keys like the other client exceptions, and StreamError is exported from litellm.proxy.client
* fix(proxy): print CLI stream errors without Rich highlighting or emoji codes
* test(proxy): cover the CLI's plain-text 400 error print
* test(proxy): annotate the new client stream test locals as Final
* fix(proxy): send the failure hook replacement on Responses streams and before the first chunk
* fix(proxy): end a Responses stream with response.failed when it fails on a built-in exception
* fix(proxy): keep a failure hook replacement before the first chunk in-band so bare stream routes still answer it
A non-HTTP failure whose hook returns an HTTP error was raised before the first
chunk, which create_response turned into the hook's status but which a bare
StreamingResponse route such as /queue/chat/completions answered as a 200 with
zero bytes. Only the original HTTP errors raise now. The replacement goes out as
the stream's error event, which create_response still answers with the hook's
status as a JSON error.
Adds audit round 2's integration cells for failure hook replacements on chat and
Responses streams, built-in exceptions on Responses streams, and the Python client.
* test(proxy): cover a queue stream whose failure hook replaces a failure before the first chunk
The audit cell posts to /queue/chat/completions with a callback that raises before the first chunk and a failure hook that answers 403. It checks for a 200 event stream whose only event is the replacement, then one [DONE], with the callback's own message absent
* fix(proxy): streaming responses hand the event loop back every 50 ms instead of holding it for the whole buffered stream
* fix(proxy): yield to the event loop on every streamed chunk, no hold budget or clock
* docs(proxy): say why the streaming loop yields on every chunk
* fix(proxy): streaming loop yields once it has held the event loop 50 ms, clock read inline
* docs(proxy): streaming loop comment names the 50 ms hold
* Revert "docs(proxy): streaming loop comment names the 50 ms hold"
This reverts commit cbf1371873.
* refactor(proxy): streaming loop hold budget is _STREAM_LOOP_HOLD_MS = 50
* Revert "refactor(proxy): streaming loop hold budget is _STREAM_LOOP_HOLD_MS = 50"
This reverts commit 356454def6.
* docs(proxy): streaming loop comment notes the yield is a floor, not a count of releases
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): expose public names for private proxy helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep original class names behind public aliases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve internal callback filtering
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep _PROXY_ class names for managed files hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep old private names in package exports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve recursive auth helper name
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): align MCP limiter tests with server enforcement
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): match main's MCP limiter tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep old private names bound in importing modules
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve compatibility imports through strict lint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): use exact pyright suppression in password helper test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): add reasons to compatibility import noqa comments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): update IN-list baseline for renamed helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): stamp the client alias on a copy of each streamed chunk so pricing sees the deployment model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): streamed alias matching a capability rule bills the deployment price
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): assert every streamed chunk carries the client alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): log the client alias on the priced streamed response, the same as non-streamed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): move proxy_server, _experimental and db tests into tests/unit/proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): keep tuple identity in proxy state restore and fix misc target paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-01 18:14:24 +00:00
Renamed from tests/test_litellm/proxy/proxy_server/test_streaming_helpers.py (Browse further)
