* fix: Send the include_usage chunk with an empty choices array
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* test: Skip chunks with no choices when reading streams
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Strip usage from plain-iterator async stream chunks
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* test: Expect empty choices on the usage chunk in the gemma tracing test
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix(streaming): send the empty-choices usage chunk only to callers that asked for include_usage
The usage chunk now has choices=[] only when the caller set
stream_options.include_usage itself. When the OpenAI handler, the proxy's
always_include_stream_usage setting, or the harness added include_usage on the
caller's behalf, the chunk keeps its legacy single empty-delta choice, so
clients that read choices[0] on every chunk keep working.
The proxy and harness mark their injection with the private
_litellm_stream_usage_injected flag, and the proxy scrubs a client-sent copy.
Plain-iterator streams keep usage on their chunk when usage was not requested,
and the LLM Shield trailing flush ignores chunks with no choices.
* fix(proxy): mark the default stream usage injection so hooks keep the legacy usage chunk shape
The proxy's default strip-mode injection adds include_usage for clients that did not ask and drops the usage chunk before it reaches them, but streaming iterator hooks still saw it with choices=[]. Marking that injection with _litellm_stream_usage_injected keeps the old single empty choice for them.
* fix(router): treat deployment and router-default stream_options as injected, and drop the client flag on the queue route
* refactor(router): move the stream usage injection check into router_utils
* fix(router): read stream_options from pydantic deployment params when marking injected usage
Update the mantle, openinference, and text-completion integration tests to expect the include_usage chunk with empty choices.
* fix(proxy): keep the stream usage injection marker out of stored request bodies
* test(streaming): cover who asked for stream usage on chat, queue, deployment and spend log paths
* test: type the queue flag test, pin the logging start time, and build the deployment usage config without mutation
---------
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): schedule default_priority requests once and reject a non-integer priority
A router configured with default_priority failed every request that carried no
priority with a 500 naming a missing 'priority' argument, and an explicit
priority re-entered acompletion and hit the same error. acompletion now resolves
the priority once (the request's integer, else the default, with 0 kept as a
real priority) and schedules through _schedule_factory directly, while
schedule_acompletion delegates to acompletion so the two paths share one queue
pass. A non-integer priority answers 400 naming the param, or is dropped under
drop_params, instead of leaking to the provider or crashing the scheduler.
* fix(router): honor router-level drop_params and schedule prompt-management models under a priority
* fix(router): drop a non-integer priority when the model's deployments set drop_params
* fix(router): resolve the drop_params deployment scope for team routes and deployment ids
* fix(router): read the request and router drop_params flags without a loop-assigned Final
* refactor(router): type the prompt-management wrapper's return and mock the provider in its test
* perf(router): consult drop_params for the priority only when it is not an integer
* test(integration): assert the new priority contract for non-integer and zero priorities
* docs(router): describe default_priority as the priority every request without one takes
* fix(router): parse default_priority once and apply it to text completions and the queue route
A digit string default is taken and any other non-integer default is ignored with a warning instead of failing every request. atext_completion and /queue/chat/completions resolve the priority the way acompletion does: a request without one takes the default, an integer queues once, and a non-integer answers 400 or takes the default under drop_params. Integration cells cover every default-priority surface on an owned two-instance rig.
* fix(router): type the priority dispatch so the lint budgets hold
* fix(router): read priority like the scheduler did, require a priority on the queue route, and keep litellm's drop_params order
* fix(router): drop a priority named in additional_drop_params and report the queued priority on streamed queue responses
* fix(router): serve deployment-id requests without waiting behind the queue and ignore a boolean default_priority
A request whose model is a deployment id now counts that deployment as healthy in the
scheduler, since routing by id already ignores cooldown. Before, the scheduler found no
healthy deployment for an id, so the request waited behind every entry ahead of it.
A boolean default_priority (true, or YAML off) is ignored at startup with a warning,
like any other value that is not a whole number.
The scheduler integration tests now share worker state for the defaulted pair, cool a
group through /v1/responses so no stale in-memory queue is left behind, and use "high"
instead of "1" for the non-integer cases. The concurrent burst test no longer asserts
that the queue drains, matching the explicit-priority burst test, since the scheduler's
read-modify-write queue can race across workers.
* test(router): keep the queue drain check as a BUG-skipped integration test
* fix(router): queue default_priority chat requests once instead of failing them
* test(router): open with the default_priority regression tests alone
* fix(router): honor deployment drop_params for a malformed priority on a request routed by provider model name
* test(proxy): move proxy_server, _experimental and db tests into tests/unit/proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): keep tuple identity in proxy state restore and fix misc target paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-01 18:14:24 +00:00
Renamed from tests/test_litellm/proxy/proxy_server/test_routes_queue.py (Browse further)
