Adds streaming, async, /v1/responses, and /v1/messages coverage for the
Together AI overhaul (#38233, #38248, #38230, #38265, #38275), plus the
legacy api.together.xyz host and TOGETHER_AI_API_BASE through
litellm.completion. Each new test fails under a one-line mutation of the
merged code.
The variant prefix pattern only understood word variants, so
data-[side=top]:z-50 or [&>*]:z-[5] slipped past the rule. Parse the
utility as everything after the last top-level colon (brackets and
parens nest) and also strip the important marker. Formats the two files
prettier flagged in CI
Type the GPT-5.x reasoning payload with a ReadOnly TypedDict so the dict
literal satisfies the type-discipline budget, and drop the now-redundant
thinking pop (the thinking mapping is already skipped for these models).
Update the cross-region capability test to expect reasoning_effort offered
and thinking/output_config withheld for GPT-5.x on Converse.
Regression LIT-6143 (the policy Flow Builder painting its guardrail dropdown
underneath a position: fixed shell at z-index 1000) was one instance of a
class of bug: pages picking their own z-index numbers above the portalled
popup layer. This removes the class.
- globals.css defines the only z-index values in the dashboard as Tailwind
utilities: z-raised, z-chrome, z-sticky, z-sticky-pinned, z-floating,
z-overlay, z-popup; every numeric, arbitrary and inline z-index across
src is migrated onto them and tailwind-merge learns the tokens
- new local/no-ad-hoc-z-index ESLint rule bans z-<n>, z-[...], z-(...) and
inline zIndex everywhere, and reserves z-popup for the portalled
primitives in components/ui (and the DataTable menus)
- the Flow Builder renders in the dashboard content area instead of as a
fixed full-screen overlay, so it has no stacking level at all
- the guardrail content-filter Add keyword / Add pattern / Custom pattern
dialogs drop the leftover z-[1100] (renamed from ABOVE_ANTD_MODAL when
antd was removed) that hid their own Action select and pattern combobox
behind the dialog, the same bug as LIT-6143
Keys minted by /key/generate get no LiteLLM_UserTable row, so /v2/model/info?user_models_only=true for such a user hit the new None guard and returned 400 where the merge base returned the user's own models. Skip the team-model merge for a missing row instead of raising, since a user with no row belongs to no team
The router's pre-content ping filter dropped AgenticAnthropicStreamingIterator's
hold-back keepalive, so a held-back turn sent the client nothing until the buffer
settled. A ping that no lifecycle frame precedes is now forwarded live, since a
fallback's message_start can still follow it without overlapping lifecycles
The proxy's cancel-refund guard checked isinstance against the iterator, but the
proxy only ever sees it behind FallbackAwareAnthropicMessagesStream and
AnthropicMessagesStreamingResponse, so a disconnect during hold-back refunded the
budget reservation anyway. Both wrappers now forward a duck-typed
has_buffered_provider_output flag, and the router wrapper follows a fallback
source so the flag tracks the stream actually being consumed
Stop advertising thinking/output_config as supported for OpenAI GPT-5.x and
skip the thinking mapping for these models, so a request combining thinking
with reasoning_effort can no longer leak a thinking block into
additionalModelRequestFields regardless of parameter order, which Bedrock
rejects with unknown_parameter.
prisma-client-py delete() returns None when the row is already gone, so the
handler reported deleted: true for a row this caller never removed. Surface
the same 404 the read path uses instead
OpenAI GPT-5.x models on Bedrock Converse expect reasoning effort under
additionalModelRequestFields as {"reasoning": {"effort": ...}}. They were
falling into the Anthropic branch and emitting a `thinking` block, which
Converse rejects with unknown_parameter.
The bedrock_converse gpt-5.6 entries were also missing supports_reasoning,
so reasoning_effort was dropped before mapping. Setting the flag lets the
existing config-driven supported-params path accept it, rather than adding
another model-name branch.
* perf(streaming): add shared JSONFragmentAccumulator for Vertex and Anthropic
Vertex's handle_accumulated_json_chunk and Anthropic's
_handle_accumulated_json_chunk each independently accumulated SSE fragments
into a JSON envelope with self.accumulated_json += fragment. Because the
attribute holds a live reference, CPython copies the whole prior buffer on
every fragment, making buffer assembly O(n^2) in total payload size.
Anthropic additionally had no completeness heuristic at all and called
json.loads on the whole buffer after every fragment, and could wedge forever
on two concatenated envelopes.
Add JSONFragmentAccumulator in litellm_core_utils/: fragments append to a
list in O(1), a could_close_json heuristic lets callers skip the join+parse
entirely until a value could plausibly be complete, and pop_next_value peels
one JSON value off the front of the buffer at a time using
json.JSONDecoder().raw_decode, keeping any unconsumed remainder instead of
failing on concatenated values. Migrate both providers onto it; Anthropic's
__next__/__anext__ end-of-stream handlers now delegate to
_handle_accumulated_json_chunk(is_final=True) instead of duplicating the
parse-and-reset logic inline.
Fixes#31861.
* test(streaming): close diff-coverage gaps in JSONFragmentAccumulator migration
Codecov flagged 11 uncovered lines in the migration: the accumulated_json
setter, __next__/__anext__'s end-of-stream drain branches in both providers,
and the pop_next_value "not found" path when a buffer's newest fragment ends
in "}" but is genuinely incomplete (an inner object closed, the outer one
didn't). Add targeted tests for each.
* fix(streaming): make JSONFragmentAccumulator's completeness heuristic O(1)
could_close_json rescanned every preceding blank fragment on each call, so a
hostile upstream that sends malformed JSON (never closing) followed by many
blank keepalive fragments could drive that scan, and the join+parse it
gates, to O(n^2) total. Track the last non-blank fragment's trailing byte
incrementally in append/pop_next_value/set instead of rescanning the buffer.
Reported by automated review on PR #36610.
* fix(streaming): make JSONFragmentAccumulator.pop_next_value O(1) per call
pop_next_value previously rebuilt the full remaining string and sliced a
new remainder on every call, so draining N concatenated JSON values
already sitting in one buffer cost O(n^2) total. Replace the rebuild-and-
slice with a materialize-once cursor: pending fragments are joined into
the buffer only when new ones have arrived since the last pop, and
consumed values are dropped by advancing an offset instead of copying
the remaining string.
* test(streaming): make JSONFragmentAccumulator drain regression test CI-stable
The 80k-value drain test used an absolute ms budget that flaked on a
busier CI runner (233.5ms vs a 150ms budget calibrated on a quiet
machine). Replace it with a doubling-ratio check: draining twice as many
concatenated values should take roughly 2x as long for O(n), not
~4x for O(n^2), and that ratio holds regardless of machine speed.
* test(streaming): suppress TQ002 on the append-laziness spy test
A test-quality gate (TQ002: don't assert only that a mock was called)
landed upstream since this branch's last rebase and now flags
test_append_never_calls_raw_decode. The test verifies append() defers
all decoding to pop_next_value, which has no caller-observable proxy
other than spying on the stdlib call it must avoid making.
* test(vertex): spy on raw_decode instead of json.loads in accumulator regression tests
Post-migration to the shared JSONFragmentAccumulator, Vertex's decode
path goes through json.JSONDecoder.raw_decode, not json.loads. The two
O(n^2)/partial-fragment regression tests still patched json.loads, which
that path never calls, so both passed unconditionally regardless of
whether the underlying implementation regressed. Verified by simulating
an eager-reparse regression: both tests now fail against it and pass
against the correct implementation.
* fix(streaming): widen JSONFragmentAccumulator's whitespace skip to match str.strip()
pop_next_value's whitespace skip only matched json.decoder.WHITESPACE's
ASCII set, narrower than str.strip() (Unicode-aware) which the O(1)-cursor
rewrite replaced. A non-ASCII separator like U+00A0 between two
concatenated JSON values on one SSE line made raw_decode fail on it, and
the buffer never advanced past that byte again, permanently stranding
everything after it for the rest of the stream. Use str.isspace() to
match str.strip()'s tolerance.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* fix(rerank): emit latency and cost headers on /rerank
Thread the logging object into the rerank httpx calls and pass hidden_params through to get_custom_headers, so x-litellm-overhead-duration-ms, x-litellm-response-duration-ms, x-litellm-response-cost, x-litellm-call-id and the LITELLM_DETAILED_TIMING x-litellm-timing-* headers show up on rerank like they do on chat completions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rerank): keep zero response cost in the /rerank cost header
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: assign the new rerank endpoint tests to the proxy-endpoints shard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: suppress TQ008 on the rerank header tests with reasons
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
