mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-20 00:11:50 +00:00
0a8a87afe0
* fix(streaming): word-sliced cache replay for stream=true cache hits * fix(streaming): align mypy and replay happy-path test with word-sliced cache replay * fix(streaming): short-circuit whitespace-only content in cache replay splitter * fix(streaming): emit tool_calls/function_call only on first replay slice * refactor(streaming): drop dead delattr guard in cache replay A non-None usage on the replay base object always lives in __pydantic_extra__ (it is attached via setattr earlier in the same function), so delattr can never raise here; the try/except AttributeError that silently swallowed a failure was dead defensive code that could only ever hide a real regression, so it is removed in both the async and sync generators. Also switches the new replay annotations from typing.List to the builtin list to satisfy the strict ruff UP006 gate and drops the unused PLR0915 noqa directives (the rule is not enabled in this repo's ruff config, so RUF100 flagged them). * fix(streaming): drop carried-over metadata from later cache replay slices The word-sliced cache replay deep-copies the full ModelResponseStream per slice, so reasoning_content, thinking_blocks, logprobs, enhancements, annotations and the rest of the per-message metadata rode on every slice, not just the first. Downstream handlers that accumulate streamed deltas would collect each one once per slice, e.g. duplicating a cached reasoning trace N times on a stream=true cache hit. Later slices are now rebuilt as a content-only delta with choice-level logprobs and enhancements stripped, so the whole metadata class stays on the first slice. Adds async (logprobs) and sync (reasoning_content/thinking_blocks/logprobs/ enhancements, plus annotations) regression tests --------- Co-authored-by: Mateo <277851410+mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| test_convert_dict_to_chat_completion.py | ||
| test_get_headers.py | ||