mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-05 08:07:05 +00:00
b408b1d6dc
The Headroom guardrail sent every message to /v1/compress, including the system prompt and the user's current instruction. On an agentic /v1/messages request the live turn is the largest compressible blob, so it came back as a hash marker; the model then called headroom_retrieve and got its own instruction returned in a tool_result block, which reads as data it fetched rather than a request to act on, so it described the content instead of doing the work. litellm already owns the policy for what a compressor may never rewrite: get_protected_indices covers the system rows, the last user row and the last assistant row, and compress() expands it over whole tool exchanges. Headroom now consults it (promoted from a private name and given tests) and expands it the same way, so the trailing tool result cannot come back as a marker standing in for the result of the call the model just made. Protected rows are withheld from the payload rather than pinned afterwards, so their tokens are not reported as savings that are never applied; the write-back discards a compressed system prompt outright, so that saving never existed. The cost is that a query-aware service no longer sees the newest user message. A response whose row count differs from what was sent can no longer be interleaved with the withheld rows, so it goes through the configured fail policy instead of being adopted. Fail-open now returns the caller's own inputs object: translation handlers detect a rewrite by identity, so a rebuilt copy sent an unchanged request through the Anthropic write-back for nothing. That write-back rebuilt the request with one anthropic_messages_pt call, which merges every run of consecutive user/tool rows, so a tool_result turn and the user turn after it arrived fused. Converting a row at a time would separate them but breaks tool pairing: with modify_params on, an assistant row whose results are converted separately reads as an orphaned tool call and the sanitizer answers it with a synthetic "tool execution skipped" result while dropping the real one. Conversion is now grouped by tool_call_id ownership, which satisfies both, and the same grouping decides which rows headroom protects, so the two agree by construction. The CCR follow-up also dropped any text the model wrote alongside its tool call, and echoed tool calls it had no results for. Both are fixed by reusing compresr's extraction helper, now shared instead of duplicated. Resolves LIT-5018
55 lines
1.9 KiB
Python
55 lines
1.9 KiB
Python
"""
|
|
Unit tests for litellm.compression.compress helpers.
|
|
|
|
get_protected_indices is the shared policy for which messages a compressor may
|
|
never rewrite. It is consumed by compress() and by the Headroom guardrail, so
|
|
the two agree on what "never compress this" means.
|
|
"""
|
|
|
|
from litellm.compression.compress import get_protected_indices
|
|
|
|
|
|
def test_protects_system_last_user_and_last_assistant():
|
|
messages = [
|
|
{"role": "system", "content": "sys"},
|
|
{"role": "user", "content": "old question"},
|
|
{"role": "assistant", "content": "old answer"},
|
|
{"role": "user", "content": "newer question"},
|
|
{"role": "assistant", "content": "newer answer"},
|
|
{"role": "user", "content": "live instruction"},
|
|
]
|
|
|
|
assert sorted(get_protected_indices(messages)) == [0, 4, 5]
|
|
|
|
|
|
def test_history_is_not_protected():
|
|
messages = [
|
|
{"role": "user", "content": "old question"},
|
|
{"role": "assistant", "content": "old answer"},
|
|
{"role": "tool", "tool_call_id": "t1", "content": "old tool output"},
|
|
{"role": "user", "content": "live instruction"},
|
|
]
|
|
|
|
protected = sorted(get_protected_indices(messages))
|
|
|
|
assert protected == [1, 3]
|
|
# The tool row and the older user turn stay compressible; protection that
|
|
# covered everything would make compression a no-op.
|
|
assert 0 not in protected
|
|
assert 2 not in protected
|
|
|
|
|
|
def test_every_system_row_is_protected():
|
|
messages = [
|
|
{"role": "system", "content": "first"},
|
|
{"role": "user", "content": "q"},
|
|
{"role": "system", "content": "second, injected mid conversation"},
|
|
{"role": "user", "content": "live"},
|
|
]
|
|
|
|
assert sorted(get_protected_indices(messages)) == [0, 2, 3]
|
|
|
|
|
|
def test_no_user_or_assistant_rows():
|
|
assert sorted(get_protected_indices([{"role": "system", "content": "sys"}])) == [0]
|
|
assert get_protected_indices([]) == ()
|