* fix(responses): propagate cache_control through responses->chat completion bridge
User/assistant messages, tool call outputs, and function calls built from
Responses API input items were dropping cache_control, breaking prompt
caching for Bedrock/Anthropic when routed through the litellm_completion
transform. Also propagate cache_control onto the last content element of
user/assistant messages, since Anthropic/Bedrock read it from the content
block rather than (or in addition to) the message level.
Adds the missing cache_control field to ChatCompletionToolMessage and
ChatCompletionResponseMessage TypedDicts to keep the assignments type-safe.
* chore(ui): regenerate schema.d.ts for ChatCompletionToolMessage.cache_control
Adding cache_control to the ChatCompletionToolMessage TypedDict changed
the proxy's OpenAPI spec; regenerated via `npm run gen:api`.
* fix(responses): don't overwrite existing content-block cache_control; add list-content test
Per Greptile review on the PR:
- The message-level cache_control was unconditionally overwriting the
last content block's cache_control when that block already carried
its own (preserved by _transform_responses_api_content_to_chat_completion_content).
Now only propagates onto the last block when it doesn't already have one.
- Added the missing test coverage for the content-block propagation path
with list content, which no existing test exercised (all used plain
string content, so the isinstance(content, list) branch was never hit).
* style(responses): extract cache_control into a named variable, trim comment
Per style review: the message-building branch read cache_control 3
different ways in 5 lines (.get() twice, then [] indexing once), unlike
the sibling tool-call-output/function-call functions this PR touches,
which each extract it into a named variable up front
(output_cache_control, function_call_cache_control). Matched that
convention here too. Also trimmed the 3-line inline comment to one
line, matching the file's usual comment length for similarly subtle
behavior.
* chore: refresh lazy OpenAPI snapshot
* fix: type cache control propagation
* fix: place cache control type suppressions
* style: format cache control type conversion
* refactor(responses): build bridged cache_control messages in one shot
* fix(responses): keep cache_control marks on bridged tool calls and within the provider cap
A function_call item's mark now rides on the tool call the bridge builds, an assistant text
item merged behind a function_call keeps its own mark, a developer item keeps its mark when
it is mapped to a system message, and a previous_response_id replay yields its stale marks
to the newest ones so the provider cap holds. The Responses layer counts the marks on
function_call and function_call_output items before it places an ordinal injection point,
and the chat pass replaces that count with its own instead of adding to it.
* fix(responses): count each cache_control mark once and keep the bridge within the provider's limits
List content carries an item-level cache_control on its last block only, so the
provider sees one cache point and the breakpoint census counts one. Marked function
tools and SDK-typed input items are handled by the Responses-layer reserve, assistant
text merged into a function call turn keeps its mark on its own block, and the
previous_response_id replay budget drops marks per turn, newest kept, instead of
stripping every older turn once one overshoots.
The request's own marks now fall under the same budget, newest kept, so five marked
items no longer answer the provider's 400. A cache_control value that is not an
ephemeral mark with an optional ttl is left off the bridged message instead of
reaching the provider as sent, and a developer message maps to the system role with
its content, name, and cache_control only, so unknown keys never reach a strict
provider.
* fix(responses): send item cache_control only to cache-point targets and follow drop_params
Item marks reach the provider only when the target turns them into cache
points, and every other target keeps main's behavior of stripping them.
Without drop_params a mark the provider would refuse reaches it as sent,
so the client gets the same 400 as on /v1/chat/completions. With
drop_params a malformed mark is dropped and only the newest 4 are kept.
The shared input-to-messages helper and the developer mapping in
base_utils go back to main, and a marked developer item reaches a
cache-point target as a system message in the bridge.
* fix(responses): budget top-level marks, follow drop_params on Nova tool marks, and match chat-keeping providers
* fix(responses): validate replayed marks and count input tokens without item marks
* fix(responses): leave unknown Bedrock ARNs unmarked under drop_params, match OpenRouter's list, and reserve only forwarded marks
* test(responses): cover item cache_control parity per chat route, drop_params trimming, and replay budgets
* fix(responses): keep item marks off inspection paths and match each chat route's keep rule
Only the request the bridge sends gets item marks now. Guardrails, token
counting, rate limiting, and input_tokens read the input with no item marks,
as on main, so an off-spec mark cannot fail a guardrail call. Tool message
and tool call marks are typed inside the bridge, leaving the shared chat
types and guardrail schemas as on main.
Databricks keeps marks only for Claude, Snowflake uses its chat route's
Claude check, and every Anthropic deployment keeps them whatever its model
name. The injection census skips SDK-typed tools, and the owned-proxy test
expects the marked messages on /v1/responses.
* test(responses): cover guardrail reads without item marks, Snowflake name parity, and SDK-typed tools in the census
* fix(responses): leave a non-list input untouched when stripping item marks
* test(responses): cover string, list, object and tuple input with and without item marks kept
Each input shape now answers the same whether the unmapped Bedrock ARN keeps item marks or strips them under drop_params, through the proxy and the SDK. Object input also runs through the OpenAI-compatible chat bridge, /v1/responses/input_tokens and the project input-token limiter
* test(cache): derive owned-proxy cell timeouts from the harness boot budget
The capped-burst cells and the uncapped response-cache cells boot their own two-worker proxy, and a fixed 240 or 360 second timeout cut them during boot on a slow runner before the harness readiness budget could decide. They now carry the same derived budget as the sibling chaos and guardrail files, with four boot and stop budgets for the restart cell that boots twice
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
d03923e6a9