* fix(responses): keep a flagged hosted deployment's prompt cache breakpoint on the chat bridge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): read the serving provider's row for prompt cache breakpoints on the chat bridge and base_model
* fix(anthropic_cache_control_hook): rename the module-level provider resolver so the recursion check passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): follow the credential dialog's new name field and auth method id in the federation e2e spec
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): map input_audio blocks in the chat-to-Responses bridge
* fix(responses): honor base_model when deciding to drop input_audio under drop_params
* fix(responses): carry a dropped audio part's cache breakpoint to the preceding part
Dropping an input_audio part under drop_params lost the prompt_cache_breakpoint
riding on it, so an injected system marker landing on a trailing audio block left
prompt_cache_options explicit with no breakpoint on the wire. The marker now moves
to the nearest preceding kept part when that part has none.
Update the CircleCI-only Responses bridge wire tests that pinned input_audio to
its stringified input_text shape: video_url becomes the stringified kind so the
marker-on-stringified-block coverage survives, and dedicated audio tests assert
the forwarded part and the drop_params carry-over.
* refactor(responses): carry dropped audio markers in one pass
Replace the per-block slice that re-read the rest of the content list for
every retained part with a single reverse scan, and skip the copy for a
message that carries no audio part. Type the unit test helper's messages
as a sequence of mappings instead of bare lists.
* test(responses): add integration cells for input_audio parts through the bridge
Cover the chat-to-Responses bridge's input_audio handling on the real proxy: the part is forwarded
as input_audio without drop_params and dropped under drop_params unless the model or its base_model
supports audio input, across httpx, the OpenAI SDK (sync stream and async), the Anthropic SDK, native
/v1/responses, tool and assistant messages, hostile input_audio values, duplicate parts, an audio-only
message, marker carry-over, a null drop_params, a global litellm_settings.drop_params, a mixed burst,
and a burst with dropped upstream connections
* test(responses): check every upstream attempt of a dropped connection
The dropped-connection cell required exactly one upstream POST per call,
which tied it to the retry policy instead of the fix. It now groups the
recorded POSTs by marker, requires one attempt per answered call, and
checks that every attempt of a doomed call carries the dropped audio shape.
* fix(responses): keep cache breakpoints on requests that reach the Responses API
The chat-completions bridge rebuilt every content block without the
prompt_cache_breakpoint marker the cache-control hook had just placed on it, so
the request went out with prompt_cache_options set to explicit mode and nothing
actually marked. Explicit mode caches only what is marked, so those deployments
lost the implicit caching they were getting before the injection point was added
The native surface had a second miss. The bridging check looked the provider
config up with the still-prefixed model while the config keys on the bare one,
so every bedrock_mantle model read as having no native Responses support, and
role-targeted injection points were deferred to a chat-completions pass that
never runs for them
* fix(responses): keep prompt cache breakpoint on stringified content blocks
* test: cover the responses/ routing prefix in the native cache point case
* fix(responses): keep only the fallback-branch marker carry, main already routes the hook
#42281 landed the text, image_url and file branch carry and the implicit default
on main, and the Responses routing lookup this branch changed has no observable
effect there, so the PR shrinks to the unknown-block fallback branch and its
regression test
* fix(responses): drop a malformed prompt_cache_breakpoint under drop_params in the chat-to-Responses bridge
* fix(responses): accept the 30m prompt_cache_breakpoint ttl under drop_params
OpenAI's Responses API takes a ttl of 30m on an explicit breakpoint marker, so the drop_params validation keeps it instead of stripping it as an unknown key. Trims the bridge test docstrings to the dated vendor citation.
* test(integration): audit cells for prompt_cache_breakpoint through the chat-to-Responses bridge
Adds the /audit cells for the bridge's prompt_cache_breakpoint carry: valid and malformed markers on every
carrying block kind and on tool and assistant content, the drop_params on and off contract, the hook-injected
system marker, the Anthropic SDK path through the Responses adapter, upstream errors, idempotent spend rows,
and the chaos cells (mixed burst, upstream outage, slow streams, worker SIGKILL and a proxy restart mid burst),
all against a scripted Responses endpoint with the request it received read back by response id
* test(integration): give every scripted responses reply its own id
The prompt-cache-breakpoint audit's scripted upstream minted the response id
from the request's marker, so three identical marked requests shared one id.
The spend-log writer skips rows whose request_id already landed, and the
idempotent-logging cell saw one row for three calls on every leg. The upstream
now mints a unique id per reply like a real provider, and the cells match a
response to its request by the marker inside that id.
* fix(responses): keep prompt_cache_breakpoint markers in the chat to responses bridge
Preserve cache-breakpoint markers through the bridge for supported models and drop them for models without breakpoint support
Co-authored-by: Simon Sorg <simonsorg13@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): honor base_model when gating bridge cache breakpoints
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): avoid recursive cache-breakpoint stripping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): justify bridge stripping casts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): keep cache breakpoints out of non-bridge converter callers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): read prompt_cache_breakpoint without validating content blocks
The marker read ran every chat content block through
TypeAdapter(dict[str, object]).validate_python, which rejects dict blocks
with non-string keys that chat completion callers passing Python dicts
could previously send; the request then failed with a pydantic
ValidationError on the bridge keep path, the strip path, and the
image/file conversions alike. item is already isinstance-narrowed to a
dict at every read site, so read the marker with dict.get directly and
drop the adapter.
Adds a regression test covering text/image_url/file blocks carrying
non-string keys on both the keep (gpt-5.6) and strip (gpt-4o) paths.
* fix(responses): cast content block before reading prompt_cache_breakpoint
basedpyright flags the raw dict.get read as reportUnknownArgumentType
(+2 against the error budget); cast the isinstance-narrowed block to
dict[str, object] first, matching the strip helpers' cast-ok idiom.
* style(responses): ruff-format the marker-read cast
* fix(responses): hoist one cast-ok content block read for the type gates
A Final assignment inside the conversion loop trips
reportGeneralTypeIssues, and per-site casts trip the LIT006 budget;
read the marker through one cast-narrowed local instead.
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Simon Sorg <simonsorg13@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): honor request cache controls on chat completions bridged to the Responses API
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): assert spend and cache-hit status on bridged no-cache rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): unskip LIT-9196 openai_responses basic translation cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): restore azure attribution check on bridged no-cache rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): exercise bridged cache controls through a real local cache instead of patched responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(responses): revert "fix(responses): keep gpt-5.4/5.5 tool calls on chat and merge bridged tool calls into one choice" (#44295)
This reverts commit ca1994e403.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): merge bridged tool calls into the same choice as the text
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): drop bridge-minted reasoning items from OpenAI replays
* fix(responses): send id-less stored reasoning items without a made-up id
The chat-to-Responses bridge gave a stored reasoning item with no id an rs_<n> id that OpenAI rejects (404 without encrypted content, 400 with it); an id-less item is accepted and verified by OpenAI itself. Decoding encrypted_content now keeps the verifiable thinking blocks of a mixed array instead of rejecting the whole array, and the verifiable-block rule lives in the shared module.
* test(integration): audit minted reasoning item replay across responses, chat bridge and messages
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(streaming): keep the provider's served service_tier on streamed chunks and spend rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): satisfy type-discipline and strict ruff budgets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): stamp the served service_tier on every Responses bridge chunk
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic-adapter): expose streamed chunks so disconnects bill partial spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(service-tier): cover anthropic and responses served-tier billing paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic-adapter): return a chunks-exposing stream so disconnects bill partial spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(service-tier): bill disconnects through the router's anthropic stream wrapper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: apply ruff format to the anthropic stream changes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coverage): ignore delegating properties the ast scan cannot see
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: keep the cast-ok reasons on the cast call line
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover served service_tier billing for streamed chat and messages, complete and disconnected
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic-cache): delegate chunks/messages/model through the messages stream cache writer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): keep service_tier on OpenAI-compatible parsed chunks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(streaming): parameterize delegated chunks and messages types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tests): follow the anthropic pass_through rename after merging main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): drain the logging worker between response cache tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): cover azure, databricks, responses bridge and gemini served tiers in the stream billing integration test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): keep the served service_tier on streamed chunks and bill it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): type the served service_tier chunk without a loose kwargs dict
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): bill the served service_tier over the requested one
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost): drop explanatory comment from the tier resolution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* ci: run the unit_selection.sh shard files on every event instead of only fork pull requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: rename fork-flag to unit-flag now that it applies on every event
* test: move tests/test_litellm root and small trees into tests/unit
Pure renames, no content changes. Follow-up commits in this PR fix
references, merge the three files that already existed in tests/unit,
keep live-provider tests in tests/test_litellm and wire CI.
* test: carry tests/test_litellm conftest isolation into tests/unit
Callback lists, routing fallbacks, cached HTTP clients, logger state, AWS,
proxy-URL and keychain env, and session-end client cleanup now reset for
unit tests too. The environment isolation owns its MonkeyPatch so a test's
own monkeypatch is undone before the model-cost teardown runs.
* test: merge, split and prune the moved root and small-tree tests
Merge batches/test_batch_utils.py and the chat_completions and messages
dispatch tests into the files that already existed in tests/unit. Keep
the live Gemini interactions tests, the async image-fetch format test and
the OpenAI embedding scorer test in tests/test_litellm since they need
real network or keys. Put test_router.py under tests/unit/test_router so
the existing package no longer shadows it. Delete eight tests the audit
found superseded by stronger ones kept in this move.
* ci: run the moved root and small-tree tests under their legacy flags
Add the misc and responses-caching-types flags to unit_selection.sh and
CircleCI, extend enterprise-routing and mcp-integration, and point the
legacy GHA shards, Makefile, redis-compat workflow, merge smoke manifest
and change classifier at the new paths.
* test: make the new tests/unit directories packages
tests/unit/test_package_layout.py requires every directory to carry an
__init__.py, and without one the moved and retained
test_litellm_responses_bridge.py modules collide on import.
* test: scope the unit socket block to tests/unit in shared sessions
The GHA shards collect the legacy test-path and the unit selection in one
pytest session. The unit conftest's loopback-only block leaked into legacy
modules that reach the network at import. The legacy conftest now lifts the
restriction at collect and setup time, and the unit conftest re-applies it
when collecting its own modules.
* test: give the shard-script tests their own GITHUB_OUTPUT
They only passed where the runner set it. The CircleCI unit job's env
allowlist drops it, so the script's redirect failed there.
* test: point the router and module-deletion checks at tests/unit
router_code_coverage and code_qa_check_tests only searched tests/test_litellm,
so the moved router tests no longer counted. The two silent-experiment tests
the audit deleted were the only direct callers of those methods; they are
replaced with tests that assert the forwarded shadow request and the
recursion guard.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
