* fix(chatgpt): keep text and accept string input for the Codex backend
Two request-side defects stop chatgpt/* serving structured output or a string
input.
The allowlist at the end of transform_responses_api_request omits "text". The
chat-to-responses bridge translates response_format into text.format, so the
allowlist discards strict schemas silently and the backend answers with prose
instead of JSON. The backend does honour text.format when it receives it.
Separately the backend rejects a bare string input with
{"detail": "Input must be a list"}, while the Responses API itself accepts
either shape, so the string needs wrapping before it is sent.
Both are request-shaping only, so neither depends on the streaming work in
#31332 or #34095, though those two block the same call and have to be in place
to observe this one end to end.
The "text" case was reported in #24356 and closed by the stale bot without a
fix.
Signed-off-by: Alexander Chernov <alexander@chernov.it>
* fix(chatgpt): annotate the input coercion for the type-discipline gate
The LIT002 budget counts mutable-collection construction, and wrapping the
string input builds a list and a dict, which put the total two over its
ceiling. Both constructions are the wire payload this backend requires, so the
list cannot be replaced with an immutable equivalent; annotate the line with a
reason instead, matching the convention used elsewhere in the tree.
LIT005 freezes reasonless suppressions at zero, so the reason is required
rather than decorative.
Signed-off-by: Alexander Chernov <alexander@chernov.it>
* refactor(chatgpt): bind the coerced input instead of rebinding the parameter
Greptile flagged the parameter rebinding and the explanatory comments against
the repo conventions in CLAUDE.md, which bans rebinding a function parameter
(LIT011) and keeps comments to suppressions, TODOs and genuinely complex logic
Bind a Final local for the coerced input rather than reassigning `input`, and
drop both prose comments; the rationale lives in the commit and the PR body
rather than duplicated at the call site. The mutable-ok suppression stays,
since it is the allowed kind and the list is still constructed
Behaviour is unchanged: 24 tests pass, and the same 4 fail with only the
provider file reverted to base
Signed-off-by: Alexander Chernov <alexander@chernov.it>
* fix(chatgpt): forward max_output_tokens to the Codex backend and return incomplete non-stream responses
The Codex backend accepts and enforces max_output_tokens, so the allowlist now keeps it
instead of dropping it. A structured-output request could otherwise stream until the
backend gave up, ignoring the caller's max_tokens.
A capped request ends with response.incomplete, so the non-stream SSE aggregator now
treats that event as terminal and returns the partial response instead of an error.
---------
Signed-off-by: Alexander Chernov <alexander@chernov.it>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): split Gemini SSE lines on CR/LF only so U+2028 in text keeps the stream
The Gemini and Vertex AI streaming path read the SSE body with httpx
iter_lines()/aiter_lines(). Those follow str.splitlines(), which also breaks
on U+0085, U+2028, U+2029, \x0b, \x0c and \x1c-\x1e. JSON leaves U+0085,
U+2028 and U+2029 unescaped inside strings, so a data: line whose text holds
one of them was cut in two. The tail is not a data: line and the head never
parses, and the accumulate-and-retry fallback then carries the unparsed
prefix forward, so every later event is appended to it and the stream ends
with no more content and no usage.
SSE lines end only at CRLF, LF or CR. The new iter_sse_lines/aiter_sse_lines
split decoded text on exactly those, treating a CR at the end of one chunk and
an LF at the start of the next as one break. The first request and every
long-decode hop now read through them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(streaming): keep merged think tags balanced and split shared SSE lines only on CR/LF
Close an open <think> block on the first content or tool-call delta and at stream end, reopen it when reasoning returns, and route the shared chat completions stream through the SSE line splitter so U+2028 and U+2029 inside JSON no longer drop chunks.
* fix(streaming): exact-match the SSE [DONE] marker in BaseModelResponseIterator
* fix(streaming): split every SSE reader on CR/LF only and keep the logged think block closed
Switch the remaining httpx stream readers (Anthropic chat, Google GenAI,
Interactions, Codestral, openai_like, Predibase, LangGraph, Vertex Agent
Engine, A2A providers, Azure AI agents, sandboxes, the A2A endpoint, and
others) from iter_lines/aiter_lines to iter_sse_lines/aiter_sse_lines, so a
raw U+2028 or U+2029 inside a JSON string no longer splits an event.
Keep the chunk that closes a think block in the logged and cached response
when a stream ends without a finish_reason, close the block before a legacy
function_call, and make sync Codestral FIM streaming send stream=true and use
the text-completion-codestral stream parser.
Test fakes that only mocked iter_lines now stream real httpx bytes and
assert the content they feed in.
* fix(streaming): split buffered SSE parsers only on CR and LF, shard the codestral tests
Buffered SSE and JSONL parsers used str.splitlines(), which also breaks on
U+2028, U+2029 and U+0085 that JSON allows raw inside strings. They now use
split_sse_lines from sse_lines.py, which also gains a str.split fast path.
Adds tests/unit/llms/codestral to the unit workflow shard.
* fix(streaming): keep U+2028 in restamped Anthropic frames and fold CRLF before splitting SSE lines
* test: type the new stream tests, drop a comment, and benchmark the SSE line splitter
* test: type the iterator class parameters in the line-separator stream tests
* fix(responses): stream tool call arguments opened in a text chunk
* fix(responses): hold a tool call that shares a chunk with text until the text ends
* test: type the a2a SSE handler helper and the sandbox fake responses
---------
Co-authored-by: Tim Vink <vinktim@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
* fix(chatgpt): derive Responses session id from prompt_cache_key
ChatGPT partitions the Codex prompt cache by the Responses session-id
header (the Codex CLI sends a stable per-conversation key derived from
prompt_cache_key). The provider only consulted explicit session ids
and otherwise stamped a per-request uuid4, so no turn ever reused the
prompt cache -- 0% cache hits on repeat turns, an order-of-magnitude
quota/cost multiplier on subscription accounts.
Resolution order after this change:
explicit session config (session_id / litellm_session_id /
metadata.session_id) > prompt_cache_key (sanitized with
_safe_header_value) > litellm-internal ids > uuid4.
Two guards, mirroring get_fireworks_session_id:
- proxy-generated session ids (missing_session_id: "generate") are
per-request and are skipped, checked in both "metadata" and
"litellm_metadata" (LITELLM_METADATA_ROUTES, responses included)
- internal per-request ids (litellm_trace_id / litellm_call_id) lose
to a stable conversation key
Supersedes the body-key forwarding of #37630: a random session header
overrides the body key on the provider side, so the header must carry
the caller's key. Related: #29978, #29993.
Verified against a real ChatGPT Codex account: stable session id ->
97-100% cached on repeat turns; random per-request -> 0%.
* style: ruff format
* fix(chatgpt): address review - hash unsafe cache keys, move tests, drop comments
- hash prompt_cache_keys containing header-unsafe chars instead of
mapping them through _safe_header_value, whose replacement char could
collapse distinct keys onto one cache shard and serialize unrelated
requests (greptile review)
- strip the explanatory comment blocks per AGENTS.md comment policy
- move the tests into the existing mapped responses transformation test
file per AGENTS.md test-placement rule
* fix(chatgpt): keep session typing and test diff scoped
* fix(chatgpt): hash surrogate cache keys, type session params, flatten tests
* test(chatgpt): record upstream requests with a mock instead of appending to a list
* fix(chatgpt): keep proxy-mapped session headers ahead of body session_id and ignore non-string cache keys
* fix(chatgpt): hash cache keys with leading or trailing whitespace so the session header stays valid
* test(chatgpt): type the session id test parameters
* fix(chatgpt): keep trace ids copied into litellm_session_id from outranking prompt_cache_key
* fix(chatgpt): mark session ids copied from trace_id so explicit ones keep precedence
* fix(chatgpt): keep the trace-id session marker out of provider params
The marker now uses the _litellm_ internal prefix, so bridges that pass litellm_params back into completion() drop it instead of sending it as a provider param
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(chatgpt): send system input items to Codex as developer
The Codex backend rejects {"type": "message", "role": "system"} input
items with 400 "System messages are not allowed", while it accepts the
same item with role=developer. The chat completions bridge emits that
shape for every non-leading system message, and typed Responses API
clients send it directly, so both paths failed. Rewrite system items to
developer in place, which keeps their position and content.
Fixes#48562
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(chatgpt): annotate system-item rewrite with read-only types
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(chatgpt): rewrite system items on the typed input before the base transform
Keeps the rewrite on known types so basedpyright sees no new unknowns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(responses): keep the provider code and message on top-level stream error events
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): log a mid-stream failure before the router falls back
CustomStreamWrapper now waits for failure logging before it raises the
error the router falls back on, so the cooldown lands on the deployment
that failed instead of the fallback that replaced it. Also keeps numeric
error codes and the top-level message on the WebSocket and ChatGPT paths.
* fix(streaming): run sync failure callbacks before a failed stream falls back to another deployment
The router cools a deployment down from a sync failure callback that reads the request's model_info. A failed stream submitted that callback to the logging executor without waiting, so it often ran after the router's fallback had overwritten model_info and cooled down the healthy fallback deployment instead, leaving the model group with no deployments
* test(responses): cover top-level OpenAI stream error events end to end
* test(streaming): read live worker pids so the second SIGKILL chaos variant sees two workers
* fix(streaming): run waited sync failure logging off the shared logging pool
* revert(streaming): keep failed-stream failure logging in the background
Waiting for sync failure callbacks before a failed stream raises fixed the router cooldown race, but it slowed healthy calls under many failing streams with a slow callback, since the wait shares a thread pool with every call's start. The race is pre-existing on main and is tracked separately, so this PR goes back to main's logging timing and keeps only the top-level error event changes
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(chatgpt): preserve requested service tier in Responses calls
* fix(chatgpt): avoid extra mutable collections in tier filtering
* test(chatgpt): refresh service-tier cases after main rebase
Keep the tier-preservation regression on the current subscription model and match the moved unit-test suite's formatting. The adapter still forwards preferences without claiming backend priority entitlement.
* refactor(chatgpt): map service tier through a lookup table after the allowlist filter
---------
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Migrate 18 provider test files from tests/test_litellm/llms to tests/unit/llms.
194 kept tests move as-is after mutation testing; 1 test deleted
(test_completion_datarobot_with_environment_variables, env-gated no-assert);
the fixture-only fal_ai cost calculator file is removed.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
