* fix(streaming): map stream read timeouts to litellm.Timeout
* revert(streaming): move the timeout mapping to a later commit
* fix(streaming): map stream read timeouts to litellm.Timeout on chat, messages, and responses
* fix(streaming): keep stream fallback and cooldown unchanged for transport timeouts
A chat stream served through the Responses bridge now keeps mid-stream
fallback on a stall, because the fallback exemption also covers a
litellm.Timeout whose cause is an httpx transport timeout. The Responses
stream hands the raw transport error to failure callbacks again, so
router cooldown sees a stall as it did before. The 408 deployment info
now lists stream_timeout.
* test(integration): cover a stalled hosted_vllm stream on chat, Responses, and Messages with its spend log row
* fix(router): keep a fallback deployment whose stream stalled out of cooldown
* test(integration): cover bridged chat, fallback and cooldown for stalled hosted_vllm streams
* feat(vertex_ai): let operators drop chosen anthropic-beta values on the Vertex AI pass-through routes
* fix(vertex_ai): drop configured betas sent through x-pass-anthropic-beta and only fail beta requests on a malformed setting
The drop list now runs on the final upstream headers, after x-pass- expansion, through an
upstream_headers_filter on pass_through_request, so x-pass-anthropic-beta cannot carry a
dropped value to Vertex. A malformed setting now answers 500 only for requests that carry
anthropic-beta or x-pass-anthropic-beta, so Gemini and discovery calls keep working.
* test(vertex_ai): accept the new upstream_headers_filter argument in the existing Vertex route tests
* fix(vertex_ai): refuse a malformed drop-betas list at boot and match list entries written with spaces
* fix(vertex_ai): split drop-betas list entries on commas like the header
Three integration tests diffed the whole shared Redis keyspace (scan_iter or dbsize before and after), so
a sibling test's cache write could satisfy or break them. Each now finds its own entry by a value only it
wrote: the Gemini reasoning test by a per-run Gemini responseId, the max_messages test by a per-test
answer text, and the S3 cache-hit test by its request marker
The sweep auth tests created models through raw /model/new on the shared gateway and never deleted them.
They now create them through Scenario.model, which deletes each model on exit, including when setup fails
* test(integration): stop GET /__observations from draining the upstream log
GET /__observations used to empty the queue, so a second read in a test
saw nothing and a "never reached upstream" check could pass whether or not
the request was forwarded. GET now returns every request since the last
reset and DELETE /__observations is the reset. An autouse fixture resets
the worker upstream before each test, which also covers tests that read
without the gateway fixture.
Discarded reads used as resets become DELETEs, helpers that accumulated
drained reads now read one snapshot, and tests with several phases reset
between them. The three vacuous negative checks (rag allowlist, guardrail
opt-out chaos, xAI batch output) now assert the absence on the same read
that shows a request which did reach the upstream.
* style(integration): space the observations unit test like its neighbours
* fix(proxy): authorize the public name of a team-scoped model on batch retrieve/cancel
A unified batch id resolves to the deployment's internal model_name
(model_name_{team_id}_{uuid}), which is in no key or team allowlist, so
authorize_model_for_key always denied and team keys got 403 retrieving or
cancelling their own batches. Authorize against model_info
team_public_model_name when the resolved deployment is team-scoped, falling
back to the resolved name otherwise.
Fixes#46958
* fix(proxy): authorize team-scoped batch models by their public name on both auth paths
* fix(proxy): accept the owning team's public name for its own team deployment on every auth path
Managed batch, file, fine-tuning, vector store, character and video ids resolve to the
team deployment's internal name again, so teams that granted the internal name keep access.
The model access check now also accepts the owning team's public name and its legacy team
aliases for that internal name, and only when the caller's team owns the deployment.
* fix(proxy): match a team deployment's public and alias names against the caller's own grants
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(guardrails): preserve Bedrock Converse redactions and honor tool-only scans
* Preserve main's per-block streaming fixes and regression coverage
* fix(bedrock): apply Converse guardrail rewrites texts-first and keep main's write-backs
Map tool results onto their text and json parts only, accept a structured row the flat texts already reproduce, let a flat-text mask win over a structured leaf, ignore texts that do not line up with the request when structured rows come back, keep tool call arguments out of the rows under skip_tool_message_in_guardrail and send no tool definitions to tool-only guardrails
* fix(bedrock): accept blank content on tool-call-only Converse turns
A vendor that returns "" or [] where an assistant row carried only tool
calls no longer fails the request closed. Also type the leaf lookup
without Any and restore the original layout of the existing unit tests.
* fix(bedrock): scan additionalModelRequestFields in Converse guardrail rows and split the handler
Guardrails that read structured_messages instead of texts (Akto, Prompt Security, CrowdStrike AIDR and others) stopped seeing strings in additionalModelRequestFields, which Bedrock can forward into the model prompt. Those strings now go to the guardrail as one user row placed before the conversation, and a row-only edit to them is written back.
The Converse to OpenAI row conversion moves to litellm/llms/bedrock/chat/converse_to_openai.py and the returned-row diffing to structured_edits.py, so the handler keeps only the glue. Rows record the Converse path of every text part, scope flags travel as one value, and path lookups return a value instead of raising. Unit tests use a real CustomGuardrail subclass instead of mocks, and the integration vendor modes get descriptive names.
* fix(bedrock): keep new Converse helpers inside the type discipline rules
* test(bedrock): pin Converse row edits that change shape, role or placement
* fix(bedrock): type the Converse body once so its edits keep their reference
* fix(bedrock): list request-field texts where their Converse row sits
* fix(bedrock): send a reshaped Converse tool result as one text block
A vendor that compresses or summarises a tool result, such as Headroom
collapsing a JSON list into a table string, returned a 500 because the
row could not be split back into the original text and JSON parts. The
tool result now carries the vendor's string as one text block where its
parts were, keeping images and documents in place. It still fails
closed when the flat texts masked that tool result or the row changed
its tool call id.
* fix(bedrock): apply guardrail tool edits on Converse and keep compression guardrails off it
A guardrail that masks a Converse tool description or schema now has that
mask written into toolConfig, a tool it removes is dropped, and a tool it
adds, renames or reorders is refused before Bedrock. Headroom, Compresr and
TypeSafe get only the flat texts on Converse again, as on main, because the
pass-through route has no loop to answer their retrieve tool
* test(bedrock): refuse a guardrail that changes a Converse tool's kind
* fix(bedrock): keep partial-row and row-rebuilding guardrails on flat Converse texts
OpenAI moderation, the LLM judge, CrowdStrike AIDR, Prompt Security, PANW Prisma AIRS
and HiddenLayer scan only part of the rows or rebuild their returned texts from them,
so on Converse they get the flat texts only, as on main.
When a vendor returns changed rows, its texts are applied only if they keep the flat
layout and change only rows the vendor also changed, so row-shaped texts of the same
length no longer overwrite unrelated Converse fields.
* test(bedrock): pin how flat texts sit next to rewritten Converse rows
* fix(bedrock): keep the Bedrock guardrail on flat Converse texts
With experimental_use_latest_role_message_only, the Bedrock guardrail places
its mask by counting text parts per row, and Converse rows do not line up
with the flat texts, so the mask could land in tool call arguments. Give it
the flat texts only, as on main.
* fix(bedrock): keep Akto on flat Converse texts
Akto maps its masked payload back onto the flat texts by value, and a
Converse row joins or serialises tool call arguments and JSON tool results,
so a masked card there could not be placed and the request was blocked.
Give Akto the flat texts only, as on main, where the mask is applied.
* fix(bedrock): list request-field texts last again, as on main
Flat texts listed additionalModelRequestFields strings right after the
system prompt, so flat-text guardrails got a different order than on main.
Akto's live verdict on a benign request depends on that order and started
blocking it. Rows already keep their own order and texts that do not follow
rows are dropped, so the flat texts go back to main's order.
* fix(bedrock): list request-field texts next to their row for row readers
Row readers that rebuild texts from rows could swap a masked request field
with a masked user turn when the field strings came last. Row readers now get
those strings right after the system prompt, where their row sits. Flat-text
guardrails keep the end-of-list order.
* fix(bedrock): list nested request-field texts in their row's order for row readers
Row readers that rebuild texts from rows could write a masked request field
onto a sibling key when the field held nested objects, because the texts walked
the fields depth first and the row breadth first. Row readers now get those
strings in the row's own order.
* fix(bedrock): walk request fields breadth first instead of resolving each path
Resolving every request-field string by its path revalidated the parent
object per string, so a wide additionalModelRequestFields took seconds. The
string walk now goes breadth first for row readers, the same order the row
lists them, in one pass. A unit test pins that texts and rows map the same
way when the vendor masks them differently.
* fix(bedrock): fail closed when a vendor's rows drop a mask its texts carry
When a vendor returns changed rows whose texts do not line up with them, the
rows win. A text the vendor masked on a field its rows left untouched (a tool
call argument, say) then reached Bedrock unmasked. Refuse that rewrite instead,
unless the edited body holds the changed text elsewhere, as texts rebuilt from
the rows do.
* fix(bedrock): read same-length texts by position when rows drop a mask
Matching a dropped text against any string of the edited body let a row
mask on another field with the same token cancel it, so the raw tool call
argument still went out. Texts of the length that was sent are now checked
position by position, and only texts of another length, which cannot be
placed, are matched by value.
* test(bedrock): count each rebuilt text against the rows' edits
* fix(bedrock): keep PromptGuard on flat texts and refuse texts that cannot be placed
PromptGuard rebuilds its returned texts from the user rows, and a vendor that
does so while masking only row content let a card in a tool call argument
reach Bedrock, which main masked. PromptGuard now gets the flat texts as on
main, and a reply whose rows changed next to texts of another length fails
closed, since what those texts mask cannot be checked.
* test(bedrock): let the structured checks decide the unsafe tool call and missing row cases
The replies in these tests carried texts that no longer matched the sent texts, so the new refusal for
texts that cannot be placed answered first and the tool call and row count checks went unexercised.
* fix(bedrock): send request field strings to row readers after the conversation, where main puts them
Row readers got the additionalModelRequestFields row before the conversation. Singulr judges only the latest user
message, so an injection in a request field that main blocked reached Bedrock. The row and its texts now come last,
as main sends those strings in texts
* test(bedrock): pin nested request field order for flat readers and the texts view
Both request field tests now use nested fields whose depth-first and breadth-first orders differ, so a flat reader
getting the row order or a texts view written in main's order fails
* refactor(bedrock): ask each guardrail whether Converse should hand it rows
Converse picked the guardrails that get only the flat texts from a list of
provider names. CustomGuardrail now has supports_structured_messages,
True by default, and the ten classes on that list override it to return
False, so the handler asks the guardrail instead of the list. Every
guardrail class gets the same answer as before
* fix(bedrock): keep Singulr on flat Converse texts as on main
Singulr wraps each flat text as a user message and its backend judges only
the latest user message, so on main a Converse tool result was the turn it
judged. Handed rows, the tool result arrives with role tool and the latest
user row is the original question, so an injection in a tool result reached
Bedrock. Singulr now returns False from supports_structured_messages
* fix(bedrock): keep Qualifire on flat Converse texts as on main
Qualifire builds its evaluation input from the last user row only when that
row's content is a string, so Converse rows (content-block lists) gave an
evaluation_id policy an empty input. Opt it out of structured messages so it
gets main's flat texts again.
* fix(bedrock): keep Conduct on flat Converse texts as on main
The conduct-litellm-guard plugin scans only user rows, so with Converse rows a
system prompt, assistant text or tool result left its scan. Opt it out of
structured messages so it gets main's flat texts again, all sent as user turns.
* fix(bedrock): refuse tool removals that leave a toolConfig Bedrock rejects
A guardrail that removed every tool, or the tool toolChoice names, produced an
empty tools list or a dangling toolChoice that Bedrock answers with an opaque
ValidationException. Refuse those rewrites before Bedrock, like other tool
changes this route cannot carry.
* fix(bedrock): list request-field strings in main's order for every guardrail
The request-field row walked additionalModelRequestFields breadth first, and
guardrails that get rows had their texts follow that order. With two or more
strings the last text changed, so Ovalix, which blocks only when the last text
is flagged, rewrote an injection main blocked. The row now walks the fields in
the order main lists them in the texts, and the texts use main's order for
every guardrail.
* test(bedrock): pin a container's own request-field strings before its nested ones
* test(bedrock): pin tool_policy and MCP end-user filtering on Converse tools
* fix(bedrock): rebuild Converse tools the guardrail changed and drop blanked text like chat
A guardrail that adds, renames, reorders or retypes a tool now gets its list sent to Bedrock,
converted the way chat completions converts tools, with cache points kept behind their tool.
When every tool is removed, toolConfig is dropped and earlier tool calls and results become text,
as chat does. Text blocks a guardrail blanks are dropped, and a turn left empty fails closed.
The body copies used to place edits are JSON copies, so deeply nested tool JSON no longer
hits the recursion limit.
* test(bedrock): keep the client tool schema on Converse masks and refuse deep JSON text drops cleanly
* fix(guardrails): keep Airia on flat texts for Bedrock Converse so its redactions still apply
* fix(guardrails): a pipeline step whose rewrite the endpoint cannot apply follows on_fail
on_error: allow sent the original request on when the endpoint refused a
guardrail rewrite it could not apply, so a Converse request the guardrail
masked reached Bedrock raw. The refusal now counts as a fail on every endpoint.
* fix(guardrails): a pipeline step blocks when its rewrite cannot be applied, whatever on_fail and on_error say
A rewrite the endpoint cannot apply is neither a content verdict nor a
technical error, so on_fail next or allow and on_error allow must not send
the original request on
* test(bedrock): trim the Converse guardrail tests
Drops the chaos and per-vendor integration cells and the duplicate unit cases,
and leaves the existing handler tests as they are on main
* fix(bedrock): put Converse request-field strings in the latest user message for row readers
A guardrail that judges only the latest user message, like Noma v2, read the additionalModelRequestFields row placed
after the conversation, so a request with extended thinking or anthropic_beta sent the user's card to Bedrock and let
an injection through. Those strings now follow the latest user message's text in its row, and get a user row of their
own only when no user message has text.
* fix(bedrock): create toolConfig when a guardrail adds the first Converse tool
A guardrail that returned a tool for a Converse request with no toolConfig got a 500, since the tool write-back only rebuilt an existing list. It now creates toolConfig, as chat completions does. Guardrails that get only flat texts have their returned tools ignored, as on main.
* test(guardrails): check Converse flat-text opt-outs without patching apply_guardrail
Each built-in opt-out is checked through its own supports_structured_messages(), the handler's flat path through an injected scripted guardrail, and Headroom through the real class, which would call its unreachable service if it got rows
* fix(guardrails): keep Converse tool matching and write-back linear in request width
Matching each returned tool rescanned the whole tool list, and every write-back
step revalidated the list or object it passed through, so a request with
thousands of tools or request fields held the event loop for seconds. Returned
tools are now looked up by name in one index, and path steps read the child
directly.
* fix(guardrails): let later Converse guardrails scan guardContent text
When a guardrail removes every tool on a Converse request that carries
guardrailConfig, tool results are rewritten as guardContent blocks. The
flat text extractor read only top-level text, so a later masker in the
same request never saw them and a card in a tool result reached Bedrock
unmasked. Chat completions and /v1/messages keep tool results visible
to later guardrails after the tools are removed.
The extractor now reads guardContent text the same way the structured
rows already do, so flat-text guardrails also scan guardContent text a
caller sends.
* fix(guardrails): keep caller guardContent out of Converse flat texts and send tool history as plain text
Flat-text guardrails read only top-level text blocks again, as on main. A guardContent block after
the caller's text no longer becomes the last text, so guardrails that judge the last text (HiddenLayer,
Qualifire) check what the caller wrote
When a guardrail removes every tool, earlier tool calls and results are now written as plain text
blocks instead of guardContent. Later guardrails still scan and mask them, and Bedrock's own
guardrail keeps checking every message, since a single guardContent block makes it skip all plain text
* fix(guardrails): split a newline-joined Converse message back into its parts
A vendor that returns a multi-part message as one newline-joined string now has its masks written back to each part, guardContent included, when no part sent already holds a newline.
* fix(bedrock): keep Converse tool history as tool messages until send and match chat on emptied turns
When a guardrail removes every tool, the tool calls and results stay tool messages for the guardrails
after it, so a scan_only_tool_results guardrail still masks them. The toolConfig with no tool is dropped
and the history turned into text only when the request is signed, as chat completions does at request
build. A caller's toolConfig with no tool gets the same treatment, where Bedrock returned 400
Request-field strings join the latest user message that has text when a tool-result-only turn follows
it, so a guardrail judging only that message reads the caller's text with them
A turn the vendor empties entirely is sent as written, as on main and chat completions, instead of a 500
Messages go to the guardrail through a JSON round trip instead of a deep copy, and a reply that changes
nothing returns early, so a long Converse conversation guards about as fast as on /v1/messages
---------
Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* test(integration): give each owned proxy a private database and Redis by default
owned_proxy and owned_proxy_process now boot on a scratch database and a
per-proxy redis-server unless the caller opts into the worker's state with
state=OwnedState.SHARED_WITH_WORKER, passes its own DATABASE_URL, or strips
DATABASE_URL. Under the replica profile the private database gets the
writer and reader roles so the reader path stays covered.
Gateway carries the database_url its proxy writes to, and the Scenario
cleanup checks and delete_key_if_present read from it, so they keep
checking something on a private database.
The 189 test files with owned proxy sites that depend on the worker's
database or Redis opt in with a shares_worker_state module marker, which a
module-scoped conftest fixture turns into the shared state. The 7 helper
modules with such sites opt in per call, since conftest fixtures and other
test files reach them.
* test(integration): reach session fixtures with the shared marker and keep the worker retry default
The module-scoped fixture that applied shares_worker_state ran after
session-scoped fixtures, so proxies those booted went private in marked
files. The marker is now applied around each marked item's setup, call
and teardown, which covers fixtures of every scope.
Owned proxies on the worker database inherited the worker gateway's
router_settings.num_retries of 0 from it. A private proxy now gets the same
default in its config copy unless the config sets num_retries itself.
Five providers files read the owned proxy's spend rows or response cache
through the worker's database or Redis and opt into sharing.
* fix(databricks): use the native Anthropic route for Claude /v1/messages
Claude models on a Databricks workspace were bridged through chat completions on /v1/messages, which drops the safeguards field and never returns safeguard_results. They now call the workspace's /serving-endpoints/anthropic/v1/messages, the proxy forwards the client's anthropic-beta and anthropic-version headers to Databricks deployments, and the beta header map marks dangerous-tool-use-2026-09-03 as supported there. Any route whose platform does not take that beta drops safeguards together with it instead of answering 400. Non-Claude Databricks models keep the chat completions bridge
* fix(databricks): derive the dangerous-tool-use beta from safeguards instead of forwarding client betas
A Databricks Claude serving endpoint answers 400 to betas Claude Code sends on every turn, prompt-caching-scope-2026-01-05 among them, so the proxy no longer forwards the client's anthropic-beta header to Databricks deployments. The beta the classifier needs is derived from the safeguards body field, the same way the Azure AI route derives it. The Rust beta-header mirror stops rejecting the Databricks dangerous-tool-use cell so its config-cell test matches the JSON again
* fix(databricks): keep the billing block, build the messages route once, and pass the partner user agent
* fix(databricks): route Unity Catalog Claude names to the gateway messages route and keep env credentials
Unity Catalog model-service names now reach the AI Gateway's native Anthropic
route instead of a serving-endpoints path that answers 404 for them, and the
native route resolves the key and base through the same order the chat route
uses, so environment-only deployments keep working.
The native route is limited to Databricks foundation-model Claude names, an
unlisted provider forwards safeguards and the beta as sent, and the ten betas
the workspace rejects are marked unsupported for Databricks in the beta map
and its Rust mirror.
* test(rust): expect Databricks to drop the advanced-tool-use beta
* fix(databricks): keep passthrough safeguards, partner user agent, and classifier fallback
Only the OpenAI-compatible JSON providers drop safeguards now, so a
deployment opted into /v1/messages passthrough forwards the field and
the beta as sent. When a route drops safeguards and the proxy has a
classifier model, the proxy answers the verdict itself, as the chat
completions bridge did. The Databricks Messages route also reads the
databricks_user_agent param like the chat route.
* fix(databricks): drop rejected server tools under drop_params, short-circuit web search, route UC names from a full Messages URL
* fix(databricks): drop container on the native Messages route, as the chat completions bridge did
* fix(databricks): keep every api_base on a route the workspace serves and gate the derived beta on the map
Gate the dangerous-tool-use beta derivation on the beta map the way #43955 does, so GitHub Copilot, DeepSeek, and opt-in passthroughs keep the merge base wire. Recognize /serving-endpoints/anthropic api_bases, fall back to the merge base routing for an api_base path the workspace trim does not know, strip Bedrock invocation metrics from the streamed message_stop, forward client betas to Databricks filtered by the map, and keep tool_choice when a dropped server tool has no name
* test(databricks): cover the native Messages route, derived beta and JSON provider safeguards drop on the wire
* test(anthropic): expect the dangerous-tool-use beta derived from safeguards next to the client's betas
* fix(databricks): send client anthropic headers only on the native Messages route and keep SDK auth
Databricks chat completions, called directly or through the Messages bridge,
no longer receive the client's anthropic-beta and anthropic-version, as at the
merge base. The native Messages route still forwards them, filtered by the
beta map, without counting them as caller credentials, so a deployment with no
api_key keeps falling back to Databricks SDK auth. A pathless api_base goes to
the native route only for Unity Catalog names, matching the chat route.
* test(databricks): cover SDK auth with client anthropic headers and a Unity Catalog workspace root on the native route
* fix(databricks): keep client anthropic headers off an opted-in passthrough api_base
A databricks Claude deployment whose api_base is no workspace route and that opts
into /v1/messages with supported_endpoints falls back to the OpenAI-like
passthrough. That config does not filter betas through the beta map, yet it
received the native-only Databricks header entry, so Claude Code's
prompt-caching-scope beta reached the upstream raw and got a 400 where the merge
base answered 200.
Each Messages config now says whether it is the provider's native route, the
OpenAI-like passthrough says no, and the handler passes that answer when it
reads provider-specific headers, so native-only entries reach only native configs.
* test(databricks): cover client anthropic betas on an opted-in passthrough api_base
A databricks Claude deployment whose api_base is no workspace route and whose
model_info.supported_endpoints opts into /v1/messages takes the OpenAI-like
passthrough. The cell sends Claude Code's prompt-caching-scope and context-1m
betas through the proxy and asserts the caller gets the upstream's message and
the gateway receives no anthropic-beta, as on the merge base. The scripted
gateway answers 400 on the prompt-caching-scope beta, as the real one does
* fix(anthropic): keep beta map loads off shared threads and build the safeguards evaluator only when the proxy classifies
Unfiltered configs such as first-party Anthropic never read the beta map to decide safeguards support. Filtered configs await the map on its dedicated loader before reading it, on both the native and the MCP paths. The native handler decides after the upstream call and builds the evaluator only when the proxy classifies, so malformed messages keep the upstream 400. The Databricks dispatch tests assert the dispatched route instead of the config class, and the sail and Eden AI wire rows expect safeguards dropped, as JSON providers do.
* test(databricks): cover a loaded beta map that marks the safeguards beta unsupported
A remote beta map that nulls dangerous-tool-use for databricks while the bundled map allows it must send neither safeguards nor the beta to the serving endpoint, with or without a client beta, and the caller gets a 200 with no verdict.
* test(anthropic): annotate the MCP beta map recordings as Final
* fix(anthropic): drop safeguards on routes without the classifier only under drop_params
A route whose upstream does not answer safeguards (a JSON provider, or a map
that marks the dangerous-tool-use beta unsupported) now forwards the field and
the beta as the merge base did unless drop_params is set or the proxy runs its
own classifier, matching how litellm treats any unsupported param. Databricks
keeps dropping both, since its merge base bridge never forwarded the field.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): end a chat completions stream with an error event when a guardrail blocks mid-stream
A streaming guardrail that raised an HTTPException after the first chunk re-raised it inside the SSE generator, so the 200 body was cut off with no finish_reason, no error event and no [DONE]. Build the error frame through sse_error_payload instead, keeping the status, message and structured detail, and encode values JSON cannot represent as strings so the frame always goes out.
* fix(proxy): keep early guardrail blocks on their HTTP status and end mid-stream errors with [DONE]
A guardrail block before the first chunk is raised again, so its response stays the
400 event-stream body it is on main. A mid-stream error event now carries the failure
hook's replacement error when one is returned, is followed by data: [DONE] on
OpenAI-format streams, and leaves the Gemini route's raise unchanged. Error details
JSON cannot encode are written as strings instead of breaking the event.
* fix(proxy): keep a per-chunk guardrail block on the first chunk on its event-stream 400
* fix(proxy): treat a keepalive ping as a started stream so a later guardrail block ends in-band
* fix(proxy): encode non-JSON guardrail details in early stream error frames
An early guardrail block whose detail holds a value JSON cannot encode sent
400 headers and then cut the body. The error frame now writes such a value as
its string form, the same as the mid-stream event. The keepalive test now runs
on a virtual clock instead of real sleeps.
* fix(proxy): show an in-band stream error event as an error in the CLI, Python client, and dashboard playgrounds
ChatClient.completions_stream raises StreamError on a data frame carrying an error, so litellm-proxy chat prints the block reason and keeps the partial reply out of history. The prompt playground and the Interactions playground read the same frame through a shared streamErrorMessage helper. Adds the audit's integration cells for chat stream error events, and makes the endpoint tests' failure-hook mocks return None so they check the real error payload.
* fix(proxy): print stream error text verbatim in the CLI and read split error events in the prompt playground
The CLI printed the server's error message as Rich markup, so a closing tag like [/pii] in a guardrail message crashed lite chat. The prompt playground now buffers partial lines across reads, ChatClient raises only on a non-empty error, StreamError redacts keys like the other client exceptions, and StreamError is exported from litellm.proxy.client
* fix(proxy): print CLI stream errors without Rich highlighting or emoji codes
* test(proxy): cover the CLI's plain-text 400 error print
* test(proxy): annotate the new client stream test locals as Final
* fix(proxy): send the failure hook replacement on Responses streams and before the first chunk
* fix(proxy): end a Responses stream with response.failed when it fails on a built-in exception
* fix(proxy): keep a failure hook replacement before the first chunk in-band so bare stream routes still answer it
A non-HTTP failure whose hook returns an HTTP error was raised before the first
chunk, which create_response turned into the hook's status but which a bare
StreamingResponse route such as /queue/chat/completions answered as a 200 with
zero bytes. Only the original HTTP errors raise now. The replacement goes out as
the stream's error event, which create_response still answers with the hook's
status as a JSON error.
Adds audit round 2's integration cells for failure hook replacements on chat and
Responses streams, built-in exceptions on Responses streams, and the Python client.
* test(proxy): cover a queue stream whose failure hook replaces a failure before the first chunk
The audit cell posts to /queue/chat/completions with a callback that raises before the first chunk and a failure hook that answers 403. It checks for a 200 event stream whose only event is the replacement, then one [DONE], with the callback's own message absent
Follow-up to #40152. Greptile flagged the two new reasoning_effort tests for
assigning config and result without Final. Annotates every local in the file
so it passes the LIT010 check cleanly
* feat(ui): show gateway URL and agent setup instructions after creating a key
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(ui): make the selected agent button visibly distinct in dark mode
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(ui): hide agent instructions by default and use a generic title
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(ui): use one shared agent instructions card with Claude Code and Codex logos
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(baseten): forward reasoning_effort to Baseten Model API
Baseten's Model API accepts a top-level reasoning_effort parameter and
validates it server-side (https://docs.baseten.co/inference/model-apis/reasoning),
but BasetenConfig.get_supported_openai_params did not list it, so LiteLLM
silently dropped it when drop_params=True and raised UnsupportedParamsError
otherwise.
Add reasoning_effort to the supported params so map_openai_params forwards
it verbatim, mirroring the hosted_vllm and deepseek providers. Baseten
returns a clear 400 for values outside a model's supported set and ignores
the parameter on non-reasoning models, so verbatim passthrough is safe for
the whole catalog.
[Claude_Code]
* fix(baseten): forward reasoning_effort to Baseten Model API
Baseten's Model API accepts a top-level reasoning_effort parameter and
validates it server-side (https://docs.baseten.co/inference/model-apis/reasoning),
but BasetenConfig.get_supported_openai_params did not list it, so LiteLLM
silently dropped it when drop_params=True and raised UnsupportedParamsError
otherwise.
Add reasoning_effort to the supported params so map_openai_params forwards
it verbatim, mirroring the hosted_vllm and deepseek providers. Baseten
returns a clear 400 for values outside a model's supported set and ignores
the parameter on non-reasoning models, so verbatim passthrough is safe for
the whole catalog.
---------
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* feat(count_tokens): configurable token_count_framing and token_count_fallback_model, count source in the response
Count endpoints now say where each number came from in litellm_count_source
(provider, fallback_model, local_estimate) and name the fallback model in
litellm_count_fallback_model. When the provider answers 400 or 404 for a model
it cannot count, the proxy counts the same request with token_count_fallback_model.
token_count_framing adds a per-model constant to local estimates. Both settings
ship in the cost map for Bedrock Claude Sonnet/Haiku 5.5, Bedrock Kimi K3, and
Databricks Kimi K3 and DeepSeek V4 Flash, and can be overridden in model_info.
* test(count_tokens): classify the new cost map keys in the schema generator and expect count_source in exact-shape tests
* test(count_tokens): e2e matrix against real inference and integration wire tests for count source, fallback and framing
* fix(count_tokens): keep the provider error when local counting is disabled, name the fallback error, keep settings in cooldown
Moves the cost map lookup off the event loop
* ci(cost map): bound token_count_framing to non-negative and token_count_fallback_model to non-empty
* fix(count_tokens): keep a failed fallback's own status, ship fallbacks only where they count, add system.ai framing
A fallback model that answers 404 returns the requested model's error unchanged, and any other fallback
failure answers with its own status and message. The eu, apac, au and jp Sonnet and Haiku 5.5 entries
no longer ship the Opus 5.5 fallback, which AWS serves only in us-east-1. A configured empty
token_count_fallback_model turns the shipped fallback off. Config framing accepts whole-number floats and
rejects negatives. The Databricks system.ai Kimi K3 and DeepSeek V4 Flash names carry the same framing as
their serving endpoint names, and the e2e matrix runs on Bedrock Haiku 5.5 too.
* test(databricks): list the system.ai Kimi K3 and DeepSeek V4 Flash names with their published DBU rates
* fix(count_tokens): skip a token count fallback model for an hour after it answers 404 on that deployment
A us. or global. Sonnet 5.5 deployment outside us-east-1 paid two extra Bedrock
calls and about 200 ms on every count, since Opus 5.5 cannot count there either.
The Kimi K3 e2e also runs the large system prompt shape now.
* test(count_tokens): use the shared integration port allocator in the fallback wire test
* fix(cost map): keep the Databricks system.ai entries to framing only, so their spend stays as today
* test(count_tokens): cover the fallback 404 skip, inference profile base_model, shipped framing, and deployment-scoped token count settings
Adds integration cells for a fallback model that answers 404, an application inference profile ARN deployment whose base_model carries the shipped fallback, and the shipped Kimi K3 framing with token_count_framing 0 turning it off. Adds unit tests that keep token count settings off the shared backend cost map key and that store a token count setting on PATCH only when the operator changed it
* fix(count_tokens): keep the counter's own 400 and the provider's error when a framed estimate cannot read the body
A 400 the OpenAI or Gemini counter returns before calling the provider is the caller's answer, so it no longer tries the fallback model or a framed estimate. When the framed local estimate cannot read a content block, the client gets the provider's error instead of a 500. Adds /audit round 2's integration tests for the fallback leg's converted body, the Anthropic envelope on fallback failures, and both framing cases.
* fix(count_tokens): answer the provider's error whenever a framed estimate raises
A tool schema the local estimate cannot format, such as a boolean subschema or an array with no items, raised AttributeError or KeyError and answered a 500 on a framed deployment whose provider had already answered 400. Any estimate failure now answers the provider's own error.
* fix(count_tokens): load the custom tokenizer inside the guarded framed estimate
A framed deployment whose custom_tokenizer could not load answered a 500 or 503 where the provider had already answered 400, because the tokenizer was selected outside the estimate's error handling. Selecting it inside answers the provider's error, logs a warning with the estimate's error, and still raises the Rust extension's fork-guard errors.
* fix(count_tokens): look up framing under the deployment's custom_llm_provider
* fix(count_tokens): share one tokenizer download per worker and bound the wait after a provider error
A framed deployment whose custom_tokenizer is not cached locally made every
count wait out the Hugging Face hub's whole retry cycle when the hub could not
be reached, and a burst of those loads held the default thread limiter that
every count's cost map lookup also needs. Counts on a worker now share one
download per minute, a failed download is reused for the rest of that minute,
and after a provider error a count waits at most 5 s before answering the
provider's error. The swallowed-estimate warning logs only the error type.
* fix(count_tokens): keep one tokenizer download per key running and load the local route per request
* test(count_tokens): type the scripted counter and wait on events instead of wall time where possible
* fix(count_tokens): run framed tokenizer loads on their own capped threads and answer a tokenizer panic with the provider error
Framed loads used anyio's default 40 threads, so a hub that drops packets let about 40 hanging loads stall every count on the worker. Loads now run on a per-worker pool of 4, and a count that finds 4 loads running answers the provider's error at once. Any finished load is reused for a minute after it finishes, so a load that fell back to tiktoken is not retried on every count. A pyo3 PanicException from the tokenizer, while loading or counting, now answers the provider's error instead of a plain 500
* feat(proxy): count tokens with token_count_model in its pinned region
token_count_model {model, region} replaces token_count_fallback_model. Bedrock Sonnet and Haiku 5.5 count with Opus 5.5 in us-east-1, or Melbourne for au. profiles, and a failure names the region to enable
* fix(count_tokens): use the renamed count model constant in the e2e test and list token_count_model with the object keys
* fix(proxy): keep the upstream error envelope when adding the region hint
* test(proxy): annotate the token count setting echo test
* ci(cost map): require non-blank token_count_model strings and test the region rules
* fix(count_tokens): answer the token_count_model's error even when framing is set
* fix(count_tokens): keep the provider error for Google contents instead of a framed estimate
* fix(count_tokens): estimate Google contents the route translated to messages
* fix(count_tokens): keep the provider error for Google contents with non-text parts
* test(proxy): drive tokenizer wait and cost map tests by events and a skippable loop clock
* fix(count_tokens): check Google text-only parts without a KeysView comparison basedpyright flags
* feat(proxy): count Bedrock Kimi K3 tokens exactly with its pinned tokenizer
* test(bedrock): keep the Kimi K3 image check in-process
* fix(count_tokens): join the region hint to an upstream message that already ends with a period
Also pins that a blank token_count_model in config falls through to the cost map default
* fix(count_tokens): count Gemini tool-call turns and Responses tools, and stop tokenizer retries from stalling counts
The Google countTokens route estimates contents with functionCall, functionResponse and thoughtSignature parts again, and still answers the provider error for media, thought and unknown parts
The Kimi K3 exact count renders Responses API and Anthropic tools through the chat tool translation the Vertex count already used, now shared from the prompt factory, and falls back to the estimate when the Converse body cannot carry a tool
A failed Kimi K3 tokenizer load is retried in the background without making counts wait, and a load still running after the wait logs a warning
* fix(count_tokens): count Kimi K3 tools the way Bedrock inference sends them
A flat Responses API function or custom tool becomes the chat tool the
Responses route sends to inference, and every other tool goes to the
Converse body as is, so hosted tools Bedrock drops are left out of the
count instead of being counted or turning it into an estimate.
* test(count_tokens): annotate new token counter test locals with Final
* fix(count_tokens): keep shipped count regions inside the deployment's geography
A plain Bedrock Sonnet or Haiku 5.5 deployment in a non-US region no longer sends its count prompt to us-east-1. Global count models and configured token_count_model keep their pinned region. Also expect litellm_count_source in the Vertex Llama count test main added
* fix(count_tokens): resolve count settings off the event loop
* fix(bedrock): send native structured output first on Opus 4.8 and Sonnet 5 and fall back to the json tool when Bedrock rejects it
* fix(bedrock): retry with the json tool when Bedrock refuses the native output schema
Count a 400 under output_config.format.schema: as a native output rejection, so a schema Bedrock cannot compile (like a pattern with a nested {n,m} group) gets the json tool retry instead of reaching the caller
* fix(bedrock): make the native output retry send what an opted-out deployment sends
The retry after a native output rejection now reruns the same optional param
mapping completion() ran, with the deployment model_info opted out of
supports_bedrock_native_structured_output, so tool_choice, effort and the json
tool schema match an opted-out deployment instead of a hand-built tool. Each
config keeps only the predicate that decides when native output came from the
opt-in key.
Bedrock batch records for Anthropic models map with the opt-in turned off,
since a batch job has no retry. Models on supports_native_structured_output
keep native output in batch
* fix(bedrock): build the messages retry from the caller's request and keep other errors unchanged
* fix(bedrock): give sync converse stream errors the upstream body as text
* fix(bedrock): let a malformed caller outputConfig reach Bedrock instead of raising
A Converse json_schema request whose caller outputConfig failed litellm's
shape check raised a pydantic error before the request was sent, so the
caller got a 500 where the merge base passed Bedrock's 400 through. The
opt-in check now treats that outputConfig as carrying no native output.
Also opts the vertex wire test's pinned Opus 4.8 json tool deployment out of
native output, and extends the Bedrock native output integration tests.
* fix(bedrock): send a refused caller outputConfig once and give opt-out advice only when native output is unsupported
A caller can send its own outputConfig on Converse. When Bedrock refused that
schema on an opt-in deployment, the retry still carried the caller's textFormat,
so Bedrock got the same request twice and the warning told the operator to turn
off an opt-in that was not the cause. The retry is now skipped when the opted-out
request would still carry a native textFormat, so Bedrock's 400 reaches the
caller after one call as on the merge base. The opt-out advice is logged only for
the account-level "Extra inputs are not permitted" rejection.
* fix(bedrock): keep the caller's continuation message on the native output retry
The Converse transform pops user_continue_message from litellm_params on the first attempt, so a retry built from those params afterwards dropped it. Both Bedrock chat routes now hand the fallback a copy taken before the first attempt.
* test(bedrock): pin additional_drop_params on the /v1/messages native output retry
* fix(bedrock): give the Converse native output retry the fake stream timeout
* fix(rag): use team provider credentials and allow viewer and JWT team access
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rag): assert persisted managed store data excludes team credentials
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(jwt): run real team model checks in the route denial test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rag): scope team provider credentials to openai vector stores
Bedrock and other providers kept their pre-PR credential and region
resolution: a team bedrock deployment's region no longer overrides the
caller's region on RAG query. Query tests now fake the provider at the
HTTP boundary with a real vector store registry.
* fix(rag): use only the team's own OpenAI deployment for RAG credentials
* fix(rag): ignore caller credential fields the query search never uses when picking team credentials
* fix(rag): treat blank credential fields as absent when picking team credentials
* test(rag): delete the stores the integration ingests create before the owned proxy exits
* fix(proxy): let a restricted key use the team deployment its wildcard or model covers
A key restricted to gpt-4o on a team whose own OpenAI deployment is openai/*
never resolved that deployment's credentials, because the key allowlist was
matched against the deployment's public name literally. The allowlist now
matches a wildcard on either side the way request authorization does, through
the auth layer's own helper, so /rag, /files and /batches resolve the team
deployment for every key that may call a model on it.
* fix(batches): read a restricted key's batch input through its team deployment
The batch access check downloaded the input JSONL with the proxy's OPENAI_API_KEY while the upload and the create used the team deployment, so a key restricted under a team openai/* deployment could not create a batch when the two keys belong to different OpenAI projects. The read now resolves the team deployment the way the create does when no model pins it
Adds the audit cells for keys restricted under the team wildcard on /rag, /files and /batches, and for the rag-vs-chat split of a key restricted to a global exact-name deployment
* fix(rag): load deployments oldest first, reject file_id on stores that cannot take one, and show ingest errors in the dashboard
* fix(rag): pick the team deployment with the lowest id, return 400 for caller mistakes, keep the ingest id in errors, and show partial success in the dashboard
* fix(rag): pick the team's oldest OpenAI deployment so adding one never moves the team
* fix(rag): keep config file team deployments first and try JWT teams in sorted order
A team's config file OpenAI deployments now win in file order, as on the
base, before its DB ones oldest first, so upgrading never moves a team that
has both. A JWT in several teams tries them in sorted order, so every worker
resolves the same team.
* fix(rag): return 400 for Bedrock, S3 Vectors and Gemini store config mistakes on ingest
* fix(rag): keep forwarded provider keys on /rag/query and mixed JWT team claims working
A forwarded x-api-key now counts as a caller credential, so the /rag/query
search uses it over the team deployment as at the merge base. Sorting the
JWT team claim compares entries as text, so a stray number or null no
longer turns into a 401. A Gemini store with no proxy key is a 500 again,
and a non-string vector_store_id keeps the merge base's 400 message.
* fix(rag): let a forwarded provider key win over team credentials on files, batches and ingest
* fix(rag): keep team credentials when a request carries its own key or api_base
A forwarded x-api-key or a body api_key now replaces only the key of the team
deployment, keeping its api_base, as chat completions does. A body
litellm_credential_name no longer turns team credentials off on /v1/batches.
* fix(batches): keep the team key on the team host when a request sends only api_base
A request api_base now replaces the team deployment's api_base only when the
request also brings its own api_key, and a non-string api_key or api_base is
ignored, as on the merge base.
* fix(files): ignore a blank request api_key or api_base next to team credentials
* fix(files): answer 400 for a request api_key that cannot be sent as an HTTP header
* fix(rag): honor a body drop_params for a forwarded key that cannot be an HTTP header
* fix(rag): read the request field for drop_params only on multipart ingest bodies
* fix(rag): type the ingest body read for drop_params
* fix(batches): count a model-routed batch's input with the deployment key it is created with
* fix(batches): reject a batch create whose x-litellm-model headers name different models
* fix(rag): treat a null provider and whitespace store credentials as unset when picking team credentials
* refactor(files): type the team deployment the credential resolver ranks
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(openinfer): add OpenInfer as OpenAI-compatible provider
Register OpenInfer in the JSON provider registry with catalog models,
proxy create fields, and unit coverage so requests route via openinfer/
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(openinfer): assert catalog invariants instead of vendor rates
Pinning per-token prices made CI fail when the vendor list changed. Keep
the backup agreement and positivity checks our code owns.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(openinfer): add openinfer to the Rust OpenAI-compatible provider mirror
* fix(openinfer): align catalog with OpenInfer's live /v1/models
Replace the four catalog rows with the two models OpenInfer serves and
publishes a price for (@oi/GLM-5.3-Flash, @oi/Qwen3.5-27B), using its
listed per-token rates and context windows, and price-check them in tests
* fix(openinfer): use OpenInfer's native Responses and Messages APIs, allow tools on unpriced models, price Qwen3.5-9B
Declare /v1/responses and /v1/messages for openinfer, since OpenInfer serves both natively. Let JSON providers declare provider-wide capabilities in providers.json, read through get_provider_info, so tool calls reach OpenInfer for models without a cost map row. Send max_completion_tokens as is, set output caps to each model's context window, mark the three rows as reasoning models, add Qwen3.5-9B at its listed rate, and wire openinfer into models_by_provider so the openinfer/* wildcard lists its models.
* fix(openinfer): let a model's own capability row win over provider defaults in get_model_info
Provider defaults now only fill capability fields the cost-map row leaves unset, so a row or deployment that turns function calling off stays off. Also marks openinfer_models as Final.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): count the whole minor line in the upgrade notice for patch releases
A patch release's notes only list its backports, so v1.105.1 showed
"0 new features, 1 fix, and 0 other updates" even though upgrading from
v1.104.x brings everything in v1.105.0 too. When the latest release is a
patch, also fetch the recent release list and sum the stable releases on
the same major.minor line (skipping rc/dev prereleases and drafts).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): use tuples for release lists and sync the dashboard API types
Satisfies the LIT001 type-discipline gate and regenerates the
latest_release_info description in schema.d.ts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(proxy): cover the bad release-list payload and non-semver latest tag
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): type the release-line Counters so basedpyright sees no unknowns
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): fetch earlier patches of the minor line by tag
Reading the first page of /releases could miss the line's .0 release when
prereleases push it off the page, and one unrelated release with a null
body failed validation for the whole list. Fetch each earlier patch tag
directly instead (v1.105.0 .. v1.105.N-1), skip tags with no stable
release, and treat a null release body as empty notes.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): bound fake-streamed calls by the request timeout, not stream_timeout
A stream=True call that the provider serves as one non-streaming request (Bedrock Converse for json_schema on models without native structured output, tools on some Llama models, ai21) gets no byte until generation ends, so stream_timeout acted as a total-duration cap. The Router now passes timeout and stream_timeout separately to chat completion calls, and completion() applies stream_timeout only when it reads a live stream.
* test(timeout): cover stream_timeout on live streams only and the router's timeout split
* fix(router): cover every fake-stream site and keep the target group's stream_timeout under prompt management
* fix(router): never shorten a fake stream below stream_timeout, and split timeouts on every surface
A faked stream now takes the longer of timeout and stream_timeout, so a caller
who raised stream_timeout to cover a slow fake stream keeps that headroom.
Azure o-series dispatch is mirrored when deciding whether a stream is faked.
text_completion, /v1/messages and /v1/responses through the Router now get
timeout and stream_timeout separately, like chat calls.
* fix(router): report the effective stream timeout, split adapter streams, and cover M365 and Rust messages
* fix(router): report and log the timeout each stream was bounded by
Native /v1/messages and /v1/responses now record the timeout they applied, so
x-litellm-timeout and the cooldown guard read the real limit instead of the raw
request value. A prompt-managed model also passes on only the caller's or its
own timeout, so the target deployment's timeout still wins.
* fix(router): keep the caller's request_timeout on a prompt-managed handoff and report the applied stream limit
* fix(router): give streams answered in one piece the longer limit and accept text timeouts on responses
* fix(router): give /v1/messages and /v1/responses streams answered in one piece the longer limit
* test(integration): cover faked, live and converted stream timeout limits per endpoint
* test(router): regression tests for faked-stream timeouts
* fix(ui): take the dashboard's user role and identity from the server
Adds an internal GET /management/v1/users/me that returns the user id, role
and email the proxy resolved for the calling key. The dashboard now reads
identity from it instead of the session cookie's claims, so its navigation and
role gating always match what the backend authorizes. The layout waits on it
behind the existing loading screen, and a failed lookup clears the session and
returns to login. Sessions that must change their password can also read it
* fix(ui): sign out when the current user lookup fails and wait for it on chat
The dashboard layout now holds the loading screen and signs out when
/management/v1/users/me fails, instead of mounting the shell and looping back
to loading as each consumer refetched the failed query. The chat layout also
waits for the identity so its panels never query without a user id. The
lookup goes through the typed client with the regenerated schema.d.ts
* fix(ui): send the session's auth header on the first user lookup and keep chat closed until authorized
The first lookup runs before AuthProvider registers the custom header name, so it now reads auth_header_name from the session cookie. Chat also waits for isAuthorized, so a failed lookup no longer mounts the shell with an empty user id
* fix(ui): type the dashboard callers of the now typed user id
useAuthorized used to hand back the cookie's user id untyped, which hid six strict type errors in its callers. The id is now string | null, so the production build failed. Callers handle the null case, and query key filters accept the boolean blocked filter
* fix(ui): look up the current user only after the proxy URL is discovered
Pages outside the dashboard shell, like chat and connect, could start the lookup before UI config discovery set the proxy base URL. On a split-origin deployment both attempts then hit the UI origin and a valid session was signed out
* fix(ui): keep a loaded identity while UI config discovery reruns
Pausing the lookup by passing a null token switched the query key, so on the dashboard the identity the layout had already loaded vanished until the shell's own UI config request finished. The key now stays the same and only the fetch waits for discovery
* fix(router): stop a chat-mode Mantle deployment from rewriting the shared Responses routing row
A deployment with an explicit custom_llm_provider and model_info.mode: chat
used to overwrite the shared litellm.model_cost row, moving every other
deployment and plain SDK calls onto Chat Completions until restart, because
the downgrade guard read a double-prefixed key and never fired. The guard now
reads the real key, and a deployment's own mode decides its routing in
responses_api_bridge_check, except on chatgpt and github_copilot where Chat
Completions is dead for responses-mode models.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): keep the bridge-check mode tests in the sharded bridge flag file
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): let a deployment mode pin replace only the shared-row mode so the bridge capability rules still run
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): keep Responses-only backends bridged under a deployment chat pin and satisfy the type gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): give the bridge casts a reason and drop the bare noqa so the type discipline gate passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): keep a responses row with no supported_endpoints bridged under a deployment chat pin
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): read the downgrade guard's mode from the row registration writes
A chat-mode openai/<name> deployment merged into the bare <name> row while the guard read
openai/<name>, so the guard never fired and the shared responses row turned to chat. An explicit
responses/ prefix now also outranks a deployment's chat pin.
* fix(router): let an explicit responses/ prefix outrank a registered chat row
The router now registers a chat-pinned openai/responses/<model> deployment under its own name, so the
lookup found mode chat and the request went to Chat Completions as responses/<model>.
* fix(router): keep the shared row's mode for every deployment pin and honor the pin on /v1/messages
A deployment's mode pin now never rewrites the shared cost-map row in either direction, including the responses/-stripped twin key, so siblings and SDK calls keep routing by the registered mode. The backend key formula goes back to main's, which keeps capability fields off rows that registration never wrote at base. The /v1/messages adapter and handler read the deployment's model_info, so a chat-pinned deployment gets a plain reasoning effort on chat completions.
* fix(router): keep copilot responses opt-in, make unmapped pins order-free, send Mantle auto summary as auto
A responses pin on a github_copilot model or a model missing from the cost map still marks the shared row, since that is how those models opt into the Responses API. A chat pin never marks a row, so registration order no longer matters. Mantle's OpenAI path gets summary auto in place of the auto summary detailed. The Azure o3-pro and o3-deep-research rows drop Chat Completions, which Azure does not serve for them.
* fix(router): skip a non-string model_info.mode when keeping the registered mode
* fix(router): keep chat-pinned Mantle tool calls and summaries on the Responses API
A Mantle row that also serves the Responses API now follows the GPT-5 tool and
reasoning summary bridge rules, since Mantle Chat Completions rejects function tools
on gpt-5.4+ unless reasoning is off and rejects the summary aliases. Mantle chat
gets a dict reasoning_effort as its effort level.
The opt-in auto summary asks each Responses config for its level, auto on Mantle's
OpenAI path, so a client's own detailed summary follows drop_params as at base.
No mode replaces a registered responses mode, and chatgpt and github_copilot rows
take any pin until they are responses, which restores the Copilot chat opt-out.
* fix(responses): ask the target for its auto summary level on the chat bridge and reasoning_effort
* test(integration): run the Mantle region-prefixed v1 base case again
This PR routes that deployment by its own chat pin, so the case passes and no longer needs the skip
* fix(bedrock_mantle): keep web search and reasoning summaries on the Responses API under a chat pin
A chat-pinned Mantle GPT-5 deployment now sends web_search_options and a dict reasoning_effort carrying a summary to the Responses API, which serves them. Mantle Chat Completions drops n above 1 and a reasoning summary under drop_params and answers 400 without it. The /v1/messages adapter probes the bridge with only the client's own summary, so an auto-injected one never moves the request
* fix(bedrock_mantle): check n and reasoning summaries only on the Chat Completions request
* fix(router): keep a non-string deployment mode off the shared routing row
A deployment whose model_info.mode was an int, a list, or an object still
reached the shared cost-map row, so an unpinned sibling on the same backend
stopped routing to the Responses API. The shared registration now drops a
mode that is not a string, and the deployment keeps its own raw value.
Adds integration cells for per-deployment routing across the OpenAI and
Anthropic SDKs, raw httpx, wildcard routes, malformed modes, an upstream
outage, config changes under traffic, and a worker kill plus restart.
* test(integration): find each routing cell's spend row by the response it logged
A bridged call logs its row under the wrapped Responses id, which the caller never sees on a streamed Messages answer, so look the row up by the answer's own ids and the upstream ids for its marker. Int and list modes 500 their own model group on main too, so those cells check only the sibling, and the summary cell sends the auto level Mantle's Responses path accepts
* fix(bedrock_mantle): run extra_body fields through the chat checks
A chat-pinned Mantle request that nests n or reasoning_effort in
extra_body now gets the same 400, drop, and effort-level handling as
the top-level fields. The bridge test injects a MockTransport client
instead of patching HTTPHandler.post.
* test(router): cover nested reasoning_effort routing, messages thinking summaries and per-target auto summary levels
* fix(router): probe a pinned deployment's own API on /health and keep a requested summary beside an extra_body effort
* fix(router): keep a custom-priced deployment's mode pin off the shared backend row on each request
* fix(router): run per-request custom pricing through the shared mode guard so a health probe keeps the row's mode
* fix(router): drop the debug log that wrote a request's model name from keep_registered_mode
* test(router): cover custom-priced chat pins, their health probes and summary aliases beside an extra_body effort
* fix(proxy): pass only the mode pin to a /health probe so a failing probe never cools its deployment down
* test(router): cover failing health probes leaving their deployment out of cooldown, with and without background checks
* test(router): kill the worker that holds burst calls, so the worker kill cell never depends on how the kernel spread them
* fix(router): route a fine-tune the same whichever deployment registers first
Main's register_model now seeds an unmapped fine-tune's row from its base
model's built-in row, mode included, on the first registration. The mode
guard read only the live row, so a responses pin marked the shared row only
when it registered before its siblings. The guard now reads the row the
registration merges onto, the live row or else the row it would seed.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): move the MCP OAuth happy path from a live Linear e2e to an integration test
GitHub Actions now runs unit tests only, and the MCP OAuth happy path needed a
captured Linear browser login to click through a real consent page. The OAuth
flows it exercised are already integration tests against an owned
authorization-server double, except one behavior: the stored per-user token
must be served by a proxy process that never saw the consent, with no shared
Redis to fall back on. That case is now in test_mcp_oauth_flows.py
The e2e test, its owned-gateway helper, its workflow, the mcp_oauth_live marker
and the JUnit gate script only that workflow used are removed
* test(mcp): chat completions run an OAuth server tool with the caller's own stored token
* test(e2e): drop the Linear chat-completion MCP OAuth e2e and its helpers
The behavior it checked, a chat completion running an OAuth server's tool with
the caller's stored per-user token, is now an integration test in
test_mcp_llm_endpoints.py that needs no saved Linear browser login
* test(mcp): decode the stored per-user credential as urlsafe base64
The plaintext check used b64decode, which drops the - and _ of a urlsafe value
and raised on roughly one ciphertext in five
* test(mcp): drop the chat OAuth integration test that duplicates test_mcp_slots F2-chat
security/test_mcp_slots.py already sends a per-user OAuth token through
/v1/chat/completions and asserts the peer receives it. Every mutant the new test
killed was also killed by the existing suite, so it added no signal
* fix(router): read an application profile's base model from metadata on a cache hit
The 1h prompt-cache pin check read base_model only from the top level of
litellm_params. On a response-cache hit the params it sees carry the base
model only under metadata.model_info, so an application-profile ARN
resolved to itself and the pin was written for 5 minutes while the wire
carried a 1h TTL. Read it through get_base_model_from_metadata, which
also checks metadata.model_info
* test(router): pin top-level base_model precedence over metadata model_info
* fix(count_tokens): answer 400 for Vertex partner models without a count API and OpenAI-shaped provider errors on /v1/responses/input_tokens
* fix(count_tokens): answer 400 for Vertex Mistral counts and type the input_tokens error tests
Vertex has no count-tokens route for Mistral or codestral models, so they now
get the same 400 as other partner models without a count API instead of a 404
that points operators at IAM.
* fix(count_tokens): unwrap prefixed provider errors on /v1/responses/input_tokens
* fix(count_tokens): say LiteLLM, not Vertex, cannot count a Vertex partner model
* fix(count_tokens): count a Vertex partner model with its configured custom_tokenizer and unwrap prefixed flat provider errors
* fix(count_tokens): keep the 400 for contents-only Vertex partner counts with a custom_tokenizer
* fix(count_tokens): pass through an OpenAI provider error that omits param or code
* fix(count_tokens): keep the status-based type for Anthropic count errors on the Responses route
* test(count_tokens): cover the Vertex partner count gate and Responses count error shapes on a real proxy
* test(count_tokens): add regression tests for the Vertex partner count gate and Responses count error shapes
The 48 MB budget on the third batch kept failing on the x86 release gate with no
leak behind it. Isolated rc/1.107.0 and v1.105.1 stacks both grew about 60 MB on
the third batch with gc-tracked objects flat, and both levelled off to a couple of
MB per batch after twenty batches. Measure batches 3-4 against a 200 MB per-worker
ceiling instead, which a native leak of 400 KB per failed request still trips. The
stored-request size check stays the deterministic catch for the v1.100.0 leak
The integration conftest raised UsageError from pytest_collection_modifyitems when a
shard-selected file collected nothing. Under xdist that ran on the workers after the
unfiltered item list was already published, so the worker ended its session with tests
still queued and the controller hit xdist's crashitem assertion, naming an unrelated
queued test and hiding the real ImportError.
A selected file that collects nothing now produces a failed collection report instead.
Files whose own collection already failed keep pytest's real error. Serial runs stop with
"Interrupted: N errors during collection", xdist runs report the error and exit 1, and
both still fail the shard.
WorkerKill returned once the victim's main thread was a zombie. On Linux the
process keeps its sockets open until its last thread exits, so a pooled
keep-alive connection to the killed worker still looked idle and the next
request through it got ECONNRESET. A zombie now counts as gone only once
num_threads() is at most 1. Where psutil cannot read threads of a zombie
(macOS) it raises ZombieProcess, a NoSuchProcess, so the old behavior holds
#46484 moved the litellm_redis_circuit_breaker_* metrics behind service_callback prometheus_system, so the chaos proxy, which only set callbacks prometheus, stopped exposing them and the breaker assertion read 0
* test(integration): run the bedrock mantle prompt cache and output limit wire tests on their own proxy
These two files register deployments with their own model settings on the
gateway. They now boot a module-scoped proxy on a scratch database, so their
setup cannot affect other tests that share the worker's proxy
* test(integration): seed the isolated gateway with the same router settings as the worker gateway
* fix(spend_logs): persist user email in spend log metadata so it shows in UI request logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_logs): update payload expectations for user email metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(gcs_pubsub): add user_api_key_user_email to the spend log fixture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_logs): drop untyped metadata key args from the debug log to keep the type gate flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_logs): pass metadata keys to the debug log as a lazy %-style argument
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(spend_logs): store user email in spend logs only behind store_user_email_in_spend_logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(spend_logs): store user email in spend logs by default, drop store_user_email_in_spend_logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_logs): store an empty user email as null, narrow the expected payload test diff
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: Add failing test case
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Restamp nested response.model in streamed Responses events
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* chore: Add tests
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* chore: Run format and lint, and fix
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Type nested Responses restamp to satisfy basedpyright gate
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix(proxy): restamp the nested model on streamed response.queued events
Background streams send response.queued as a GenericEvent, so its nested
response.model still carried the backend name. Also drop the new comments
* fix(proxy): restamp a nested response kept as a dict on streamed lifecycle events
* test(proxy): drive typed events through the nested restamp no-op test
* fix(proxy): restamp the nested model on guardrail block frames in streamed responses
* test(proxy): assert a cache-polled background response reports the requested model
* fix(proxy): restamp nested model on dict events and on every SSE data line
---------
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
