* fix(guardrails): check Anthropic-format request tools in tool_permission pre_call
The pre_call hook only collected tools shaped as {"type": "function", "function": {...}},
so flat tools from /v1/messages ({"name", "input_schema"}) and /v1/responses
({"type": "function", "name"}) were never checked and deny rules silently passed.
Reuse the shared anthropic_tool_names helper so every request tool name is checked,
and handle Anthropic tool_choice {"type": "tool"} in both block and rewrite modes.
Fixes#40583
* fix(guardrails): detect Responses API named tool_choice in tool_permission
Responses API named choices put the name at the top level ({"type": "function", "name": ...}),
so rewrite mode removed the denied tool but left a tool_choice pointing at it.
* test(guardrails): run tool_permission pre_call tests through real default_on gating
Build the guardrail as an always-on pre_call guardrail instead of patching should_run_guardrail.
* test(guardrails): type the new tool_permission pre_call test parameters
* fix(guardrails): keep the Responses API custom tool type distinct
Anthropic's user-defined tools carry type "custom", so pre_call mapped
that to "function" for tool_type rules. The Responses API has its own
custom tool type, and mapping it let a custom tool slip past a
tool_type deny rule when the default action is allow.
Normalize only for the formats where custom means a plain function, and
read tool names directly in the rewrite path, which never needed a type.
* test(guardrails): cover flat tools through Messages and Responses
Add blocked and permitted flat-tool cases through the running proxy, with
endpoint coverage and metadata for the current e2e harness
Require a guardrail policy verdict when checking denied tools so an
unrelated provider error cannot satisfy the regression test
Co-authored-by: Christopher Krah <christopher.krah@plexito.de>
* fix(guardrails): check namespace and nameless tools and relax tool_choice in tool permission rewrite
* fix(guardrails): check forced, grouped, loaded, and hosted tools by their real type
* fix(guardrails): drop references to removed tools and count loaded tools in tool permission rewrite
* fix(guardrails): check tools sent outside top-level tools in tool permission
* fix(guardrails): judge namespace members by their bare name as a fallback and keep rewrites valid for Gemini, Bedrock and hosted tools
A namespace member is checked by namespace__name first, and by its bare name when no rule names the joined one. Rewrite mode now prunes Gemini allowedFunctionNames and turns off forced calling when no function is left, drops a removed hosted tool from allowed_tools, reads Bedrock toolSpec and systemTool blocks by name, skips cachePoint markers, and leaves a tools field that is not a list for the provider to reject instead of failing with a 500
* fix(guardrails): let only name rules decide a namespace member's bare-name fallback
* fix(guardrails): judge malformed tool types, empty Gemini name lists and pass-through Anthropic tools like the endpoints do
A tool or tool_choice whose type is not a string now reads as untyped instead of crashing the guardrail with a 500, and a null type key is never read as a Gemini tool part. An empty allowedFunctionNames list no longer turns off the functions a rewrite kept. An Anthropic custom tool reads as a function by its input_schema on every route, so a pass-through route judges it the same as /v1/messages. A Responses WebSocket frame now forwards the extra_body tools a rewrite stripped
* fix(guardrails): judge a named hosted tool by its type and keep a nested chat custom tool custom
* fix(guardrails): keep judging Anthropic client tools and type-named server tools by their name
* fix(guardrails): judge a named tool by its name alone only when the name is its type's own
* fix(guardrails): judge the function a tool names beside a declaration list or a top-level name
* fix(guardrails): judge Gemini hosted keys whatever the tool type and a namespace's function
* fix(guardrails): judge legacy functions entries like tools and send Gemini a namespace's members
* fix(guardrails): judge a namespace member by every name a provider reads and send Gemini its schema converted
* fix(guardrails): judge a legacy functions entry as a function and relax a tool_choice in extra_body
* fix(gemini): keep a namespace whole when inlining its members would repeat a function name
* fix(guardrails): judge a namespace member by every bare name that shares its qualified name
* fix(vertex_ai): declare a repeated namespace member name once instead of sending namespace names to Gemini
* fix(guardrails): judge a tool_choice sent in extra_body like a top-level one
A forced name only extra_body.tool_choice carried was never checked, so block mode let a denied name through to the provider
* fix(guardrails): judge a Gemini namespace member returned by its bare name and force a named namespace's members
* test: keep formatter churn out of the tool_permission and Gemini test files
* fix(guardrails): judge extra_body.input tools, argument-aware alias rules, and exact forced Gemini functions
Tools loaded in extra_body.input are now checked and stripped like top-level input. A returned
namespace member is judged by the alias whose first name rule its arguments satisfy. A forced
Gemini name that matches a top-level function keeps that function. Alias grouping and Gemini
declaration dedupe now run in one pass instead of quadratic scans.
* fix(guardrails): judge and rewrite every tool field sent inside extra_body
extra_body is merged into the upstream body, so functions, function_call,
mcp_servers and toolConfig nested there reached the model unjudged. Treat
extra_body as a second request layer that is collected and rewritten the
same way as the top-level request.
* fix(guardrails): judge Gemini hosted keys by key and count only the tools a provider receives after extra_body
* fix(guardrails): relax a tool_choice that names a dropped namespace member by its bare name
Gemini and Anthropic chat send namespace members under their bare names, so rewrite mode now treats a dropped member's bare name, and an emptied namespace's own name, as removed when no kept tool still carries it
* fix(gemini,guardrails): keep a top-level function over a same-named namespace member and leave malformed tool values to the provider
Gemini and Vertex chat now declare a top-level function over a namespace member that shares its name, so a forced call gets the top-level schema. post_call judges a returned name a top-level tool carries as that tool, and pre_call no longer raises on an unhashable or non-string tool name, input item type, or function_call name.
* fix(guardrails): judge a returned namespace member named like a hosted MCP server as that member
* fix(gemini,guardrails): send a namespace as main did when another tool has a member's name
Gemini chat inlines a namespace's members only when no other tool, namespace
member, or legacy functions entry declares the same name, and otherwise sends the
namespace by its own name as main did. A tool_choice naming such a namespace
forces the namespace itself. The tool permission post_call reads top-level names
the way Gemini declares them and judges a call the Responses bridge tags with a
namespace as that member.
* fix(gemini): send colliding namespace members as namespace__name functions and judge them in post_call
When a namespace member shares a name with another tool, Gemini chat now gets that namespace's members as namespace__name functions instead of one function named after the namespace, so every function it can call is a name the rules judged. Declared names are counted once, and a forced member maps to each holder's qualified name.
post_call judges a returned namespace__name as that member, by its qualified or bare name the way pre_call does, so a bare-name allowlist or denylist applies to it.
* fix(gemini): keep generated member names unique and let a top-level function own its exact name
A namespace that listed a member twice, or a generated namespace__name that
repeated a top-level function, sent Gemini two declarations with one name, and
Gemini answered 400 where main answered 200. A name a namespace lists twice is
now one member, and a top-level function keeps its exact name.
post_call now judges a returned name a top-level function declares as that
function, never as the namespace member the name could also mean.
* fix(gemini): qualify namespace members by the namespace's top-level name
The guardrail judges a member as `name__member` from the namespace's top-level
name. Gemini built the name from `function.name` when the tool also carried one,
so a request could pass pre-call and reach Gemini under a name no rule judged.
* fix(vertex_ai): force only the namespace members Gemini is sent
A forced namespace or member on Gemini chat could still allow a generated name that a top-level function or an earlier namespace owns, so tool_choice forced an unrelated function. The forced names now come from the same collision checks the declarations use
* fix(vertex_ai): reject a forced namespace Gemini gets no member of, and resolve forced names in one pass
* fix(guardrails): judge a namespace member by a rule on its namespace's name
---------
Co-authored-by: Christopher Krah <christopher.krah@plexito.de>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(e2e): retire the quiet_stack lock
quiet_stack held an exclusive host-wide lock for six tests, draining every
other xdist worker while they ran. On the full e2e lane that left 35-38% of
worker time idle. Each of those tests now lives where it can run without
pausing the suite
- the two memory tests move to perf/ behind the perf marker, so the perf lane
runs them on its own fresh stack. Their refusing deployments use a literal
key because the perf stack carries no provider keys
- the stored-request size check from the memory test becomes an integration
test on an owned proxy that stores prompts
- the discount and margin spend-row tests are dropped from e2e. Their
breakdown assertions join the existing integration pricing test, which
already owns the discount and margin repricing flow
- the Bedrock guardrail cost test stays in e2e without the pricing reset
fixture, since nothing else on the stack writes pricing config now
- the MCP semantic tool filter test turns on a proxy-wide setting, so it gets
a new proxy_wide_config opt-in marker and runs after the main run
* test(e2e): register the claude_code deployments as one batch
The claude_code session fixture registered its 24 deployments one at a time on every xdist worker, and each one waited out the 15 s propagation window on its own, about 6 minutes per worker. ProxyClient.register_models posts the whole batch, polls each model until every replica serves it, then waits out propagation once from the last write. A model that fails to post or never becomes servable comes back as ModelNotRegistered, and an unservable one is deleted, so the fixture keeps reporting per-deployment failures the way it did
* feat(ui): rebuild the public AI Hub as a searchable list with a details pane
The public AI Hub at /ui/model_hub_table (and the older /ui/model_hub) now
renders PublicAiHub, built from shadcn parts only: a header with the
gateway's base URL and the admin's links, segmented tabs for models, agents,
MCP servers and skills, and a list beside a details pane on desktop that
becomes a bottom sheet on phones. The open item and tab live in the URL.
It keeps every feature the old page had: server-side search, provider, mode
and feature filters, sorting, paging, health, rate limits, prices, custom
metadata, supported params, usage snippets, the require-auth redirect and
the not-enabled state. cURL shows only for chat models, since the old one
called chat completions for every mode.
The navbar hides the community buttons below the sm breakpoint on every page,
and the public navbar uses a smaller logo on phones, so Docs and Blog no
longer collide with them.
* fix(ui): harden the public AI Hub after adversarial review
- Show "This AI Hub isn't available" only when every hub call fails; a
failed models call now shows an inline error with Try again and leaves
agents, MCP servers and skills usable (a gateway with no models answers
the models list with 400)
- Row links keep the page's other query params, so open-in-new-tab and
copy link no longer 404
- A shared ?open= link resolves models beyond the first page, keeps the
opened item when a search filters it out, and says when it is no
longer listed instead of showing a different item
- Searching, sorting and paging no longer blank the details pane or drop
the Operational badge
- Key MCP servers by their listed name, which is never null
- Keep sub-cent prices and show Free for zero cost, fold evaluation into
Decisions, sort agents, MCP servers and skills by name, restore
word-wise search, and show a skill's source, author and date added
- Drop the invented description for models without a mode, the empty
agent modes row, the stray auth dash on MCP rows and NaN dates
- Do not retry the UI config call, and close the 1023-1024px gap between
the sheet and the details pane
- Update the public hub Playwright spec for the new tabs and list, and
prune the lint suppressions the old route pages needed
* fix(ui): surface hub load errors and keep the open item steady
Agents, MCP servers and skills now load through apiClient so an HTTP error
shows the inline error instead of an empty "published yet" state, and an
all-failing gateway reaches the unavailable page. Try again shows loading
instead of an empty hub, the open model stays in the details pane when a
search or page change hides its row, a zero-result search keeps the pane,
phones get the not-listed sheet for a stale link, an empty open param means
nothing opened, free models read Free in the list, and an invalid skill date
is hidden
Replace the hand-rolled combobox in the Cmd/Ctrl+K palette with shadcn's
base-vega Command and CommandDialog (cmdk) and bind the shortcut with
react-hotkeys-hook. cmdk now owns arrow keys, wraparound, Home/End and IME
composition, Base UI keeps the dialog mounted for its closing animation and
returns focus to the trigger, and mod+k is ignored while focus is inside
another dialog
Results keep their fixed Keys, Actions, Pages order with shouldFilter
disabled. The highlight resets to the first row whenever debounced key
results load, so Enter after typing still opens the top key
cmdk 1.1.1 leaves aria-activedescendant empty or stale after it picks a row
on its own (dip/cmdk#413), so a MutationObserver keeps it on the row that is
actually aria-selected until upstream fixes it
* feat(realtime): support latest OpenAI audio models
* fix(transcription): type SDK stream duration metadata explicitly
* fix(realtime): preserve transport settings and translation duration
* refactor(realtime): reuse SDK client initialization
* fix(audio): bill finalized streams and authorize transcription models
* refactor(audio): keep transcription pricing selection together
* fix(realtime): reconcile model metadata and test contracts
* fix(realtime): preserve provider duration and model capabilities
* fix(realtime): keep transcription validation compatible with cost map guard
* fix(realtime): address transcription and billing review findings
* fix(realtime): preserve input usage and provider routing
* fix(realtime): bill retained output and route Azure aliases
* fix(realtime): reject malformed provider durations
* fix(realtime): preserve batch cache rates and Azure v1 routes
* fix(transcription): reject nonfinite provider durations
* refactor(realtime): align annotations with current lint rules
* fix(realtime): preserve translation deployment pricing and input usage
* test(transcription): verify synchronous stream finalization
* fix(realtime): retain missing translation input usage at close
* fix(transcription): reject streams that bypass output guardrails
* fix(realtime): reconcile audio protocols billing and current CI
* fix(realtime): trust confirmed formats and authorize default models
* fix(realtime): preserve absent models and reject unsupported guardrails
* fix(realtime): reject translation policy pipelines
* chore(ci): rerun transient infrastructure timing failure
* fix(realtime): reconcile audio protocols with current upstream
* fix(audio): validate provider responses at SDK boundaries
* fix(realtime): preserve minimum OpenAI SDK compatibility
* fix(realtime): preserve deferred Live session authorization
* fix(realtime): preserve upstream authentication and transcription behavior
* fix(audio): scope provider duration precedence to configured models
* fix(typing): avoid Final declarations inside transcription cost loop
* refactor(audio): reuse transcription mapping and duration resolution
* fix(realtime): enforce model budgets and preserve translation routing
* fix(audio): log streamed transcripts whose done event reports detected languages
* fix(audio): pass stream, languages and keywords through unchecked for non-OpenAI transcription providers
Providers without a streaming transcription handler got UnsupportedParamsError
for stream=True, languages and keywords, where the merge base forwarded them as
provider-specific params. Only openai and azure validate them now.
* fix(audio): bill streamed gpt-transcribe from the provider's reported duration
The stream path computed cost after logging had replaced the transcription's
duration usage, so it fell back to the measured file length while the
unstreamed call billed the provider's seconds.
* fix(transcription): log streamed transcription usage from the provider response
The stream logging path built the spend payload from the response after
its duration usage was transformed away, so whisper-1 with stream=true and
streamed gpt-transcribe logged an empty usage object. Bill and log from the
provider's TranscriptionResponse in both the sync and async paths.
* fix(realtime): keep main's routing, protocol and transport behavior around the new translation routes
- Send OpenAI realtime and translation client_secrets and calls through litellm's httpx client again, so ssl_verify and custom CA settings apply and upstream errors keep main's shape
- Use the input transcription model as the session model only for transcription sessions
- Drop the new /openai/v1 translation aliases and record each lazy feature's own routes, so the /openai pass-through and the MCP routes resolve as on main
- Let a configured Azure realtime_protocol win for every model, with GA as the default only when none is set
- Return x-litellm headers on streamed transcriptions and mark the request done
- Refuse translation sessions before connecting upstream when realtime guardrails apply or the provider is not OpenAI or Azure
- Accept json and text for gpt-transcribe, and drop other formats under drop_params instead of failing
- Bill translation sessions on output audio seconds only, at the single published rate
- Document the 201 SDP answer on the translation calls routes and the transcription 200 response
* fix(realtime): restore main behavior for guardrails, Azure transcription, and realtime clients
Streamed transcription runs unstreamed when a deployment post_call guardrail applies, so the
guardrail still sees the transcript. Azure transcription maps v1, latest and preview to the
default dated version on the deployments route for every model. The SDK client secret keeps
session.model ahead of model, realtime WebSockets connect to api_base again, realtime metadata
keeps only guardrails, and managed agents need an explicit transcription model. Reverts the two
unrelated streamed Responses error codes and the duplicate cached_tokens_details field, and the
gpt-transcribe format check only reads the request provider's cost map entry.
* fix(realtime): keep main's transcription, client secret and route behavior where the PR changed it
Transcription reaches every provider as on main again, sends stream only when asked, forwards language and
languages together, keeps form key order, bills provider seconds only when above zero, honors base_model for the
new flags, keeps main's Azure api_version on the proxy path, and streams carry chat's headers, an error event on a
mid-stream failure, and a spend row when the stream ends early. Translation cost reads both per-second rates from
one source and spend rows carry the billed seconds.
Realtime transcription WebSockets keep model in the upstream URL outside OpenAI's hosts, client secrets forward
explicit nulls, accept conversation and delay, and answer 400 on a malformed body, the /openai translation paths
are back to main, translation refuses only for guardrails realtime sessions run, nested aliases resolve and later
session.update models are authorized, translation client secrets run per-model auth, beta clients bridged to GA
get beta-shaped session events, and transcription sessions send the deployment model in both shapes.
* fix(realtime): address review findings on transcription billing, streaming and realtime auth
Per-second pricing replaces token billing only when the priced entry sets no token rates, translation rates fall back per key to the cost map, transcription cache keys include languages and keywords, a stream request a post_call guardrail forces unstreamed comes back as redacted SSE, Azure beta clients are bridged only for GA-only deployments, nested transcription models on the realtime WebSocket are checked against the key, and translation spend carries key and team metadata
* fix(azure): pick the GA realtime protocol for deployments whose model_info.base_model is GA-only
* fix(realtime): preserve prior merge resolutions after rebase
* fix(realtime): enforce translation policies and normalize Azure transcription URLs
* fix(cookbook): stop translation once output idles after the upload ends
* fix(realtime): bill translation on input audio and drop pre-read stream cost headers
Translation spend now uses the input audio seconds the session sent, priced
through input_cost_per_second, instead of the translated output duration.
A streamed transcription no longer returns cost headers before the stream is
read, since the cost is only known after the last event. Also removes an
unused realtime guardrail helper.
* fix(realtime): drop top-level session nulls, bill dropped transcription streams, and bridge beta response fields
Client secret, translation and transcription session bodies drop top-level nulls again, as main did, while nested nulls such as turn_detection still pass through. Transcription client secrets send the deployment model as session.model. A streamed transcription the client drops still runs its success callbacks, and the stream ends with data: [DONE] like OpenAI's. The beta to GA bridge maps max_response_output_tokens, speed and noise reduction, remaps response.create, and drops temperature under drop_params. The Entra token for realtime HTTP routes is fetched off the event loop
* test(e2e): cover realtime translation, live transcription, client secrets, and streamed transcription
* fix(realtime): close review gaps in translation billing, transcription streams, and Azure GA
Bill translation sessions from the deployment's own per-second rates, add per-second prices for
Azure gpt-4o transcribe models, keep stream-scoped guardrails seeing the caller's stream flag, map
mid-stream transcription errors to LiteLLM exceptions with a [DONE] frame, map GA responses back to
the beta shape for bridged beta clients, honor the minting key's guardrail opt-out on translation
calls, restore the Azure transcription session URL, and flag GA-only Azure realtime models
* fix(realtime): bill a translation rate the deployment omits from the session model
* test(e2e): check translated speech is not English instead of exact French wording
* fix(realtime): keep main's behavior for non-bridged backends, stream error frames, gateway URLs and Azure transcribe prices
Remap beta events to GA only for bridged beta clients and translation sessions, bridge beta clients on GA-pinned Azure deployments, log error-only upstream sessions as failures, flag dated Azure GA realtime models, drop the unpublished Azure per-second transcribe rates, log a mid-stream transcription failure once, keep main's SSE error frame outside transcription, drop unsupported transcription params under drop_params, keep the caller's Azure auth headers, keep a gateway /v1 path, let key opt-outs release string-callback guardrails on translation, and keep the cookbook waiting while a long upload is still sending
* fix(realtime): keep the proxy's realtime session gate off translation sessions
A key that opts out of a legacy callback guardrail now gets a working translation session, since the transcription-guardrail gate no longer sends a realtime session.update the translation upstream rejects
* fix(realtime): keep websocket model limits, bill measured transcription length, route transcription sessions on the allowed model
A WebSocket opened without a query model now authorizes as an empty model, transcription cost prefers the measured upload length over the provider's reported seconds, transcription sessions overwrite nested models with the authorized top-level model again, and a non-boolean transcription stream value answers 400 or is dropped under drop_params.
* fix(ci): copy the rust bridge extension into the openapi docs gate's base worktree
* fix(model-catalog): accept the realtime translation endpoints in the cost map
* fix(cookbook): close the translation session and read until session.closed
* fix(realtime): treat an empty transcription stream as unset and pass the cost-map guard
An empty `stream` form value runs the transcription unstreamed again, as
OpenAI does, instead of answering 400. The four translate cost-map entries
drop the translation routes from supported_endpoints until the main
branch's catalog parser accepts them, and the nested transcription model
lookup uses the router registry entry point.
* test(proxy): drop the mid-stream error frame pin that main's error shape change replaced
* fix(transcription): bill uploaded seconds when a stream fails mid-transcript
* fix(proxy): bill uploaded seconds when a streamed transcription breaks mid-stream
* fix(audio): keep transcription's positional arguments and redact streamed error details
- Move the new languages, keywords and stream options to the end of transcription's parameters, so a prompt or format passed by position keeps its meaning
- Redact internal hostnames, paths and tracebacks from a streamed transcription's mid-stream error event, as chat streams already do
- Move the CA-bundle check for the realtime HTTP routes, which serves real TLS on loopback, from the unit suite to the SDK integration suite
* fix(audio): stream the guarded transcript to SDK callers that asked for a stream
* fix(logging): keep a streamed transcript out of callbacks that turned off message logging
A streamed transcription's assembled response reached success callbacks with its text intact under turn_off_message_logging or a callback's message_logging=False, because the shared streaming redactor had no transcription branch. It now redacts the text, as the unstreamed call already does
* fix(proxy): stream a transcription the deployment streamed when the client sent no stream field
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(e2e): add the Codex file edit and long session rows
The file edit row has Codex change a random value into config.txt with the patch tool and checks the file on disk. The long session row has it follow a chain of random file names one cat at a time. new_workspace now returns the resolved path, because Codex reports patched files by their real path (/private/var on macOS).
* test(e2e): make the Codex long session prove every earlier command output reached the model
* test(e2e): Codex auto-compaction through LiteLLM (#49389)
* test(e2e): add the Codex auto-compaction row
Codex reads a random value, compacts through the gateway with its remote compaction forced on, and is asked for the value after the file holding it is deleted. The row reads the compaction from the thread's rollout file and marks Claude on Bedrock as a strict expected failure, since the Responses-to-chat bridge ignores the compaction_trigger item
* refactor(e2e): validate Codex rollout lines with the models directly
* test(e2e): drive the real Codex CLI through the gateway and add the command round trip row
Adds tests/e2e/codex: a harness that runs codex exec --json with a throwaway HOME and CODEX_HOME, a custom responses provider pointed at the gateway, the model list read from the gateway's /v1/models, the sandbox on and approvals off. Each test registers its setup's deployment via /model/new and calls it with a key scoped to it, both deleted on teardown. The first row checks that a value only a command can print comes back through the gateway on all five setups
* test(e2e): type the Codex self-test event helper as a Mapping
* test(e2e): clean up Codex test keys even when setup fails, and fail runs where Codex exits with an error
* test(e2e): record a step when the Codex harness sets up its home and workspace
The suite rules ask every public harness helper that does something to record a step, so a failed run shows whether it died while setting up Codex or while running it
* Update run.py
wip
* fix(responses): log the encrypted response id on streamed /v1/responses spend rows
With response id security on, the stream hook encrypts the resp_ id on every
frame, but logging had already received the SDK's own copy of the terminal
response, so the spend row, the standard logging payload and callbacks got the
managed id (or the chatcmpl id on a bridged stream) the client never saw. The
hook now records the exact id it advertises on the request's logging object,
and the logging layer stamps that id on the logged streamed response
* fix(responses): return the id the client sent on GET and cancel, and keep read rows off the create row's id
Every GET /v1/responses/{id} re-encrypted the response id, so a client that continued a bridged conversation with the
id a GET returned referenced a ciphertext no spend row was filed under. The follow-up lost turn 1 and opened a new
session. GET and cancel now answer with the exact id the client sent when it is an id this proxy issued and it decodes
to the response the provider returned. A create with previous_response_id still gets a fresh id for its own response,
and a raw provider id still gets a managed id
Because a read now carries the create's id, a GET or cancel spend row is filed as <id>_retrieve_<call id> or
<id>_cancel_<call id>. That keeps one row per read, as before, and a read row can never take the create row's
request_id, so the create row survives whichever row is written first and a follow-up replays turn 1 once. The
background settlement row keeps <id>_background_cost, and the cost poll's own status checks keep the response id
* fix(responses): log a fresh id for each read and cancel, echo the held id only to its owner
A GET or cancel of a stored response now logs a new encrypted id of its own,
so callbacks and spend rows never reuse the create's id, while the owner of
the id still gets back the exact id they sent. An admin reading someone
else's response gets an id bound to the admin, as before. The spend row id
suffix for reads and cancels is gone, so with response id security off the
rows and session history match the default branch. A streamed GET follows
the same rules.
* fix(responses): echo the sent id to every reader that passes the response access check
A GET, streamed GET or cancel of a response now answers with the id the caller sent whenever the caller passed
the access check, not only when the caller is the user or team the id was issued to. An admin, the master key
or a teammate who read someone else's streamed response got a fresh id back, and the session lookup could not
reach the first turn from it, so a bridged follow-up through that id lost the conversation history
Every read and cancel still logs a fresh id bound to the reader, never the create's id
* fix(responses): keep a read or cancel row in session history only when it brings new output
A GET, streamed GET or cancel sent with a session header joins the
create's session, and its stored response repeats the create's output.
The replay then sent the same assistant turn once per read.
Session history now keeps an aget_responses or acancel_responses row
only when its stored Responses output has an item id that no earlier
kept row carried. A polled background response keeps the first poll
that returns the finished output, since its queued create row has none,
and later polls of the same output are dropped. Every other row keeps
today's rule
* fix(responses): replay each history item once, echo the sent id on delete, keep background cost files apart
A create row now owns its output items in session history, and a read or cancel row adds only items no
create and no earlier read carried, in one linear pass, so a background response polled while it grows,
or a read that ends before its streamed create, replays the answer once and after its prompt
DELETE /v1/responses/{id} answers with the id the caller sent, on a copy, and logs a fresh encrypted id
like a read does
Azure Data Lake Storage per-event files and unbatched GCS objects are named after the event's spend row
id, so a streamed background create and its cost event, which now share the client's id, keep separate
files
The integration tests that decoded the managed id now match the exact id the client received
* fix(responses): keep main's answer for a raw-id delete and main's storage names outside background cost events
A DELETE by a raw provider id returns the provider's id as main did, and logging still records a fresh
encrypted id. A streamed GET matches the held id by upstream response id, like a plain GET
Only a background response's cost event gets the _background_cost suffix on its Azure Data Lake Storage
file and unbatched GCS object, so batch retrieves and events with no id keep main's names
An item a session read replays takes the newest read's copy at the first read's place, so an item a
poll caught half written replays finished
Tests pin the client's id over the router's continued response id, and the storage test no longer reads
the clock after the upload
* fix(spend): give background cost events their own OpenMeter and DynamoDB ids, replay id-less read items
A streamed background create and its later cost event share the client's
id, so OpenMeter dropped the second event and DynamoDB overwrote the
create's item. Both now key the cost event as <id>_background_cost, like
its spend row and storage file.
Session history again replays a read's output items that carry no id,
as main did.
* fix(integrations): keep OpenMeter and DynamoDB logging working without the proxy extras
* fix(integrations): give background cost events their own ClickHouse request id and DeepEval span id
A streamed background create and its later cost event now share the id the
client received, so ClickHouse and DeepEval saw both events under one id.
Route both through logged_event_key, the way the Azure storage, GCS,
OpenMeter and DynamoDB loggers already do, so the cost event's ClickHouse
request_id matches its spend row's `<id>_background_cost` and its DeepEval
span gets its own uuid. ClickHouse response_id stays the raw provider id.
* test(deepeval): annotate the background cost span test's locals with Final
* test(responses): run the DELETE wire test that pins the echoed client id
DELETE /v1/responses/{id} now answers with the id the caller sent, so the
skip that marked it as a known bug no longer applies. The test passes on
the /v1, /openai/v1 and bare routes.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(e2e): share the CLI version rule, env lockdown and rate limiter in tests/e2e/cli_harness
Moves the npm release rule (now taking the package name), the env allowlist plus fresh HOME, and the per-upstream rate limiter out of claude_code so the Codex suite can reuse them. Claude Code behavior is unchanged, its CI, cron and Dockerfile paths now call the shared resolver with @anthropic-ai/claude-code
* test(e2e): freeze the packument map and note the rate limiter self-test
* test(e2e): race real processes on one rate limiter bucket so a missing lock fails
Eight spawned processes start together and drain a bucket that never refills. With the flock they get exactly its burst between them, without it every attempt is granted
* test(e2e): keep the old Claude Code version resolver path for the Buildkite runner images
The runner Dockerfiles still run tests/e2e/claude_code/pr_gate_version_resolver.py and silently install @latest when it is missing, which drops the three day review window. The old path now forwards to cli_harness/npm_version_resolver.py
* fix(anthropic): sync the beta header allowlist with what anthropic, vertex_ai, and azure_ai accept
* test(e2e): send the azure_ai beta check without the code execution tool
* test(anthropic): require a vertex_ai and azure_ai decision for every anthropic beta
* fix(anthropic): forward the betas Claude Code sends that anthropic, vertex_ai, and azure_ai accept
* test(e2e): send Vertex's header-only betas in the beta acceptance case
* fix(anthropic): keep forwarding Vertex computer-use betas and point beta tests at platforms that still lack them
A live probe found each Claude 4.5 and 4.6 model on Vertex needs one of the two
computer-use betas for the computer tool, so dropping both broke it. The e2e
acceptance test leaves those two out of its all-at-once Vertex call, and the
integration and unit tests that relied on a beta being dropped now use one
the platform still lacks.
* test(anthropic): point unsupported-beta tests at betas the platforms still drop
The Azure AI handler test used claude-code-20250219 as the beta Azure drops, and the
Rust parse test used it as an unknown value. This PR forwards it on Azure AI and maps it,
so both now use values that stay dropped or unknown.
* fix(anthropic): add computer-use-2025-11-24 for the computer_20251124 tool
The computer_20251124 tool fell through to the default computer-use-2024-10-22 beta, which the
provider rejects for that tool version. Map it to computer-use-2025-11-24, which the beta map now
forwards on anthropic and azure_ai
* fix(anthropic): send no computer use beta for an unknown computer tool version
An unknown computer tool version fell back to computer-use-2024-10-22, which the anthropic and azure_ai columns now forward. It now gets no beta, as every platform effectively sent before. Every computer tool version also skips defer_loading and allowed_callers, computer_20251124 included
* test(azure_ai): read the branch's beta map in the auto-injected beta wire test
* fix(anthropic): keep defer_loading and allowed_callers on the computer_20251124 chat tool
* test(anthropic): read the branch's beta map in the azure_ai safeguards beta wire test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Deletes the A2A e2e test that registered a third-party agent, which depended on a site the project does not control, along with its helpers and its coverage registry row. Adds an integration test where a scripted A2A agent receives a skill call as a data part and returns a data artifact, asserting the agent and the caller each get theirs unchanged
* test: move Sail and Weave e2e tests to integration tests
The Sail service_tier and completion_window cases now run against the scripted upstream in tests/integration/pricing, asserting the outbound body and the billed rates for chat, responses and messages
The key-scoped Weave delivery cases now run against an OTLP collector in tests/integration/observability. The case where only the otel callback is configured is skipped with a BUG reason
Removes the e2e copies, the Weave read-back client, the Sail registry rows, the Weave docs paragraph and the unused devin MCP server entry from the stage mirror config
* test: move Presidio masking e2e tests to integration tests
Moving them to integration tests. The pre_call, post_call and default scope cases now run against a mock Presidio analyzer and anonymizer that follow the real wire contract, and a scripted model upstream, on chat completions, messages and responses, with and without streaming
The spend log cases check the stored response and the guardrail record with a proxy that stores prompts in spend logs. The e2e copy, its Presidio guardrail params body, its registry rows and the doc paragraphs naming the two PRESIDIO variables are removed
Phoenix is unused by the e2e suite and has no account behind it. Drops the callback from the stage mirror gateway config, and updates the OTEL trace test's failure message and the e2e contributing notes that still pointed at the arize_phoenix preset.
* test(e2e): add gemini tool call id length repro for LIT-8912
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): add thought_signature_in_tool_call_id to keep tool call ids short for clients that cap them
* fix(gemini): pair a tool result with its call when the client re-adds the thought signature to the call id
Strands, which the AgentCore harness runs, puts provider_specific_fields.thought_signature back on the
assistant tool call id as <id>__thought__<signature> while its tool result keeps the bare id, so the Gemini
conversion found no matching call and answered 500. Ids now match with the signature suffix stripped.
Also adds thought_signature_in_tool_call_id to the litellm_params inventory test and gives the pass-through
tests' mocked logging objects the litellm_params a real Logging object always has.
* fix(gemini): keep the thought signature on streamed Messages and Responses tool calls and accept a string setting
* fix(gemini): read the thought signature back from Responses input and coerce the per-request setting
A Responses function_call input item now keeps its provider_specific_fields, so a
client that sends the item back gets its real thought signature replayed to Gemini.
A per-request thought_signature_in_tool_call_id of "false" or 0 now keeps ids short,
and a value that is not a boolean answers 400, or is dropped under drop_params,
instead of a 500.
The e2e test now uses the sdk fixture and @meta subjects
* fix(caching): key the response cache on thought_signature_in_tool_call_id
* fix(responses): send thought signature fields only with short ids and drop malformed routing kwargs whole
* fix(main): validate routing kwargs after the locals snapshot so fallbacks and model_list calls keep working
The e2e test now uses gemini-3.8-flash with enough max_tokens to finish thinking before the tool call.
* fix(gemini): keep non-chat routes tolerant of a malformed thought_signature_in_tool_call_id, match tool results by exact id first, and keep the deployment's value when drop_params drops a malformed override
* fix(router): validate strict litellm params through a helper so star imports cannot re-bind a public Final
* perf(router): skip strict validation when a request overrides no deployment param
* fix(responses): keep a dict tool call's thought signature on the streamed function_call done item
* refactor(main): return invalid supplemental params as a value and test the streaming tool call with a real response
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(realtime): send Gemini Live setup on the client's first message so session.update modalities apply
Gemini and Vertex AI Live accept exactly one setup per session, and the proxy sent its
default audio setup at connect, so a client's session.update (GA output_modalities or
beta modalities) was dropped and every answer came back as audio. The setup now goes
out on the client's first message: a session.update becomes that setup, anything else
gets the model default. Bedrock Nova Sonic keeps its connect-time setup through a
provider hook. Text setups label deltas response.output_text.delta, and downgrading
TEXT to AUDIO on an audio-only model logs a warning
* fix(realtime): keep one Gemini Live response modality and skip the "inf" token limit in the setup
* fix(realtime): merge every session.update sent before the first content frame into the Gemini Live setup
Gemini and Vertex Live accept one setup per connection, so the bridge used to
turn only the first session.update into that setup and drop the rest without
telling the client. Now every session.update that arrives before the first
content frame is held, deep-merged (later values win per field), acknowledged
with a session.updated built from the setup it would produce, and the merged
session becomes the one setup sent with the first conversation item, audio
frame, or tool result. The session events also surface the systemInstruction
text as instructions, which the old string check never matched because the
setup carries it as a Content object.
* fix(realtime): cap the session.update merge depth so the recursive detector accepts it
_merge_session_fields recurses once per nested mapping level, so CI's
recursive_detector flagged it. Bound the descent at eight levels (the OpenAI
session object nests four deep at most), replace anything deeper wholesale
with the later session.update's value, and list the helper with the other
depth-capped functions the detector allows.
* fix(gemini-live): accept a string output modality and drop values Live cannot produce
A bare string output_modalities is read as a one-element list and modality values Gemini Live cannot produce are dropped with a warning instead of reaching the backend, so a malformed session.update no longer costs the session. Transcription guardrails now fold their VAD disable into the first held session.update only, so the merged setup carries it exactly once.
* fix(gemini realtime): answer an error for session updates Gemini Live cannot apply
An output modality the model cannot produce, or a session.update sent after the
first content, now gets an invalid_request_error event instead of a silent swap
or drop, unless drop_params is set. The synthetic session.created reports the
model's default modality, setupComplete no longer adds an extra session.updated,
and merging nested session updates is linear in their size
* fix(gemini realtime): send the setup before guardrail blocks and idle deadlines, mark push-to-talk turns
A pre_call guardrail blocking the first typed message now goes out after the
setup, so Gemini and Vertex no longer close the session with 1007.
Push-to-talk sessions (turn_detection null or create_response false) wrap each
audio turn in empty activityStart and activityEnd messages, which Gemini needs
when automatic activity detection is off.
A session with no content after 60 seconds sends its setup, since Vertex
refuses a setup 300 seconds after connect. Held session.update frames past
10 MB answer session_update_too_large, and deferred mode now errors on a
second session.update unless drop_params is set.
* fix(realtime): ack unchanged Gemini Live session.update resends and honor null turn_detection
An update that leaves the applied setup unchanged now gets session.updated
instead of session_update_not_applied, so clients that resend their session
with the first response keep working. Deferred setup mode merges held updates
like the default mode, turn_detection null turns Gemini's activity detection
off, drop_params drops a voice outside the documented Live voices, and the
lazy setup deadline moves to 240 seconds
* fix(realtime): keep Vertex activity detection off for null turn_detection and set up deferred sessions before text content
* fix(realtime): refuse unscreenable Gemini audio, send held setup after buffered audio, reject unknown Vertex voices
A realtime_input_transcription guardrail cannot screen Gemini Live audio before the answer starts, so audio input under one is refused with an audio_input_not_screenable error instead of reaching the model. In deferred setup mode a session.update that follows buffered audio is sent as setup at once. Only conversation content triggers the setup, so control events no longer lock in a default one. Vertex rejects an undocumented voice on the session.update, or drops it with drop_params, while Google AI Studio forwards any name.
* fix(realtime): hold first content so a following session.update joins the Gemini setup
Convert flat realtime tool schemas through Gemini's schema builder, and collapse a text and audio pair to what the model can produce on text-only models too
* fix(realtime): check AI Studio Live voices like Vertex
* fix(realtime): forward any voice to AI Studio Live models that accept it
* fix(realtime): map OpenAI TTS voices before the Live voice check, keep outage spend rows, share the playground session config
The Live voice gate now judges the Gemini voice an OpenAI TTS name maps to, so fable and onyx reach Vertex as Umbriel and Orus again. An upstream close before the held setup was sent is logged as a session, not a refusal. The playground sends one server_vad session config for typed and voice turns, and the e2e, integration, and dashboard tests follow lazy setup.
* fix(realtime): take the lazy setup clock as a dependency so its tests run on a stepped clock
The deadline and grace tests waited on real 50 ms timers, and the merge cost
check compared wall-clock runs. The session now reads an injected clock,
time.monotonic by default, the two tests step it, and the merge cost check
moves to the CodSpeed benchmarks. The Nova Sonic style handler test passes
the backend through wrap_backend, which main added after the test was written
Neither chart bundles Lens anymore. The Lens subchart, its lock entry,
the vendored chart archive, the templates/lens resources and the
bundled-Lens install smoke are gone. A connection-only lens block wires
the gateway and backend to a Lens deployed from its own chart, and an
enabled lensWorker value fails the render instead of being ignored.
Supersedes #47869, which made the same change for helm/litellm only.
* fix(batches): read OpenAI batch usage into the batch token counts
* fix(batches): serve OpenAI's batch usage shape from retrieve and the cost poller
* fix(batches): keep the chat usage keys next to OpenAI's batch usage fields
Batch usage now carries every chat key it had plus OpenAI's five batch fields with the same counts, so poller-billed batches keep prompt_tokens and cache-write tokens, and older proxies still read the new rows. Usage with missing or null detail counts reads its real input and output counts.
* test(batches): check the batch usage counts in the e2e terminal-state test
* fix(types): apply nested include and exclude to both batch usage shapes
* fix(batches): log no cost for a batch retrieve that is not final yet
A retrieve now reads OpenAI's batch usage into the chat counts, so a retrieve
that saw the batch completed before its output file arrived priced those counts
for the logging callbacks, and the final retrieve priced the whole batch again.
A batch that is not final now logs no cost, as it did before.
Batch usage dumps also pass exclude_unset and exclude_defaults through to the
usage keys.
* fix(batches): pass through every usage detail field the provider sends
Batch dumps rebuilt input_tokens_details and output_tokens_details from the chat keys, which dropped any detail field past cached_tokens and reasoning_tokens that the merge base passed through. The dump now starts from the provider's stored details and only sets the two counts derived from the chat keys
* fix(batches): skip sync loggers on the cost poller's own retrieve and honor exclude_none on provider details
* fix(batches): keep sync logger events at $0 on poller-owned retrieves, pass unparsed provider counts through, log batch usage in chat shape
* fix(batches): store only chat usage extras on batch spend rows
Spend rows for batch retrieves copied OpenAI's batch fields (input_tokens,
output_tokens and their details) into metadata.additional_usage_values next
to the chat counts they duplicate. Build the row from the batch's chat usage,
shared with the logging payload through batch_chat_usage.
* feat(proxy): reject chat routes on models declared non-chat via model_info.supported_endpoints
A model whose every deployment sets a non-chat model_info.mode and a supported_endpoints list with no chat endpoint now answers /v1/chat/completions, /v1/messages and /v1/responses with a 400 before any provider call. Models without both fields keep today's behavior.
* fix(proxy): let fallbacks answer and cover more chat routes in the non-chat model check
The check now steps aside when a request, key, team or router fallback would answer, skips such models in the rate-limit fallback loop, ignores mode and supported_endpoints copied from the cost map by old Admin UI saves, treats a blank mode as unset, and also covers /v1/completions, the Responses WebSocket, Gemini generateContent and /v1beta/interactions
* fix(proxy): only step aside for fallbacks that can answer chat, gate the Responses WebSocket after alias resolution, and compare declarations to the loaded catalog
* fix(proxy): gate the Responses WebSocket with the router after aliases resolve, and ignore client-set internal fallback flags
* fix(proxy): walk fallback groups breadth first and release reservations on a rejected WebSocket
Each fallback group is looked up once however often the request repeats it, and a WebSocket closed by the endpoint gate runs the failure hook so rate-limit reservations are released
* fix(proxy): walk fallback layers without recursion and gate the compact Responses route
The fallback walk now folds one layer at a time, so the recursion check passes, and /v1/responses/compact on a model declared non-chat gets the same 400 as the other chat-shaped routes
* fix(proxy): keep chat on rows whose cost map entry serves chat
A row that carries the cost map key of an entry that serves chat counts as declaring nothing, so a catalog edit that adds a chat path never turns old Admin UI copies into 400s
* fix(proxy): check chat sent to a deployment id or backend model name
* fix(proxy): check a model the read-through loads from the database
* fix(proxy): step aside for fallbacks the check cannot follow and wildcards that serve a backend name
A fallback target with no deployments of its own, a client routing stamp, and a wildcard that does not match the requested backend name no longer turn a working request into a 400
* fix(proxy): ignore a client model_group that is not a string when building rate-limit headers
A client metadata.model_group sent as an int or a list crashed the error handler with a 500 once the embedding-only check rejected the call before routing. The header helper now treats a non-string stamp as no model group. Drops the pattern clause in the backend-name lookup, since the group lookup already returns wildcard matches. Adds the third audit round's routing-stamp cells.
* fix(proxy): keep a non-string client model_group out of spend rows
* fix(proxy): validate standard fallbacks before the typed fallback-group lookup
* feat(router): opt in to non-chat deployment filtering
* fix(router): skip unresolved local rate-limit fallbacks
* fix(router): ignore caller-provided endpoint selector values
* fix(router): normalize declared chat endpoint prefixes
* test(router): cover preflight error-value behavior
* test(router): derive replay metadata defaults from model info
* fix(router): preserve fallback aliases and check synchronous text calls
* test(routing): rewrite the WebSocket model with a pre-call callback, since main no longer runs guardrails at socket admission
* test(routing): keep the endpoint gate's owned proxies on the worker's database
* fix(router): keep the endpoint gate on model_info patches that omit it and on same-group stream retries
A partial PATCH of model_info reset a stored enforce_supported_endpoints true back to false, because the
field defaulted to False and the update merges model_dump(exclude_none=True). The field now defaults to None
so an omitted flag keeps the stored value
The same-group retry of a stream dropped before content called the provider without the endpoint selection
context, so it could land on an enforced embedding-only deployment of the same group. It now runs inside
endpoint_selection like every other router attempt
* fix(router): probe rate-limit fallbacks through get_routable_deployments so the router surface does not grow
Main's router fence lets code outside the router use only listed Router members and
router_utils modules. The proxy's fallback probe now calls the listed
get_routable_deployments with the request's call type, and the gate's 400 moves to
litellm.exceptions as EndpointNotSupportedByModelError. It stays separate from main's
UnsupportedEndpointError, which the router retries on the group's other deployments,
while the gate refuses the whole group
* test(responses): cover the WebSocket error frame for a model that refuses the endpoint
* fix(proxy): let the rate-limit fallback probe try a fallback this worker has not loaded yet
The probe now skips a fallback only when the endpoint check removed every deployment it has, so a model added on another worker still reaches route_request and its database lookup
* fix(proxy): keep key router settings off the Responses WebSocket, read unrooted or single-string endpoint lists, and add status to the refusal frame
* fix(router): warn when enforce_supported_endpoints has no usable list and match chat endpoints regardless of case, trailing slash, or Gemini path
* test(router): type the load-warning test's caplog and router
* test(routing): match the endpoint gate's integration tests to base fallback behavior on unloaded fallbacks and the Responses WebSocket
---------
Co-authored-by: Mateo <mateo@Mateos-MBP-2.localdomain>
* test(integration): update stale expectations to intended provider behavior
Each expectation moved because a merged PR changed behavior on purpose.
- chatgpt string input is wrapped as one user message (#41522)
- Gemini 2.5 tool media sits beside function_response (#48428)
- sampling knobs move to Gemini 3.5 Flash, which keeps them (#47645)
- a body base_model is stripped without allow_client_pricing_override (#47639)
- ai_decide answers without usage return usage null (#48261)
- azure_ai forwards mapped client anthropic-beta values (#48033)
- the OpenCode scripted upstream mints a fresh id per request
* test(e2e-ui): type the guardrail name before deleting it
GuardrailDeleteModal requires the exact name before Delete enables since
#45893. All three guardrail delete flows now go through one helper.
* test(observability): accept any sampled traceresponse flags and expect a 400 for a non-object body
OpenTelemetry 1.45 (#48055) sets the W3C random trace id flag on new traces,
so the gateway's traceresponse now ends in -03. The producer copies the server
span's own trace flags, so the tests now accept any two-digit flags with the
sampled bit set instead of pinning -01
#48438 made a non-object JSON body a 400 "Invalid JSON payload: expected an
object" on purpose, so the [1] body case is renamed non_object_body and
expects that 400
* test(observability): stop waiting on OpenTelemetry's export log line in the langtrace outage test
The outage test waited for "Exception while exporting Span batch" in the proxy
log, which is OpenTelemetry's wording and no longer appears with 1.45. The test
keeps asserting what the gateway guarantees, callers unaffected during the
outage, a live gateway, and every span after recovery delivered once with no
outage span duplicated
* test(guardrails): count the Responses disconnect spend rows #47611 added
Since #47611 a streamed /v1/responses disconnect writes its own spend row, so
the concurrent disconnect test saw 20 rows where it counted 10 by model group.
It now reads each row's call type and expects ten acompletion rows with the
outage verdict and ten aresponses rows with success. /v1/messages disconnects
still write no row on main, so they are left out of the count rather than
pinned as absent
The responses-httpx case of the single disconnect test no longer needs its
LIT-8603 skip and passes
* test(observability): size K4's timeout from the harness boot budget
K4 boots three two-worker proxies in a row. On CircleCI each boot takes 43 to
58 seconds, so a passing run took 179.97s against its fixed 180s cap and 6 of
30 runs hit pytest-timeout. The cap and the restart gate wait now come from
readiness_seconds and graceful_stop_seconds, like other multi-boot tests
* test: remove e2e checks and a CI line count that pin implementation details
Deletes the Auto Router template select anchoring spec, the 390px logs pagination variant, and the call id tooltip and clipboard step in the logs spec. Each one tests library or test helper behavior rather than what users see, and each has been red or flaky on main without a product regression
Also drops the exact line count on the fan-in requirements export in the integration coverage job. It went from 6 to 10 when uv.lock gained a Python 3.15 fork, while the install was still correct. A package the filter drops still fails the job, either at the hash-checked install or at the next step's import
* docs(agents): do not test what shadcn and Base UI already guarantee
Adds the boundary to the dashboard AGENTS.md next to the test tiers, and points the Playwright suite's entry in tests/e2e/AGENTS.md at it
* refactor(mcp): drop the experimental label from the MCP server and client paths
Move litellm/proxy/_experimental/mcp_server to litellm/proxy/mcp_server and litellm/experimental_mcp_client to litellm/mcp_client, with the mirrored tests moved under tests/unit/proxy/mcp_server and tests/unit/mcp_client. Both old packages stay as shims that warn with DeprecationWarning, re-export the new package's public names and alias every submodule in sys.modules so deep imports keep resolving to the same objects. The OpenAPI docs gate now puts the merge base's enterprise package on PYTHONPATH so its base counts do not mix the checked-out enterprise code with the base litellm tree
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(mcp): keep tests/unit/proxy/_experimental as a sharded root for the shim tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): alias namespace subpackages and bind attributes in the deprecated MCP shims
pkgutil.walk_packages skipped mcp_server/auth because it has no __init__.py, so old imports of litellm.proxy._experimental.mcp_server.auth.* failed, and sys.modules entries alone left litellm.experimental_mcp_client.tools unreachable as an attribute. Walk the package on disk and setattr each alias on its old parent.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages
Move the unified /v1/messages code out of a directory called pass_through, since it is not the
passthrough feature. The inner messages sub-package becomes native, and the Vertex AI partner
model experimental_pass_through package becomes messages. The old packages stay as import shims
that warn once and alias every submodule in sys.modules, so old deep imports still resolve
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): keep Final variables out of loops in the rename shim
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): match the warning text in the rename shim test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): resolve old pass_through imports lazily through a meta path finder
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): mark the alias loader spec restore for the type discipline gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(anthropic): format the renamed package shim
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(anthropic): keep the rebind-ok marker on the spec restore line
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): carry the new package search path on alias specs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): declare the shim test locals Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
gpt_5_transformation.py becomes gpt_reasoning_transformation.py (OpenAI and Azure),
the Bedrock anthropic_claude3_transformation.py files become anthropic_claude_transformation.py,
anthropic_claude2_transformation.py becomes anthropic_legacy_text_completions_transformation.py,
and vertex_ai_partner_models/llama3 becomes vertex_ai_partner_models/llama. Class and symbol
names are unchanged.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep Aria accessible in the dashboard header
* test(ui): wait for every pagination fixture log
* test(ui): update assistant setup for Aria branding
* test(perf): add small_bodies_30rps and mixed_size_requests_4rps load cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(perf): run the load cases at 50 rps small and 5 rps mixed, the pod's knee
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(perf): ramp the load cases until the pod stops keeping up and report sustained_rps
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(perf): hand the ramp to the observation and widen the mixed ramp to 14 rps
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(perf): floor the small ramp at 20 rps and emit the per-step readings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(perf): emit the ramp's hold bound and each step's unexpected statuses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(perf): read each ramp step's cpu, redis latency and connections from the samplers' windows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(perf): floor mixed_size_requests_ramp at 8 rps from the lane's 10 rps sustained step
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(perf): compare step readings with math.isclose so the e2e basedpyright gate stays clean
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): show the budget spend limit and rate limits without a collapsed section
The create and edit budget dialogs hid max budget and reset period behind a collapsed Optional Settings section that sat 80px below the last field, and dropped both values from the request unless the section was open on submit
Both dialogs now share BudgetFormFields. Budget ID comes first, then the spend limit as one row ("$500 per month"), then the three rate limits as short fields beside their labels. A secondary Optional badge marks the spend limit and the rate limits group, so people can still tell those fields are optional. Every field is always submitted, and the edit dialog sends the existing spend limit and reset period back unchanged
* fix(ui): send only changed fields when editing a budget
Saving the edit budget dialog resent every field, so an unrelated rate
limit edit rounded a stored sub-cent spend limit to 2 decimals (0.001
became 0) and resent the reset period, which makes the proxy recompute
the next reset for custom durations. The dialog now sends the budget ID
plus the fields the user changed, and the proxy keeps everything else.
Also updates the budgets Playwright spec for the new field labels and
the removed Optional Settings section.
* fix(ui): track budget edit changes from render so a reverted field cannot hide a later one
react-hook-form's formState is a render-time snapshot. Reading dirtyFields
only inside the submit handler meant that after a field was changed and
changed back, a later edit to another field was missing from the first
save. Reading dirtyFields during render keeps the snapshot current.
* refactor(ui): rename the budget components to PascalCase
Renames budget_modal, edit_budget_modal and budget_panel (and their tests) to BudgetModal, EditBudgetModal and BudgetPanel, and drops their filename-pascal-case suppressions.
* test(perf): set every perf ceiling to 1.2x the lane median
* test(perf): derive the upgrade verdict test's samples from the case ceilings
* test(perf): keep memory_growth_megabytes at 100 and 600, the metric swings 60 MB between reruns
* test(perf): keep small litellm_overhead_seconds p99 at 1.2x the lane max, it is the 3rd slowest of 300 requests
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(completion): send prompt_cache_retention to OpenAI instead of silently dropping it
litellm.completion(), Router.completion() and the proxy's /v1/chat/completions
accepted prompt_cache_retention and listed it as a supported OpenAI param, but
completion() never handed it to get_optional_params, and the leftover-kwargs
paths skip known OpenAI params, so it never reached the request body. Make it a
named argument of completion() and acompletion() like prompt_cache_key and pass
it on.
A unit guard sends every OpenAI body param through completion() and fails when a
param added to OPENAI_CHAT_COMPLETION_PARAMS has no case proving it reaches the
wire, the same hole store and prompt_cache_key fell into in #33184. Live e2e
tests capture the body OpenAI receives for the proxy, the SDK and the Router on
both openai/gpt-5.5 and openai/responses/gpt-5.5.
* chore(completion): type the param-forwarding unit test and keep the basedpyright count flat
Every new completion() kwarg adds one unknown-argument error at each call
site that splats an untyped dict into it; typing print_verbose's argument
and Chat's params removes more than prompt_cache_retention added.
* test: freeze the outbound-body model and read captured requests from the mock
Addresses review: the e2e body model is frozen, the backend params are a
tuple, and the unit test reads the request from the respx route instead of
appending to a list.
* test(perf): measure Admin UI journey latency in a browser on the 1TB restore
* test(perf): settle on rendered rows, chart bars and the usage view trigger
* test(perf): cover the UI journey summaries in the e2e harness tests
* test(perf): fail UI journeys on failed samples and p50/p90 ceilings
* test(perf): add Usage, Models and Keys journeys and seed realistic keys, teams, users and models on the disposable restore
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): apply the 16 maxTokens floor to Kimi K3 on Converse
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): use lane AWS credentials for the Kimi K3 messages test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): bypass the proxy cache and declare test metadata for the Kimi K3 probe
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(perf): gate the gateway's own overhead header, redis latency and postgres load in the perf cases
* test(perf): keep the gateway_added_seconds limits next to the header ones
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
