The gpt-5.4+ responses-bridge gate classified the endpoint from the call-level
api_base alone, while the OpenAI chat handler resolves arg > global > env >
default. A custom base configured via litellm.api_base or OPENAI_BASE_URL/
OPENAI_API_BASE was therefore invisible to the gate: it read blank as the
default OpenAI endpoint and bridged a request the custom backend has no
/responses route for.
Extract that resolution into one _resolve_openai_api_base() and have both the
gate and _complete_custom_openai() call it, so the gate can never classify an
endpoint the request won't hit. The gate compares the resolved base against the
default (import litellm seeds OPENAI_BASE_URL to the default, so "override is
non-None" is not a safe custom-endpoint signal); whitespace collapses to the
default as before. reasoning_effort="none" remains the escape hatch.
Chat Completions nests a named tool_choice under its tool type while the
Responses API keeps it flat; ChatCompletionNamedToolChoiceParam and
ChatCompletionNamedToolChoiceCustomParam both mark the nested key required. The
messages arm normalized tool definitions but forwarded tool_choice at whatever
level Cursor sent it, so a flat {"type": "custom", "name": "ApplyPatch"} reached
OpenAI unchanged and was rejected while the tool defs beside it nested correctly
A tool definition and a named tool_choice carry the same envelope, so both now
convert through a single _convert_tool_envelope, and _normalize_tool_dialect
moves tools and tool_choice together on each arm. That covers all four cells of
{tool def, tool_choice} x {to chat, to responses} and removes the shape where
one field can be converted while the other is missed, replacing three helpers
with two and cutting 24 lines
Also restores the end-to-end assertion that a flat tool_choice reaches
chat_completion nested, which had been flipped to pin the passthrough behavior
- share one _CustomToolCallAccess mixin across the 4 new custom-tool
classes instead of hand-rolling dict access on each
- inline the single-use _nest_flat_chat_tools / _flatten_chat_tools_for_responses
list wrappers at their call sites
- drop _nest_flat_chat_tool_choice: it rewrote object-form chat tool_choice
into {type,custom:{name}}, a shape OpenAI rejects; real Cursor never sends
tool_choice on the messages arm, so pass it through unchanged
A blank api_base (empty or whitespace) resolves to the default OpenAI
base downstream but is not None, so the constraint-enforcing-endpoint
check misclassified it as a custom backend and skipped the unset-effort
auto-bridge, leaving gpt-5.4+ function-tool requests to 400 at OpenAI.
The check now treats None, empty, and whitespace api_base alike; a real
custom base still opts out. Verified with get_llm_provider, which passes
a blank api_base through while resolving the provider to openai
Chat-only OpenAI-compatible backends registered under the openai
provider with custom api_base and gpt-5.4+ model names served
tools-without-reasoning fine and have no /responses route, so the
unset-effort arm added for real OpenAI would have silently rerouted
previously working deployments. The arm now fires only when api_base is
unset (default OpenAI endpoint) or the provider is azure; an explicit
reasoning_effort keeps its pre-existing bridging behavior on any
api_base. Flagged lines also modernized to PEP 604
An import probe proves nothing about the real SDK: it may be absent (it
lives in the proxy-runtime extra) and the tests/test_litellm/llms/anthropic
test package can shadow it once collection puts that path on sys.path,
which made the test order-sensitive across collection sets
Three gaps from the bridge becoming a mainstream path for chat traffic.
The chat to responses message converter only mapped function tool_calls,
so history carrying the native custom tool calls this PR introduced
raised "tool call not supported" on follow-up turns; custom entries now
map to custom_tool_call items and their results to
custom_tool_call_output. The stream translator returned an empty delta
for output_item.done on tool items, which left the responses guardrail
handler's tool extraction permanently empty (dead on staging too, where
the built chunk was discarded); stateless callers now receive the
complete tool call while per-stream callers keep the suppressed delta
that prevents client-side duplication. Cursor routing keyed on the
presence of a messages key, so a null or empty stub next to a real
agent-mode input array picked the chat arm; routing now keys on
messages content
The bridge gate compared reasoning_effort against the string "none", so
litellm's dict form ({"effort": "none"}) wrongly bridged; the gate now
reads the effort value from either form and treats a summary inside the
dict as Responses-only regardless of effort. Helicone and lunary
previously skipped custom tool calls entirely; both now serialize them
(helicone as a tool_use block from the custom payload, lunary with the
custom name and input in its function fields, keeping type custom), with
new mapped tests for both integrations
OpenAI's chat completions rejection applies to function tools only;
custom (grammar) tools are served natively with reasoning on, live-proven
by a 200 on a custom-only gpt-5.6 chat request. Gating on any truthy
tools needlessly bridged custom-only requests, and the bridge maps custom
tool calls back function-shaped, so the native chat custom tool_call
surface added earlier in this PR was bypassed exactly where chat serves
it natively. The gate now checks for a function-type tool in either the
nested chat or flat Responses def shape; the same coarseness existed on
the explicit-effort arm before this PR and is fixed by the shared leg
OpenAI enables reasoning by default for gpt-5.4+ (unset reasoning_effort
means medium server-side) and Chat Completions rejects function tools
whenever reasoning is on, so a tools request without an explicit
reasoning_effort 400d instead of auto-bridging to the Responses API; the
bridge heuristic now treats unset effort as reasoning-active and honors
the documented escape hatch by keeping explicit "none" on chat
completions. The cursor input arm also gains the mirror of the
messages-arm normalization: chat-nested tool envelopes, grammar formats,
and object tool_choice flatten to the Responses dialect before dispatch
Live Cursor Ask-mode captures show the shape dialects mix PER LEVEL: the
tool envelope arrives chat-nested while the grammar format inside it is
still Responses-flat, so a normalizer that pattern-matches whole-tool
templates misses every hybrid. The cursor arm now normalizes the envelope
level and the format level independently and idempotently, making it
total over the envelope x format matrix; a parametrized 8-cell test pins
every combination. The reference BYOK bridge was checked and forwards
chat bodies verbatim, so there is no prior art for these hybrids
Cursor's ApplyPatch is a grammar-constrained custom tool; the Responses
surface carries the grammar flat while chat completions wraps the same
fields in a grammar object, so the nested envelope from the previous
commit still 400d at OpenAI (tools[N].custom.format.grammar). Adds a
shared flat to nested format helper pair in prompt_templates/common_utils
used by the cursor messages arm and the chat-to-responses bridge, nests
flat Responses-style tool_choice objects on the cursor arm, flattens chat
custom tool_choice on the chat-to-responses bridge, and maps custom
tool_choice to function tool_choice on the responses-to-chat bridge to
match that bridge's custom-to-function tool downgrade
Cursor Ask mode sends chat bodies whose tools array mixes nested function
tools with flat Responses-style custom tools; the /cursor messages arm now
nests those before delegating, published via the request parsed-body cache.
Core chat parsing gains first-class custom tool call types mirroring the
openai SDK union: a single dict dispatch feeds the provider-dict sinks,
Delta dispatch stops both stream re-parse sites from silently swallowing
custom deltas, the chunk builder accumulates custom input for spend logs,
function-assuming consumers (json-mode gate, multi_tool_use repair,
helicone, lunary) skip custom entries, and the chat-to-responses bridge
flattens nested custom tools to the Responses flat shape
- delegate messages-shaped bodies to the standard chat completions handler
- strip chat-only stream_options before the Responses pipeline
- fix cursor_data_generator signature (request kwarg) and duck-type the
stream gate so router-wrapped streams convert instead of leaking raw
Responses events
- convert custom_tool_call items and events to chat tool_calls in the
streaming and non-streaming paths; remap streamed tool_call indices to
0-based sequential; accumulate raw and pydantic tool calls into one choice
- normalize generic pydantic output items through the raw-dict handler
Replace Any-typed seams with real types in the files carrying the highest
reportAny/reportExplicitAny density. The dominant source was the repository
layer: BaseRepository.table is declared Any, so every repository read poisoned
its rows and every downstream call. Typed pass-through accessors under a
_PrismaTableActions Protocol pay that crossing once per table, and TypedDicts
and Protocols replace the remaining Any-typed request, row, and tool payloads
across the team, key, SCIM, spend, MCP, guardrail, video, and websearch
surfaces
No casts, no type: ignore, no noqa, no new Any annotations, no behavior
changes. Whole-tree basedpyright: reportAny 20,840 -> 19,397,
reportExplicitAny 7,253 -> 6,518, all rules 151,424 -> 149,066, with no rule
increased in any file. Budgets ratcheted: basedpyright -2,358, ruff-strict
-300, type-discipline -68
* fix(ui): show pass through route selections in team/key forms and match team id substrings in team search
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(teams): keep team id search index-friendly with a prefix match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(teams): keep /v2/team/list search id matching exact by default and add an opt-in prefix mode
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): stop model writes 500ing on another pod's delete
A model write judges the reload it triggers by diffing this pod's router before and
after, and reports anything that stopped serving as damage. On a pod that has not yet
polled a delete another pod made, the snapshot still lists that model; the reload then
evicts it because the db no longer has it, and the guard reads its own correct
reconcile as degradation. The row is written and served, but the caller gets a 500.
Since propagation between pods is a 30s db poll, any delete followed by a create
inside that window can land on a pod that has not caught up, so a delete-then-create
pair returns 500 whenever the two requests hit different pods.
_delete_deployment already computes exactly the set that settles it: the ids the db
and config still want. Thread it up through _update_llm_router, add_deployment and
clear_cache to the verdict, and intersect the drop set with it so an id the db no
longer has stops counting as collateral. Where no reconcile ran the set is None and
every drop is still reported, so a genuinely broken reload is caught as before.
_delete_deployment now returns that set instead of a delete count; the count had no
callers in the proxy, and the tests asserting it already assert the eviction calls.
* test(proxy): fold reload-verdict test commentary into docstrings and assertions
Greptile flagged the inline comments against the repo's no-new-comments rule. The
case-by-case context moves into the test docstring, and the two return-contract
assertions carry their reasoning as failure messages instead.
* test: fix clear_cache mock return type in model block/unblock tests
The comment said the Responses WebSocket route never runs
add_litellm_data_to_request. It does, via common_processing_pre_call_logic,
so the note recorded a request-flow constraint that does not hold
The gate stays in route_request, which is the dispatch chokepoint and where
the previous handling lived
Handling of the client-supplied mock testing params was split across three
places with different behavior for each. Three were dropped from every proxy
request, two reached the router untouched, and a request that asked for a
synthetic failure came back as an ordinary success with nothing to indicate
that no failure had been injected
Put all six behind one opt-in, general_settings.
dangerously_allow_mock_testing_request_params, and reject rather than drop
when it is unset, so a fallback drill cannot report a pass for a test that
never ran. The rejection names the params it saw and the config key to set,
which is also the answer for anyone following the older docs
The flag is config-file only. It is deliberately absent from
ConfigGeneralSettings, and that absence is what makes /config/update drop it
on parse and /config/field/update reject it; the tests pin both so the field
cannot be added back for tidiness without the reason surfacing. Enabling it
logs a startup warning naming every param it unlocks
BREAKING CHANGE: mock_timeout and mock_testing_rate_limit_error now require
general_settings.dangerously_allow_mock_testing_request_params to be set in
config.yaml. Previously they were accepted unconditionally
Tool-level MCP entitlements are enforced in one place,
check_tool_permission_for_key_team, reached from pre_call_tool_check. Two
dispatch paths reached a tool handler without passing through it.
execute_mcp_tool's legacy fallback dispatched into the local tool registry
after retrying the unprefixed name, with no allowed/banned-tool check, no
key/team/org tool permissions and no parameter validation. It now runs the same
gate, and only when something can actually dispatch: when the unprefixed name is
absent from the local registry too, the existing 404 stands rather than becoming
a misleading "server unavailable".
The server the tool-level checks need is available even though the tool name is
not in the tool -> server mapping: a non-empty prefix has already been compared
against the caller's allowed_mcp_servers by exact name, so the named server is
in that list. It is resolved from allowed_mcp_servers rather than from the
manager's registry, because the registry can return a server the caller holds no
grant for, and matching on anything other than name would accept a server the
server-level check never validated. The remaining case is a prefix segment that
is empty, which the server-level check skips entirely because it is gated on a
non-empty server name; that now fails closed with 503 instead of dispatching for
a caller holding no server grant at all.
An entitled caller's legacy call therefore still dispatches, so a configuration
that worked before keeps working; only the unentitled call is refused, now with
the entitlement gate's own 403.
call_tool ran pre_call_tool_check inside `if proxy_logging_obj:`, so an absent
logging object would have skipped authorization silently. This half is defensive
with no live hole: all four call sites source the module-level ProxyLogging
singleton from proxy_server.py, which is never None. The shape was still wrong.
pre_call_tool_check now runs its three authorization checks unconditionally and
only the guardrail hooks, which are dispatched through the logger, depend on one
being present.
A third reported path, where allow_all_keys, BYOM-submitted and
upstream-delegated servers are unioned in after the resolver's ceilings, was
investigated and found not to be a defect. The widening is real, but a server's
tool surface is already boundable for every caller at registration through
MCPServer.allowed_tools / disallowed_tools, enforced by
check_allowed_or_banned_tools ahead of the entitlement check, and per-caller
narrowing plus the org tool ceiling remain available. Nothing here changes that
path.
Resolves LIT-4956
Opening a session from the logs table stored no ?session_id (row clicks
called openLog, which deletes it), so session mode was derived from the
clicked row's session_total_count. Rows fetched by the session drawer
come from /spend/logs/session/ui, which does not enrich that field, so
selecting any log inside the session view swapped in an unenriched row
and collapsed the drawer to a single-log Trace view
Row clicks on a multi-call session's row now call openSession, and
selectLog writes ?session_id when the session view is active, so session
mode is anchored in the URL instead of derived from row data
The create button now sits in the tab bar beside the tabs, the way Teams
lays it out, with one divider between them and the rule running the full
width underneath.
Adds a fillHeight mode to DataTable that treats the parent's height as a
ceiling rather than a target, so the table still sizes to its rows and a
short one keeps its footer under the last row, while a long one scrolls
its rows under a sticky header instead of scrolling the page. This replaces
the hardcoded viewport-height caps those tables would otherwise need. Two
details the mode has to fix: the Table primitive's own overflow container
would capture the sticky header, and rows would show through the
semi-transparent header tint.
The hook resolves the newly created user through UserRepository and builds the
audit entry from that row. Pin both halves: the entry carries the persisted
row's fields rather than the /user/new response, and a user id that resolves to
nothing produces no entry at all.
Matches the Virtual Keys layout: a page header with the wallet icon, the
create button directly beneath it, and the tab bar below that, on the same
page padding Teams and Access Groups use so the table no longer sits against
the window edge.
Reset and Created start hidden, so the table opens on the four columns it
has always shown and the two new ones are opt-in from the Columns menu.
Swap `Model(**payload)` for `Model.model_validate(payload)` at the seams where
the payload comes back untyped, so basedpyright stops widening every target
field to Any. None of the models involved override `__init__`, so validation
goes through the same core validator either way.
Also route UserRepository through its own typed helpers (find_many, update,
find_by_id) instead of the raw Prisma table, drop the redundant `_to_model`
override signature, and call generate_key_helper_fn with explicit arguments in
the SSO callback rather than splatting an untyped dict.
Whole-tree basedpyright: reportAny 21481 -> 20834, reportExplicitAny 7258 ->
7252, with every other rule unchanged or lower.
* feat(spend-logs): record when a spend log row is the auto-router's own classifier call
The complexity router's classifier sub-call copies the parent request's metadata
verbatim, so its spend log row carries the caller's key, team and user and is
indistinguishable from traffic the caller actually sent. Nothing on the row says
otherwise: call_type is "acompletion" either way, model_group is overwritten to the
classifier's own model group so the row never looks auto-routed, and routing_decision
is absent exactly as it is on an ordinary request.
Record the fact the system already knows at call time. internal_call_origin is
declared on SpendLogsMetadata, which is the allowlist _get_spend_logs_metadata
projects onto, and stamped in _classifier_call_metadata; both classifier paths
already route through that one function and it feeds the metadata and
litellm_metadata buckets alike, so every request surface is covered at one site.
The key is reserved rather than caller-supplied, so it joins routing_decision in the
untrusted-metadata strip and a caller cannot label their own traffic as router
overhead.
The classifier call also inherited no session identity, so the router minted a fresh
trace id and the row landed in a session of its own. Forwarding the parent's session
puts it in the trace of the request that triggered it, which is where an operator
looks for what the routing cost.
* feat(ui): show which log rows are the auto-router's own classifier calls
A classifier row now carries internal_call_origin and shares its parent's session,
so the session trace lists it beside the request that triggered it. Without a marker
in the sidebar it reads as another call the caller made, which is the confusion this
resolves.
The tag renders only for a recognized origin, so ordinary traffic and any future
origin this build does not know about stay unlabelled rather than being asserted as
classifier calls.
* feat(proxy): add a generic list contract for management/v1 entity lists
Paging, sorting, filtering and search for an entity collection, declared once
as a ListSpec and served by handle_list. The route injects a ListExecutor that
owns its table, so this module never imports Prisma.
The caller's scope is derived from the caller alone and ANDed with whatever
they filtered on, so a query parameter can only narrow what they may read.
This is the shared half of the budgets list; it lands here so the endpoint has
something to register against, and drops out when the framework arrives on its
own branch.
* feat(proxy): add GET /management/v1/budgets
The Budgets page reads /budget/list, which returns the whole table as a bare
array with no way to page, sort or filter it. A customer with enough budgets to
fill the page has no way to find one.
Registers LiteLLM_BudgetTable against the management/v1 list contract: sortable
on budget_id, max_budget, tpm_limit, rpm_limit and created_at, default order
newest-first with budget_id breaking ties, search on budget_id, and filters for
budget_duration, max_budget and created_at. budget_duration is deliberately not
sortable; the column holds "7d"/"30d" strings, so a lexicographic ORDER BY puts
"30d" ahead of "7d".
tpm_limit and rpm_limit are BigInt? in Prisma, so rows validate through a
pydantic model on the way out and serialize as JSON numbers.
A caller without admin view is refused 403 as a problem document rather than
served an empty page. /budget/list is untouched.
* fix(proxy): rework the budgets list onto the merged list contract
PR #35308 landed a different shape than this branch was written against: `where`
is a tuple of frozen predicates rather than a Prisma-shaped mapping, `ListSpec`
carries both the row and the wire type, and `where_sql` / `order_by_sql` render
for a raw-SQL executor. The budgets executor now queries through `query_raw` the
way the spend logs facet does, selecting only the columns it serves.
Also casts datetime binds in `where_sql`. They cross into the query engine as
JSON, so an uncast placeholder arrives as text and Postgres refuses
`timestamp >= text` outright; every `filter[created_at][gte|lte]` was answering
500. The cast reads the bind as an instant and drops it to naive UTC to match
Prisma's TIMESTAMP(3) column, the same one /spend/logs/ui applies.
* refactor(proxy): fold the predicate renderer instead of recursing
recursive_detector flags `_render_all`, and the flag is fair: it recursed once
per predicate, so the stack grew with the number of filters on the request for
no reason. Walking a predicate list is a running bind index, which is a fold.
`_render` still re-enters for `AnyOf`, but its clauses are plain comparisons
built by `?q=`, so that nesting is one level deep and no caller can drive it
deeper.
* fix(tool-management): drop unsupported prisma select kwarg from team lookup
POST /v1/tool/policy returned HTTP 500 for every request carrying a team_id,
with "LiteLLM_TeamTableActions.find_unique() got an unexpected keyword argument
'select'". _resolve_team_id_to_object_permission_id looked the team up with
select={"object_permission_id": True}; prisma-client-py 0.11.0 has no select
kwarg on find_unique, so the call raised TypeError and the handler's except
block turned it into a 500. Both the initial read and the fallback read taken
when a concurrent writer wins the update_many race were failing the same way,
so per-team tool blocking was unreachable
The kwarg is dropped rather than replaced; the generated client has no
projection API, and this is a single-row lookup where selecting all columns
costs nothing worth working around
The existing tests missed this because the team table was an AsyncMock, which
accepts any keyword. The new double binds each call against the real
find_unique and update_many signatures, so an unsupported kwarg raises the same
TypeError production does
* refactor(test): tighten typing on the tool policy team table double
Replaces the double's Any annotations and bare list types with concrete ones:
kwargs are object, rows are Sequence[MagicMock] held as a tuple, and the call
logs are list[dict[str, object]]. Behaviour is unchanged; the double still
binds every call against the real generated prisma action signature, verified
by reintroducing the select kwarg and watching both regression tests fail
The management list route now exists, so budgetItem, the list envelope and
the response type come from schema.d.ts instead of being hand-written
against the contract. The optional fields widen accordingly, so the rate
limit and reset cells accept undefined alongside null.
