Concurrent completions read-add-set the global proxy spend scalar in the management cache, so simultaneous callbacks lost increments. Track proxy-wide spend in the shared spend counter (spend:user:litellm-proxy-budget) instead, which increments atomically and reseeds from the durable proxy budget row.
Fixes#35567
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): show pass through route selections in team/key forms and match team id substrings in team search
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(teams): keep team id search index-friendly with a prefix match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(teams): keep /v2/team/list search id matching exact by default and add an opt-in prefix mode
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): stop model writes 500ing on another pod's delete
A model write judges the reload it triggers by diffing this pod's router before and
after, and reports anything that stopped serving as damage. On a pod that has not yet
polled a delete another pod made, the snapshot still lists that model; the reload then
evicts it because the db no longer has it, and the guard reads its own correct
reconcile as degradation. The row is written and served, but the caller gets a 500.
Since propagation between pods is a 30s db poll, any delete followed by a create
inside that window can land on a pod that has not caught up, so a delete-then-create
pair returns 500 whenever the two requests hit different pods.
_delete_deployment already computes exactly the set that settles it: the ids the db
and config still want. Thread it up through _update_llm_router, add_deployment and
clear_cache to the verdict, and intersect the drop set with it so an id the db no
longer has stops counting as collateral. Where no reconcile ran the set is None and
every drop is still reported, so a genuinely broken reload is caught as before.
_delete_deployment now returns that set instead of a delete count; the count had no
callers in the proxy, and the tests asserting it already assert the eviction calls.
* test(proxy): fold reload-verdict test commentary into docstrings and assertions
Greptile flagged the inline comments against the repo's no-new-comments rule. The
case-by-case context moves into the test docstring, and the two return-contract
assertions carry their reasoning as failure messages instead.
* test: fix clear_cache mock return type in model block/unblock tests
The comment said the Responses WebSocket route never runs
add_litellm_data_to_request. It does, via common_processing_pre_call_logic,
so the note recorded a request-flow constraint that does not hold
The gate stays in route_request, which is the dispatch chokepoint and where
the previous handling lived
Handling of the client-supplied mock testing params was split across three
places with different behavior for each. Three were dropped from every proxy
request, two reached the router untouched, and a request that asked for a
synthetic failure came back as an ordinary success with nothing to indicate
that no failure had been injected
Put all six behind one opt-in, general_settings.
dangerously_allow_mock_testing_request_params, and reject rather than drop
when it is unset, so a fallback drill cannot report a pass for a test that
never ran. The rejection names the params it saw and the config key to set,
which is also the answer for anyone following the older docs
The flag is config-file only. It is deliberately absent from
ConfigGeneralSettings, and that absence is what makes /config/update drop it
on parse and /config/field/update reject it; the tests pin both so the field
cannot be added back for tidiness without the reason surfacing. Enabling it
logs a startup warning naming every param it unlocks
BREAKING CHANGE: mock_timeout and mock_testing_rate_limit_error now require
general_settings.dangerously_allow_mock_testing_request_params to be set in
config.yaml. Previously they were accepted unconditionally
Tool-level MCP entitlements are enforced in one place,
check_tool_permission_for_key_team, reached from pre_call_tool_check. Two
dispatch paths reached a tool handler without passing through it.
execute_mcp_tool's legacy fallback dispatched into the local tool registry
after retrying the unprefixed name, with no allowed/banned-tool check, no
key/team/org tool permissions and no parameter validation. It now runs the same
gate, and only when something can actually dispatch: when the unprefixed name is
absent from the local registry too, the existing 404 stands rather than becoming
a misleading "server unavailable".
The server the tool-level checks need is available even though the tool name is
not in the tool -> server mapping: a non-empty prefix has already been compared
against the caller's allowed_mcp_servers by exact name, so the named server is
in that list. It is resolved from allowed_mcp_servers rather than from the
manager's registry, because the registry can return a server the caller holds no
grant for, and matching on anything other than name would accept a server the
server-level check never validated. The remaining case is a prefix segment that
is empty, which the server-level check skips entirely because it is gated on a
non-empty server name; that now fails closed with 503 instead of dispatching for
a caller holding no server grant at all.
An entitled caller's legacy call therefore still dispatches, so a configuration
that worked before keeps working; only the unentitled call is refused, now with
the entitlement gate's own 403.
call_tool ran pre_call_tool_check inside `if proxy_logging_obj:`, so an absent
logging object would have skipped authorization silently. This half is defensive
with no live hole: all four call sites source the module-level ProxyLogging
singleton from proxy_server.py, which is never None. The shape was still wrong.
pre_call_tool_check now runs its three authorization checks unconditionally and
only the guardrail hooks, which are dispatched through the logger, depend on one
being present.
A third reported path, where allow_all_keys, BYOM-submitted and
upstream-delegated servers are unioned in after the resolver's ceilings, was
investigated and found not to be a defect. The widening is real, but a server's
tool surface is already boundable for every caller at registration through
MCPServer.allowed_tools / disallowed_tools, enforced by
check_allowed_or_banned_tools ahead of the entitlement check, and per-caller
narrowing plus the org tool ceiling remain available. Nothing here changes that
path.
Resolves LIT-4956
Opening a session from the logs table stored no ?session_id (row clicks
called openLog, which deletes it), so session mode was derived from the
clicked row's session_total_count. Rows fetched by the session drawer
come from /spend/logs/session/ui, which does not enrich that field, so
selecting any log inside the session view swapped in an unenriched row
and collapsed the drawer to a single-log Trace view
Row clicks on a multi-call session's row now call openSession, and
selectLog writes ?session_id when the session view is active, so session
mode is anchored in the URL instead of derived from row data
The create button now sits in the tab bar beside the tabs, the way Teams
lays it out, with one divider between them and the rule running the full
width underneath.
Adds a fillHeight mode to DataTable that treats the parent's height as a
ceiling rather than a target, so the table still sizes to its rows and a
short one keeps its footer under the last row, while a long one scrolls
its rows under a sticky header instead of scrolling the page. This replaces
the hardcoded viewport-height caps those tables would otherwise need. Two
details the mode has to fix: the Table primitive's own overflow container
would capture the sticky header, and rows would show through the
semi-transparent header tint.
The hook resolves the newly created user through UserRepository and builds the
audit entry from that row. Pin both halves: the entry carries the persisted
row's fields rather than the /user/new response, and a user id that resolves to
nothing produces no entry at all.
Matches the Virtual Keys layout: a page header with the wallet icon, the
create button directly beneath it, and the tab bar below that, on the same
page padding Teams and Access Groups use so the table no longer sits against
the window edge.
Reset and Created start hidden, so the table opens on the four columns it
has always shown and the two new ones are opt-in from the Columns menu.
Swap `Model(**payload)` for `Model.model_validate(payload)` at the seams where
the payload comes back untyped, so basedpyright stops widening every target
field to Any. None of the models involved override `__init__`, so validation
goes through the same core validator either way.
Also route UserRepository through its own typed helpers (find_many, update,
find_by_id) instead of the raw Prisma table, drop the redundant `_to_model`
override signature, and call generate_key_helper_fn with explicit arguments in
the SSO callback rather than splatting an untyped dict.
Whole-tree basedpyright: reportAny 21481 -> 20834, reportExplicitAny 7258 ->
7252, with every other rule unchanged or lower.
* feat(spend-logs): record when a spend log row is the auto-router's own classifier call
The complexity router's classifier sub-call copies the parent request's metadata
verbatim, so its spend log row carries the caller's key, team and user and is
indistinguishable from traffic the caller actually sent. Nothing on the row says
otherwise: call_type is "acompletion" either way, model_group is overwritten to the
classifier's own model group so the row never looks auto-routed, and routing_decision
is absent exactly as it is on an ordinary request.
Record the fact the system already knows at call time. internal_call_origin is
declared on SpendLogsMetadata, which is the allowlist _get_spend_logs_metadata
projects onto, and stamped in _classifier_call_metadata; both classifier paths
already route through that one function and it feeds the metadata and
litellm_metadata buckets alike, so every request surface is covered at one site.
The key is reserved rather than caller-supplied, so it joins routing_decision in the
untrusted-metadata strip and a caller cannot label their own traffic as router
overhead.
The classifier call also inherited no session identity, so the router minted a fresh
trace id and the row landed in a session of its own. Forwarding the parent's session
puts it in the trace of the request that triggered it, which is where an operator
looks for what the routing cost.
* feat(ui): show which log rows are the auto-router's own classifier calls
A classifier row now carries internal_call_origin and shares its parent's session,
so the session trace lists it beside the request that triggered it. Without a marker
in the sidebar it reads as another call the caller made, which is the confusion this
resolves.
The tag renders only for a recognized origin, so ordinary traffic and any future
origin this build does not know about stay unlabelled rather than being asserted as
classifier calls.
* feat(proxy): add a generic list contract for management/v1 entity lists
Paging, sorting, filtering and search for an entity collection, declared once
as a ListSpec and served by handle_list. The route injects a ListExecutor that
owns its table, so this module never imports Prisma.
The caller's scope is derived from the caller alone and ANDed with whatever
they filtered on, so a query parameter can only narrow what they may read.
This is the shared half of the budgets list; it lands here so the endpoint has
something to register against, and drops out when the framework arrives on its
own branch.
* feat(proxy): add GET /management/v1/budgets
The Budgets page reads /budget/list, which returns the whole table as a bare
array with no way to page, sort or filter it. A customer with enough budgets to
fill the page has no way to find one.
Registers LiteLLM_BudgetTable against the management/v1 list contract: sortable
on budget_id, max_budget, tpm_limit, rpm_limit and created_at, default order
newest-first with budget_id breaking ties, search on budget_id, and filters for
budget_duration, max_budget and created_at. budget_duration is deliberately not
sortable; the column holds "7d"/"30d" strings, so a lexicographic ORDER BY puts
"30d" ahead of "7d".
tpm_limit and rpm_limit are BigInt? in Prisma, so rows validate through a
pydantic model on the way out and serialize as JSON numbers.
A caller without admin view is refused 403 as a problem document rather than
served an empty page. /budget/list is untouched.
* fix(proxy): rework the budgets list onto the merged list contract
PR #35308 landed a different shape than this branch was written against: `where`
is a tuple of frozen predicates rather than a Prisma-shaped mapping, `ListSpec`
carries both the row and the wire type, and `where_sql` / `order_by_sql` render
for a raw-SQL executor. The budgets executor now queries through `query_raw` the
way the spend logs facet does, selecting only the columns it serves.
Also casts datetime binds in `where_sql`. They cross into the query engine as
JSON, so an uncast placeholder arrives as text and Postgres refuses
`timestamp >= text` outright; every `filter[created_at][gte|lte]` was answering
500. The cast reads the bind as an instant and drops it to naive UTC to match
Prisma's TIMESTAMP(3) column, the same one /spend/logs/ui applies.
* refactor(proxy): fold the predicate renderer instead of recursing
recursive_detector flags `_render_all`, and the flag is fair: it recursed once
per predicate, so the stack grew with the number of filters on the request for
no reason. Walking a predicate list is a running bind index, which is a fold.
`_render` still re-enters for `AnyOf`, but its clauses are plain comparisons
built by `?q=`, so that nesting is one level deep and no caller can drive it
deeper.
* fix(tool-management): drop unsupported prisma select kwarg from team lookup
POST /v1/tool/policy returned HTTP 500 for every request carrying a team_id,
with "LiteLLM_TeamTableActions.find_unique() got an unexpected keyword argument
'select'". _resolve_team_id_to_object_permission_id looked the team up with
select={"object_permission_id": True}; prisma-client-py 0.11.0 has no select
kwarg on find_unique, so the call raised TypeError and the handler's except
block turned it into a 500. Both the initial read and the fallback read taken
when a concurrent writer wins the update_many race were failing the same way,
so per-team tool blocking was unreachable
The kwarg is dropped rather than replaced; the generated client has no
projection API, and this is a single-row lookup where selecting all columns
costs nothing worth working around
The existing tests missed this because the team table was an AsyncMock, which
accepts any keyword. The new double binds each call against the real
find_unique and update_many signatures, so an unsupported kwarg raises the same
TypeError production does
* refactor(test): tighten typing on the tool policy team table double
Replaces the double's Any annotations and bare list types with concrete ones:
kwargs are object, rows are Sequence[MagicMock] held as a tuple, and the call
logs are list[dict[str, object]]. Behaviour is unchanged; the double still
binds every call against the real generated prisma action signature, verified
by reintroducing the select kwarg and watching both regression tests fail
The management list route now exists, so budgetItem, the list envelope and
the response type come from schema.d.ts instead of being hand-written
against the contract. The optional fields widen accordingly, so the rate
limit and reset cells accept undefined alongside null.
A bedrock guardrail configured mode: during_mcp_call never ran. ProxyLogging
remapped the event to during_mcp_call and dispatched, but bedrock's own
async_moderation_hook then hard-coded during_call and re-checked, so the second
check rejected the very requests the guardrail was configured for and the tool
call proceeded unscanned with no error.
Remap call_mcp_tool the way model_armor already does, which matches the remap
ProxyLogging.during_call_hook itself performs, and teach the shared
get_guardrails_messages_for_call_type helper that an MCP tool call carries its
payload in the same messages key, without which the hook passes the gate and
then bails on an empty message list.
recursive_detector flags `_render_all`, and the flag is fair: it recursed once
per predicate, so the stack grew with the number of filters on the request for
no reason. Walking a predicate list is a running bind index, which is a fold.
`_render` still re-enters for `AnyOf`, but its clauses are plain comparisons
built by `?q=`, so that nesting is one level deep and no caller can drive it
deeper.
The SLO measures how many gateway replicas happen to be warm rather than the
request path. Clearing the floor needs roughly 5-7 replicas at ~10-14 RPS each;
stage idles at one and reactive HPA scale-up lands minutes into a ~3 minute
test.
It failed both of its assertions on consecutive days: 93.3% errors at an
inflated 264 RPS, where the failing requests never reached a pod and closed-loop
RPS rose because they failed fast, then 16.7 RPS with zero failures.
The covers marker and registry row stay put, so the cell returns to the gap
list rather than disappearing from the denominator.
The ID-JAG egress arm could only assert a caller that presented its own IdP
identity token on the request, so an agent holding a brokered LiteLLM
credential got a 412 and never reached the upstream. The assertion captured at
SSO login was already persisted per user for exactly this purpose, but nothing
read it back.
The arm now falls back to that stored assertion, keyed on the authenticated
principal's user_id. The identity is always taken from the credential the
gateway authenticated, never from a caller-supplied field, so no caller can
select whose identity is asserted upstream. A missing, expired, or
unidentified subject stays a 412; ID-JAG exists to assert a specific user and a
missing subject has no safe substitute. A store outage is the one exception: it
is surfaced as a typed AssertionStoreUnavailable and mapped to 503, so a
database blip cannot 500 the egress or the upstream-401 retry, and does not
tell the user to sign in again over something they cannot fix.
Sourcing a subject from the store rather than the request changed what
invalidation can rely on, so the exchanged-token cache changed with it. The
entry is now addressed by a slot key derived from the principal, plus the
caller's own token when it presented one, with a fingerprint of the subject
token and config stored beside the bearer and compared on every read. A
mismatch reads as a miss and re-mints, so a rotated assertion or an edited
server config cannot be served a bearer authorized under the old inputs, and
two callers cannot receive each other's. Invalidation is a single delete of a
key it can always compute, needing no store lookup on the recovery path.
The upstream-401 invalidate-and-retry path was also gated on a truthy inbound
subject token, which skipped recovery entirely for store-sourced calls. The
gate is now mode-aware: token_exchange still requires an inbound token because
it has nothing else to mint from, id_jag does not.
oauth2_id_jag is also now selectable in the admin dashboard with its own field
set, instead of being reachable only from config.yaml or the REST API. The
auth-type selects drop antd list virtualization: at eleven options the last one
no longer mounts, which is a scroll in a browser but makes the option
unreachable to anything reading the rendered list.
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
PR #35308 landed a different shape than this branch was written against: `where`
is a tuple of frozen predicates rather than a Prisma-shaped mapping, `ListSpec`
carries both the row and the wire type, and `where_sql` / `order_by_sql` render
for a raw-SQL executor. The budgets executor now queries through `query_raw` the
way the spend logs facet does, selecting only the columns it serves.
Also casts datetime binds in `where_sql`. They cross into the query engine as
JSON, so an uncast placeholder arrives as text and Postgres refuses
`timestamp >= text` outright; every `filter[created_at][gte|lte]` was answering
500. The cast reads the bind as an instant and drops it to naive UTC to match
Prisma's TIMESTAMP(3) column, the same one /spend/logs/ui applies.
All three send a `telemetry` object in the arguments to Datadog's
search_datadog_logs tool. Datadog tightened that tool's input schema to reject
unknown properties, so every call now fails validation with 'unexpected
additional properties ["telemetry"]' before the behavior each test exists to
prove is reached.
`telemetry` was never a documented Datadog parameter; the tests relied on the
server ignoring extra properties. The proxy transmitted exactly what the tests
supplied and surfaced the upstream error faithfully, so this is test-side.
The covers markers and registry rows stay put: the collector counts a cell as
covered only when a test pytest would actually run declares it, so skipping
hands all four cells back to the gap list where they belong.
Replaces the double's Any annotations and List/Dict aliases with concrete
types, matching the equivalent double in the tool policy tests: kwargs are
object, records are Sequence[Mock] held as a tuple, and the call log is
list[dict[str, object]]. Behaviour is unchanged; the double still binds every
call against the real generated prisma action signature, verified by
reintroducing the select kwarg and watching the regression tests fail
* fix(aiohttp): dispose recycled client sessions deterministically
LiteLLMAiohttpTransport replaced its cached aiohttp.ClientSession on
loop-mismatch, loop-inspection failure, and "Session is closed" retry
without reliably closing the previous session:
- the close task from asyncio.create_task() was never referenced, so
it could be garbage-collected before running;
- the (RuntimeError, AttributeError) fallback branch replaced the
session without closing it at all;
- sessions bound to a closed event loop were abandoned to the GC
("rely on GC"), and sessions bound to a loop running in another
thread were closed from the wrong loop.
Replaced sessions surfaced as intermittent "Unclosed client session" /
"Unclosed connector" errors from the event-loop exception handler at
GC time.
_close_recycled_session() now covers the three lifecycles a recycled
session can be in: same-loop closes keep a strong task reference until
completion; sessions owned by a loop running elsewhere are closed on
their own loop via run_coroutine_threadsafe; sessions whose loop is
gone are disposed synchronously through the connector teardown that
aiohttp's own finalizer uses, which releases pooled connections and
silences the finalizer warnings.
Fixes#24230
* fix(aiohttp): guard threadsafe close callback against cancelled futures
---------
Co-authored-by: Anmol Jaiswal <68013660+anmolg1997@users.noreply.github.com>
* feat(proxy): add generic list handler for /management/v1
Adds the ListSpec/QueryPlan machinery the control-plane list endpoints are
meant to share, so a resource declares what it exposes instead of hand-rolling
its own paging, sorting and filter parsing.
build_query_plan is pure: it turns query parameters into a QueryPlan or an
RFC 9457 problem without any I/O, which is what lets the plan be asserted as a
value. The database half is a ListExecutor protocol injected by the caller, so
this module has no Prisma dependency at all.
Four things the framework guarantees rather than leaving to each resource: the
spec's unique tiebreaker is always the final sort key, so pages cannot repeat
rows when the leading column is all nulls; ordering is NULLS LAST in both
directions, since Postgres otherwise floats empty values to the top the moment
the sort direction flips; the scope predicate is a separate conjunct ahead of
every caller filter, so a filter on a scoped column cannot widen it; and a
denied scope is a 403 problem rather than a 200 with an empty list.
No route and no consumer yet; budgets registers against it next. The facet
endpoint's has_more shapes are untouched, and a test pins them so page mode
cannot quietly absorb them.
* fix(proxy): accept the bare filter[field] form in the list framework
Section 5 of the design doc spells equality without an operator bracket
(`?filter[status]=active`, and `/management/v1/keys?filter[team_id]=` in the
sub-resource paragraph); only the other operators carry a second bracket. The
parser only understood `filter[field][op]`, so the canonical spelling came back
as an unknown query parameter.
`filter[field]` now resolves to the field's `eq` operator, which means it still
goes through the declared operator set rather than around it: a field that does
not offer `eq` rejects the shorthand. The allowed-parameter list advertises the
bare spelling for `eq` and the bracketed one for everything else.
Drops two guards from the key parser that could not fire. Operator validation
already rejects every malformed operator, and `field in spec.filters` already
rejects every field nobody declared, so a well-formedness check on top of them
was unreachable; the tests cover the malformed keys directly instead.
* fix(proxy): validate list specs at construction and reject repeated params
Two gaps a review flagged on the list framework.
The page-size cap was only enforced against a supplied page_size, so a spec
whose default_page_size exceeded its max_page_size served more rows than the
resource allows on exactly the request that omits the parameter. A default of
zero was worse: it reached the total_pages division and made the resource 500 on
every request. ListSpec now validates 1 <= default_page_size <= max_page_size
when it is built, so a misconfigured resource fails as it is registered rather
than per request. default_sort is checked against sortable for the same reason;
caller-supplied sort was already validated, but the default never passed through
that path and a typo there reached the ORDER BY clause untouched. Raising is
right here despite the usual model-failures-as-values rule: there is no request
in flight and no caller to answer.
Repeated query parameters silently collapsed to their last value, so
?page=1&page=999 paged from 999 and a repeated sort key quietly won, which is
the same silently-altered-semantics failure the surface already rejects unknown
parameters to avoid. They are now a 400. The check lives in handle_list rather
than build_query_plan because a Mapping[str, str] cannot represent a repeat at
all; the boundary that can see one is the boundary that rejects it. A denied
scope still outranks it, matching every other rejection here.
Also corrects the order_by_sql docstring, which claimed every field reaching it
had been validated against sortable. That held for caller-supplied sort only.
* refactor(proxy): model list predicates as frozen values instead of dicts
The LIT002 budget rejected the framework: building a where-fragment meant a dict
literal per operator, and a dict keyed by a column name chosen at runtime cannot
be frozen into a TypedDict or a dataclass field, so there was no spelling of the
old shape the rule would accept.
Replacing the fragments with a tagged union removes the construction entirely. A
plan's where is now a tuple of frozen Compare / Within / IsNull / AnyOf, matched
exhaustively, and the field name is a value rather than a key. That also retires
the Mapping[str, object] the plan used to carry, which said nothing about what
was inside it and left the fragment shape as a convention two sides had to keep
agreeing on. Scope predicates take the same type, so a resource declares its row
filter in the same vocabulary rather than hand-rolling a backend dict.
where_sql renders a plan for a raw-SQL executor, binding every caller-supplied
value to a numbered placeholder and writing only spec-declared column names into
the statement. It is the counterpart to order_by_sql, which already existed for
the same reason: nulls ordering forces the executor onto raw SQL, so the escaping
and placeholder arithmetic belong in one reviewed place rather than in each
consumer.
Also folds the two remaining mutable builds out of the module (set comprehensions
and Counter to frozenset/tuple, the serialized page to a tuple pydantic coerces),
and lifts the LIKE escaper into common.py so the facet endpoint and the framework
share one copy instead of two that can drift.
No behavioural change to the facet endpoint; its tests, including the one pinning
the escaping, pass untouched.
An MCP permission level answers which servers and tools it permits, and a
level that answers nothing places no restriction. Key auth was reading a
lookup FAULT as that same answer, so the end user, agent and org ceilings
quietly disappeared for as long as one lasted, while the keyless
gateway-admitted path failed closed on the very same fault.
Those levels now separate the two fault classes the user level already
did. A principal row that NAMES an object_permission_id whose contents
cannot be read is a known entitlement with unknown contents, so it denies.
A lookup that fails before we can tell whether the principal is entitled
at all still places no ceiling, that being the state which existed before
the level did; denying there would refuse MCP to the majority of callers,
who have no such entitlement configured. The keyless path is unchanged.
Resolves LIT-4960
aiohttp 3.14.0 and 3.14.1 re-arm the sock_read timer on a keep-alive
connection after it has already been returned to the idle pool. The stray
timer stamps a SocketTimeoutError on the pooled connection without closing
it, so the pool keeps handing it out and the next request to pick it up
fails instantly on an error left behind by an earlier, unrelated request.
Because a single pool is shared across providers, the failures appear
simultaneously across Vertex AI, Bedrock, Anthropic and OpenAI-compatible
deployments as sub-millisecond "Connection timed out" errors.
uv.lock resolved aiohttp 3.14.1 and the published images install via
`uv sync --frozen`, so every image built from that lock shipped the
regression. The wheel's own metadata declared `aiohttp>=3.10,<4.0`, which
also left pip consumers free to resolve into the same broken window, so
both the runtime floor and the uv constraint move to >=3.14.2.
Upstream fixed this in aio-libs/aiohttp#12954, released in aiohttp 3.14.2;
the lock now resolves 3.14.3. Raising the floor rather than capping below
3.14 keeps the advisories that the existing 3.14.1 floor cleared, so no
osv-scanner ignores are needed. litellm requires Python >=3.10 and aiohttp
3.14.2 requires >=3.10, so no supported interpreter loses support.
Both new tests fail on the previous pins and pass on these.
PR #35185 added classifier_context_window_size and classifier_context_per_turn_chars
to ComplexityRouterConfig; they worked via config.yaml and the API but had no UI
control on the Add Model or Edit Auto-Router screens. Wires the two fields into
both, shown only when the LLM classifier is selected.
* fix(mcp): annotate connected-app reachability on the gateway connect page
The MCP connect page resolved its server grid through the dashboard identity
(admin shortcut or view_all returns the whole registry) while the gateway DCR
session it sets up resolves servers as an admitted subject through grant
sources only, so the page showed servers and tool counts the session is never
served. GET /v1/mcp/server now accepts connected_app_view=true and stamps each
returned server with connected_app_reachable, computed by the same
_reload_admitted_user + get_allowed_mcp_servers pair the live session uses.
The connect page requests the flag in connect mode and renders unreachable
servers dimmed with a label, excluded from the Connected count and tool-count
fetches. Failure to build the admitted set marks everything unreachable, which
matches what such a session would actually be served. Default behavior without
the param is unchanged for every existing consumer.
* fix(mcp): block connecting unavailable servers from the connect-mode detail view
A server the connect page marks unavailable could still be added through its
detail view Connect action, so the selection could contain servers the
connected-app session is never served. The unavailability decision now lives in
one predicate, connectUnavailabilityLabel, consumed by the card indicator, the
detail view action area, the toggle-on path, the oauth auto-select effect, and
the Connected count, so no interaction path can disagree with the label. This
also closes the same pre-existing hole for servers marked not supported on this
connection, whose detail view likewise offered Connect, and removes a
grandfathered nested ternary, ratcheting the eslint suppressions baseline down
* fix(mcp): hide unreachable servers on the connect page instead of dimming them
Product decision: the connect page should only show what a connected-app
session will actually be served, so annotated-unreachable servers are now
filtered out of the connect-mode list at fetch time rather than rendered
dimmed. Unsupported auth types keep their existing dimmed label since they are
a property of the server, not the caller. A user with zero reachable servers
gets an explanatory empty state pointing at grants. The list filter is the
single source: counts, tabs, auto-select, detail view, and tool-count fetches
all derive from the already-filtered state
* fix(mcp): guarantee the connect view lists every session-reachable server
The connect view's membership came from the dashboard resolver with the
admitted-subject answer only annotated on top, so a server reachable by the
session but missing from the dashboard list would be invisible on the page; an
under-report, the mirror of the bug this PR fixes. The connect view now unions
in any session-reachable server the dashboard resolver did not list, built
from the registry and redacted through the same ladder, so page membership
equals the admitted set by construction in both directions
* fix(mcp): honor connected_app_view only for the dashboard UI session credential
The reachability view resolves through the owning user's admitted identity, so
a caller-passed virtual key could use the param to enumerate servers beyond
its own scope (ids, names, descriptions of the owner's wider grants). The view
is now gated on is_ui_session_credential, a predicate factored out of
resolve_ui_session_team_ids so the two user-identity widening sites share one
trust boundary: the SSO-minted dashboard session token acting as its user. Any
other credential gets the param as a no-op and the admitted resolver is never
consulted for it
* fix(mcp): resolve UI sessions with the admitted-user context everywhere, not per endpoint
The list endpoint unioned in session-reachable servers itself while tool
counts, Connect actions, and credential endpoints still authorized through
build_effective_auth_contexts, whose contexts carry team grants but never the
user row's own object permission; a user-granted server could render on the
connect page while every interaction on it failed. The admitted-user context
(the same auth a gateway session resolves with) is now appended inside
build_effective_auth_contexts for UI session credentials, so the page list and
every per-server action endpoint answer identically, and the list endpoint's
one-off union is deleted. Caller-passed keys are still never widened
(is_ui_session_credential gate inside the context builder) and a reload
failure falls back to team contexts only
* fix(mcp): resolve non-admin dashboard sessions as the admitted subject on tool routes
Server reachability on the REST tool routes came from the widened context
union while tool permission checks ran on the bare session key, which carries
no object permission, so a dashboard user could invoke tools their user-level
grant excludes. Rather than bookkeeping which context granted which server,
the routes now choose one principal at the boundary: acting_user_auth swaps a
non-admin UI session for the admitted-subject auth, the same identity a
gateway session resolves with, so reachability, per-source fail-closed tool
ceilings, rate limits, and billing attribution all bind through the admitted
arms that already exist downstream. Admin sessions keep their operator view
and caller-passed credentials are never widened. One swap point per route,
no per-server principal picking, no parallel permission logic
* fix(mcp): derive the connect page's detail view from the reachable server list
The detail view held its own copy of the server object, so it outlived the list it came
from. When a refetch dropped that server as unreachable, the open detail view kept
rendering it and its Connect action still ran: the guard looked the server back up by id
or name in the current list, found nothing, and fell through, because a missing target
read as "nothing to block" rather than "no longer connectable"
Store the selected server's id and derive the row from the list instead. A server the
list no longer carries cannot be the detail view's subject, so the stale render, the
stale tools query and the guard bypass stop being reachable states rather than being
blocked one at a time. handleToggle now takes the server it is toggling, which deletes
the lookup that could miss at all
* refactor(mcp): one owner for the identity a dashboard session acts as
Three call sites reloaded the admitted subject independently, and the management
endpoint carried its own copy of the reload, the HTTPException swallow and the logging.
admitted_user_context is now the only place that answers "what user identity does this
dashboard session act as", and the connected-app reachability helper reads it, which
also drops its dead empty-user_id branch
That owner now carries the request's tracing span onto the admitted principal.
_reload_admitted_user builds a fresh auth from the user row and has no span of its own,
so swapping it in on the REST tool routes silently detached every downstream lookup and
the tool-call logging from the request's trace
Toolset scoping and the acting-as-user swap are mutually exclusive, so they now share
one owner on the tools list route. The admitted subject resolves per grant source and a
team source deliberately carries none of the caller's object_permission, so a toolset
narrowing layered on top would evaporate on every team-granted server: the request would
be admitted through the toolset grant and then served tools from servers the toolset
never named. A request carrying a toolset name stays on the caller's own credential,
exactly as it did before the swap
* fix(mcp): commit every async connect-page write against the list as it stands
Three continuations in the panel decided against state captured before their await and
committed after it, so a reachability refetch landing in between could not be seen
handleToggle validated the server at click time and then, once listMCPTools resolved,
wrote its name into the selection whatever the list had since become; a server the
refresh had dropped was selected anyway. It now re-asks connectableNow at the commit,
and that predicate resolves the id against the current list, so absence fails closed
instead of reading as nothing to block
The load pipeline was worse, because its cancel flag was shared across runs: the
successor's effect body reset it to false before the predecessor's fetch resolved, so a
superseded load could still run setServers and put the dropped server back on the page
outright. The flag is now a per-effect local that only that run's cleanup can clear,
which is also what makes unmount stop the chunked tool-count loop again. The load
passes its own liveness check down to the tool-count and oauth-status writes rather
than having them consult a flag they share with every other run
* fix(mcp): write the connect-page server list to its ref as it is committed
connectableNow resolves a server id against serversRef, but that ref was a mirror kept
in step by a passive effect, so it lagged the state it mirrored by however long React
took to render and flush. A continuation resolving inside that window read the previous
list: the commit-time reachability check would find a server the refetch had already
dropped, call it connectable, and select it, which is the mismatch the check exists to
prevent
The lag was the whole defect, so the mirror is gone. commitServers writes the ref and
the state together, at the one point the list is ever replaced, and the ref is now
never older than the last committed list. Readers that want the newest answer
(connectableNow, the oauth auto-select effect) get it; rendering still derives from
state, so what is on screen is unchanged
Pinned by a test that resolves the refetch and the in-flight Connect in the same tick,
with no render flushed between them, which is the interleaving the earlier regression
could not reach. The two prop mirrors are deliberately untouched: their staleness is
inherent to appending to a parent-owned list from an async callback rather than caused
by the mirror, and no reachability decision reads them
Greptile found that the OpenAPI fallback added a commit ago collapsed operation
IDs that registration keeps apart: foo/bar and foo.bar register as two tools but
sanitize_openapi_tool_name rewrites both to foo_bar, so a policy naming either
also decided the other.
The cause was two owners for one map, and picking the wrong one. Registration
names an operationId inline at _register_openapi_tools with
operation_id.replace(" ", "_").lower(), which keeps / and . ; the separate
sanitize_openapi_tool_name replaces every character outside [a-zA-Z0-9_-] and
belongs to register_tools_from_openapi, which has no production caller. Nothing
made the matcher use the one that actually registers, so it used the lookalike.
That inline expression is now openapi_tool_name in utils, and both registration
and the matcher call it. Replaying the registering function is the whole safety
argument, and it is structural rather than a claim: two operationIds that
register as two tools normalize to two names here by construction, because this
is the map that registered them. A coarser lookalike cannot be substituted
without a test failing.
The matcher also loses its exact-then-fallback split. The transform is identity
on native servers and idempotent on already-registered names, so normalizing
both sides is exact matching where no OpenAPI spec is involved. Executable lines
drop by three this round; the branch is +5 over the merge-base for four shared
owners that removed duplication at six call sites.
