The generated UserItem calls metadata and model_max_budget free-form objects, which is true
of the columns and not assignable to what the details panel reads, so the response type now
restates them as the shapes /v2/user/info already claims.
The user edit form now saves through PATCH /management/v1/users/{user_id}
instead of /user/update, so emptying a control actually clears the setting
rather than saving as a no-op. TPM and RPM limits are editable and shown on
the details panel, and /v2/user/info returns them so the form seeds correctly.
FastAPI annotates tags as a list, so the tuple the route passes is a reportArgumentType
error even though any sequence works. Suppressed rather than switched to a list literal,
which LIT002 rejects.
The management_v1 fixture reached into UserRepository.table and the spend-counter helper to
stand the route up. Seeding prisma_client.db is enough, since the repository reads the table
off it, and the other two patches were doing nothing: the admin-name patch set the value the
constant already had, and the counter only fires when the payload carries spend.
The user_info_v2 tests each built their own prisma double, so a fixture now seeds the rows the
group reads and the tests just say which users exist.
The patch request and item models now annotate their collections read-only, and the
merge-patch resolvers hand back MappingProxyType rather than plain dicts, so the only
mutable dict left is the one the shared write path stamps into before Prisma takes it.
The resolver seam is a Protocol instead of a Callable alias.
Also regenerates _lazy_openapi_snapshot.json, which had drifted on the base branch, and
schema.d.ts, which the new route belongs in.
POST /user/update cannot clear a user setting. Its field resolution drops any null
(and any [] or {}), so a request asking to remove a tpm limit gets a 200 back with
the old value still in the database and nothing to say the clear was discarded.
The filter is a leftover. It was written when the request was serialized with
data.json(), which emitted every unset field at its non-None default (models=[],
metadata={}, spend=0), so without it every update wiped them. exclude_unset=True
has done that job since #10993, leaving the filter to swallow deliberate clears
only, which is why max_budget needed its own fields_set carve-out to become
clearable at all.
Rather than change what /user/update means for existing callers, the clearable
contract goes on a new control-plane route. An omitted field is left alone, an
explicit null clears the setting, unknown body keys are refused with a 422 problem
instead of ignored, and a user id that does not exist is a 404 rather than the
silent create the underlying upsert would otherwise do.
Only field resolution is new. Authorization, the self-escalation guard, metadata
merging, the entitlement upsert, audit logging and cache invalidation stay in the
one shared write path, now taking the resolver as a parameter, so the two surfaces
cannot drift on the parts that matter. That path's role guard also moves from
"user_role is not None" to presence, since a merge-patch caller clearing their own
role to null would otherwise have slipped past it.
Also adds the mutation machinery the surface was missing: an ItemResponse envelope,
problem_responses() for declaring RFC 9457 failures in OpenAPI, and a 422 branch in
the control plane's validation handler so a rejected body is no longer reported as
a query-parameter fault.
* feat(auto_router): write and preview the classifier prompt an edited tier set sends
An edited tier set replaces the whole rubric, so the built-in prompt editor is
refused there and the operator had no way to steer the classifier or add
calibration examples of their own. classification_prompt has always been
accepted beside tier_definitions as the rubric's opening; the dashboard just
never exposed it.
Custom mode gets its own Edit prompt dialog bound to that field. The dialog
previews the assembled prompt from the proxy, debounced against the draft, so a
built-in tier that leaves its description blank shows the shipped criteria it
inherits. The preview and the live classifier both call
custom_tier_classification_prompt, verified byte-identical against a running
proxy, so the preview cannot drift from what the router sends.
The preview POSTs on the same path as the shipped GET, because the prompt is the
operator's own text and must not reach access logs through a URL. The path joins
admin_viewer_routes so a role that may call the GET is not refused the POST, and
the request model applies the write gate's own strip and cap so the preview
refuses what the save would refuse.
* fix(ui): name the orphaned keyword rules inside the tier editor before Done
* fix(ui): drop stale classifier prompt preview responses
* style(ui): format the stale preview regression test
The unconditional region read let a base_model or custom pricing
deployment resolve to the regional cost-map key: a bedrock kimi
base_model shifted to regional rates and vertex claude-opus-5 with a
us-east5 key priced 0.0. Region now applies only when the model name
comes from the provider response (provider_response_model or the
response's own model), matching the base branch. Restores the #38069
regression test and adds region-on-provider-model and base-model-free
cases
The x-litellm-response-cost header on non-streaming /v1/messages responses is
recomputed from the response body because the Anthropic TypedDict cannot carry
hidden params. That recompute ran after the body's model field had already been
restamped to the client-facing alias, so the cost calculator priced the alias
(for example together_ai/muse-glimmer-30b) instead of the deployment model that
spend logging uses. On Together AI that alias is unregistered and falls into the
parameter-size bucket, so the header overbilled cold requests by about 2.3x and
priced cache reads at zero on warm ones while recorded spend stayed correct.
Move the restamp after every cost read of the response so the header and the
spend logs price the same model, and add a regression test that pins the header
to the provider-reported model while the body still returns the alias.
* fix: enforce MCP toolsets attached to a team, org, or internal user
object_permission.mcp_toolsets was resolved into servers and tools only at
the key level; every other principal read mcp_tool_permissions and silently
ignored its toolsets. A team/org/user toolset alongside a server grant was
inert (all tools callable), a toolset alone granted nothing, and an inert
team toolset let the org server list substitute for the empty team result,
handing the caller every org server.
Resolve toolsets at each level that resolves mcp_tool_permissions, union
their servers into that level's granted server set, and count a declared
key/team toolset toward has_lower_level_mcp_restrictions so the org list
can only cap, never substitute, even when the toolset resolves empty.
Resolves LIT-5749
* fix: deny when a team's declared MCP toolset cannot be resolved
The team server resolver swallowed UnloadableEntitlementError into an empty
list, so a dangling team toolset dropped the team ceiling instead of denying,
unlike the org and user paths. Re-raise it so the top-level resolver denies.
Also anchor the test-quality suppression comments on the patch opener lines
the gate reads, with per-seam reasons.
The previous check proved _load_endpoints_config returns a fresh object by
clearing the first result and reloading. That mutates shared state and only
works while the loader happens not to cache, so a future cache would corrupt
every later test rather than fail this one.
Compare the two loads by identity and equality instead. Verified red-before-green:
adding a module-level cache to the loader fails this test, removing it passes.
The search tool create and edit forms both built a payload carrying
api_base, timeout and max_retries read off form values that neither
zod schema declares, so all three were always undefined. Drop them.
JSON.stringify omits undefined-valued keys, so the request body on
the wire is unchanged.
SearchToolLiteLLMParams and SearchToolInfo in the page's types.tsx
were hand-rolled with a [key: string]: any index signature, which is
why a param could go missing from a form with nothing complaining.
SearchToolLiteLLMParams is now the generated OpenAPI component and
neither type carries an index signature, so the payload builder can
only set params the backend declares.
Also remove a stray ", ]" text node that rendered as visible garbage
next to the connection test dialog's Close button.
* refactor(proxy): move the shared list framework to a surface-neutral package
The list framework and its RFC 9457 problem machinery sat under
management_endpoints/management_v1/, which was the right home while
/management/v1 was its only consumer. The public surface is about to build
on the same framework, and a control-plane package is the wrong thing for a
public route to import.
Moves list_framework.py in full, plus everything in common.py except
MANAGEMENT_V1_PREFIX, to litellm/proxy/list_api/. Every importer is updated
directly instead of leaving re-export shims, so each symbol keeps exactly
one import path. ManagementProblem keeps its name: renaming it would touch
the app-wide exception handler and every call site for no behavioural gain.
The framework's own tests move alongside the code they cover. The fastapi
removed-name guard in test_common.py now globs both packages, so budgets.py
and spend_logs.py stay covered after leaving the framework's directory.
Pure move, no behaviour change: the 179 tests across both packages pass
unchanged.
* feat(proxy): add paginated GET /public/v1/model_hub
The public Model Hub page loads every public model group in one call.
Measured on a live proxy with 300 published groups, /public/model_hub
answers with 328 KB in a single response and the page renders all 300 rows
into the DOM. At a few thousand models that is multiple megabytes and a
page that stops responding, which is what a customer reported.
Adds GET /public/v1/model_hub, the first resource on the unauthenticated
/public/v1 surface. It is built on the shared list framework, so it gets
the {data, meta, links} envelope, RFC 9457 problems, strict unknown and
duplicate query parameter rejection, and sort validation without
reimplementing any of it. Sorting covers model_group, mode, the token
limits and the per-token costs, `q` searches model_group, and the filters
are the ones the page actually offers: mode and providers. Default sort is
alphabetical, which is what a browse list wants and what these rows can
support: they carry no creation timestamp.
/public/model_hub is untouched. The shipped UI still calls it and its
migration is a separate change, so this is purely additive alongside it.
Model hub rows are computed off the running router rather than read from a
table, so this adds InMemoryListExecutor: the same QueryPlan applied in
Python instead of rendered to SQL. It matches the SQL executors where it
counts, NULLS LAST in both sort directions and NULL satisfying no
comparison, so a filter means the same thing on either. The other three
public hubs have the same shape and can reuse it as is.
The fix itself is ordering. The endpoint being superseded reads every
latest health check and joins it against the whole model list, so paging
the response alone would have changed nothing. Here the health lookup is
an injected dependency the executor calls on the page slice, after the
filter and the sort, so it resolves health for the rows being served and
no others. PrismaClient gains a bounded read for that, next to the
unbounded one it mirrors. The regression test pins the ordering by
asserting which model groups the lookup is asked about, and fails against
an enrich-then-slice implementation.
* fix(proxy): address self-review findings on the public model hub list
Five adversarial review passes over the branch. What they found:
`is_null` was the one predicate in the in-memory executor that read a
repeated field's container instead of its elements, so a field holding only
nulls was indistinguishable from a populated one. It now lifts over elements
like every other predicate does. Not reachable through this endpoint, whose
only repeated field grants `contains` alone, but the executor is written to
be reused by the other three hubs and the inconsistency was a trap for them.
The fastapi removed-name guard globbed the framework packages but not
`public_endpoints/public_v1`, which `proxy_server` also imports unguarded at
module level, so the new package had none of the protection the test claims
to give. It now covers all three.
Regenerates the dashboard's API types, which the OpenAPI sync check requires
whenever the proxy's route surface moves. The diff is the 65 generated lines
for the new operation and nothing else; no dashboard code changes here.
Also trims comments and docstrings that argued for a decision or restated a
signature rather than explaining code, and wraps a docstring line that ran
past 120 characters.
* ci: run the relocated list framework tests in the proxy-endpoints shard
The framework's tests moved from tests/test_litellm/proxy/management_endpoints,
which the proxy-endpoints shard claims, into a new tests/test_litellm/proxy/list_api
that no shard named. Both coverage guards caught it: the semantic shards have no
catch-all bucket, so the directory would have run nowhere.
Claims it alongside management_endpoints, where the same tests ran before.
* docs(proxy): stop restating the list spec in the model hub route docstring
The docstring listed every sortable field, the page-size cap and the filter
set, all of which already live in MODEL_HUB_LIST_SPEC and all of which the
endpoint hands back in the allowed array of a rejected request. Two copies of
one spec is a prose update owed on every change to the real one.
Keeps what a caller cannot derive from the endpoint itself: what the resource
is, that it needs no authentication, and a working example. Regenerates the
dashboard types, which carry the docstring as the operation description.
* fix(proxy): reject a repeated sort field instead of sorting by it twice
sort took any number of comma-separated keys, and the in-memory executor runs
one full sorted() pass per key before slicing. Naming one allowed field N times
therefore bought N passes over every published model group, synchronously on the
event loop, from a route that needs no credentials. Measured on 300 groups:
0.001s for one key, 0.034s for a thousand, 0.166s for five thousand, and it
grows with the catalogue this endpoint exists to make large.
A repeated field cannot change the ordering, so rejecting repeats costs a caller
nothing and bounds the passes at len(sortable), a number the spec author picks
rather than the caller. That beats an arbitrary cap: no magic number, and the
bound holds for every resource built on the framework.
The tiebreaker is appended after parsing, so sorting explicitly by it stays legal.
Budgets renders one ORDER BY in SQL and never had the amplification, but the
check belongs with the rest of the sort validation rather than in one executor.
* fix(proxy): make the search disjunction one level deep by type
Two CI gates, one cause. AnyOf declared its clauses as Predicate, so both
consumers had to recurse to evaluate one: the SQL renderer through
_render/_render_all, and the in-memory executor through _holds. The recursion
detector flags the latter, and its reason is the same one this PR already ran
into once, a caller-controlled cost that shows up as CPU.
Nothing actually builds a nested AnyOf. _search_predicate is its only producer
anywhere in the repo and it emits Compare leaves, in every call site and every
test. Declaring clauses as tuple[Compare, ...] makes that a fact the type
checker keeps rather than a comment, and _holds then evaluates a disjunction of
leaves with no recursion at all.
Also marks the new health read's broad except, which the strict gate counts,
and covers the ordering comparison operators. The endpoint exposes only
eq/in/contains, so gt/gte/lt/lte were live code no test evaluated.
* fix(proxy): keep the new health read inside the type-discipline ceiling
The bounded health query added ten LIT002 violations, which pushed the
codebase total past its budget. The gate counts across the tree and compares
to the merge base, so a file already carrying debt does not absorb new
violations.
Returns an empty tuple rather than an empty list on the two no-result paths:
the signature already promises a Sequence, so that is a free two-violation
reduction and a better type. Builds prisma's order argument from a tuple of
pairs, which turns four literals into one. The three that remain are prisma's
own API shape and each carries its reason.
Both budget gates now pass against the merge base.
* fix(proxy): clear the two basedpyright errors the new route added
The type-check budget is over its ceiling on the base already, so the gate
blames any increase: reportArgumentType 2574/2564 and reportPrivateUsage
1815/1808, one each, both from this file.
fastapi types a route's tags as list[str | Enum], so the tuple was an argument
error; budgets.py has the same one and it is part of what put the rule over.
Passing a list is what the signature asks for, marked because an inline list
is a construction the discipline gate counts.
_get_model_group_info is private by name but is the shared reader the endpoint
this supersedes imports the same way, so the import carries a rule-scoped
ignore with that reason rather than a copy of the function.
basedpyright now reports zero errors across both new modules, and all three
budget gates pass against the merge base.
* feat(ui): edit the auto-router tier set with custom classifier-defined tiers
The editor over the model layer beneath it. An Edit tiers button turns the tier
list into an editor: a tier takes a name, a classifier definition, and models,
between two and eight rows. Restore defaults resets to the built-in four rather
than stacking them on top. Keyword rules follow a rename, an orphaned rule
blocks the save, and both forms dry-run the exact payload against the backend
validator before writing.
The edit modal hydrates a stored custom set into rows, and an untouched
open-and-save round-trips byte-identically, per-model reasoning efforts
included. A form that never opens the editor submits the same bytes as before.
The cost-optimization tier chart renders arbitrary tier names: the guard that
returned no models for a non-built-in name is gone, and the fixed four-color
array gives way to the shared cycle.
* refactor(ui): extract tier editor sections to clear new lint warnings
* test(ui): drop narration comments per repo convention
* fix(ui): default editingTiers so the build's type check passes
* fix(ui): restore the mid-dry-run submit guard and its regression tests
Every test in the file seeds, reads and deletes the same fixed group and team
ids, and auth_ui_unit_tests runs pytest with -n 2. Two tests landing on the two
workers at once tread on each other: one worker's _clean_db DELETE wipes rows
the other just seeded, and its sync writes land in the other's read.
Both shapes showed up on 13a0976bb6, a commit that renames a passthrough test
and nothing else. test_reconcile_is_idempotent... read back an empty table, and
test_reconcile_handles_a_null_array_column read the idempotent test's team on
its own second group.
Scoping the ids to PYTEST_XDIST_WORKER keeps each worker in its own rows. Tests
on one worker still run in sequence, so no isolation is lost.
Reproduced against a local Postgres: -n 2 failed 6 out of 6 runs before, passed
6 out of 6 after, and serial runs are green either way. Stripping the COALESCE
guard from the mirror's SQL still fails the suite, so the ids are all that
changed.
Mutation testing surfaced three factory functions whose tests ran against them
but asserted nothing that a mutation could break, so every planted bug survived.
- litellm/llms/litellm_proxy/skills/code_execution.py: the OpenAI and Anthropic
tool schemas were unpinned (the Anthropic one was not reached by any test at
all) and the handler's default fallbacks were unchecked
- litellm/containers/endpoint_factory.py: the endpoints.json contract, the
generated sync/async function set and the response-type mapping were unpinned
- litellm/llms/openai_like/dynamic_config.py: the generated Responses API config
class had no coverage of auth header, URL resolution or the store override
The openai_like tests clear _responses_config_cache around each test. Without
that, the module-level cache hands back a class built before the mutation and
the tests pass against mutated code.
Verified by re-running mutmut per scope:
llms/litellm_proxy 45.2% -> 62.8% (70 mutants newly killed)
containers 36.8% -> 84.3% (45 mutants newly killed)
llms/openai_like 55.7% -> 66.9% (34 mutants newly killed)
The models API reports 131072 input / 65536 output for the preview model
and the Interactions API accepts 100k tokens but rejects 130k, so the
1,048,576 input limit copied from the docs was wrong.
Gemini omni 1.1 flash and omni flash preview only answer on the Interactions
API, so both now list /v1beta/interactions as their endpoint and 1.1 flash
gets the 131072 / 65536 limits the models API reports.
grok-4.20-multi-agent and -latest now match the dated entry (mode responses,
/v1/responses only), and all three drop function calling and tool choice
since the API rejects client-side tools outside a beta.
kimi-k2.7-code gets the capability flags kimi-k2.6 carries (tools, reasoning,
JSON mode, image and video input) plus max_output_tokens.
grok-imagine-image-2.0 gets a low quality tier at $0.04 so quality=low is
not billed at the $0.06 default.
Greptile flagged this test as coupled to OpenAI's availability. The coupling was
not the status assertion it pointed at, and it predates this PR: the test it
replaced called /v1/assistants live the same way, and pass_through_endpoints gates
success logging on `response.status_code < 400`, so an upstream outage has always
meant no log fires and the payload assertions fail regardless.
The target is now a local HTTP server on an ephemeral port, so the test is offline
either way. It still exercises the generic passthrough handler, since
_is_supported_openai_endpoint does not claim a 127.0.0.1 URL any more than it
claimed /v1/moderations, and it now also asserts what the upstream actually
received rather than only what came back.
respx was the obvious approach and does not work here: it patches httpx transports,
and the passthrough issues its request through the custom aiohttp transport, so the
call went to the real api.openai.com and returned 401 while respx sat unused.
Mutation checked: gating off the success enqueue fails the test, and tampering with
the logged response body fails it.
`_count_content_list` accepted text, image_url, tool_use, tool_result,
thinking and tool_reference, and raised on anything else, so an
Anthropic-native `{"type": "image", "source": {...}}` block aborted the
whole count. That is the documented Anthropic image format and exactly
what /v1/messages receives.
Three user-visible effects. /v1/messages/count_tokens and
/utils/token_counter return 500, and the router's context-window
pre-call check swallows the ValueError and returns every deployment
unfiltered, so an oversized prompt carrying an image is dispatched to
the provider instead of being rejected locally with a 400.
Prices the block through the existing image path: a base64 source
becomes a data URI, a url source passes through, and a file source
falls back to the default image token count. Blocks nested inside
tool_result.content are covered too, because _count_anthropic_content
recurses back into _count_content_list.
Fixes#36604
