The search tool create and edit forms both built a payload carrying
api_base, timeout and max_retries read off form values that neither
zod schema declares, so all three were always undefined. Drop them.
JSON.stringify omits undefined-valued keys, so the request body on
the wire is unchanged.
SearchToolLiteLLMParams and SearchToolInfo in the page's types.tsx
were hand-rolled with a [key: string]: any index signature, which is
why a param could go missing from a form with nothing complaining.
SearchToolLiteLLMParams is now the generated OpenAPI component and
neither type carries an index signature, so the payload builder can
only set params the backend declares.
Also remove a stray ", ]" text node that rendered as visible garbage
next to the connection test dialog's Close button.
* refactor(proxy): move the shared list framework to a surface-neutral package
The list framework and its RFC 9457 problem machinery sat under
management_endpoints/management_v1/, which was the right home while
/management/v1 was its only consumer. The public surface is about to build
on the same framework, and a control-plane package is the wrong thing for a
public route to import.
Moves list_framework.py in full, plus everything in common.py except
MANAGEMENT_V1_PREFIX, to litellm/proxy/list_api/. Every importer is updated
directly instead of leaving re-export shims, so each symbol keeps exactly
one import path. ManagementProblem keeps its name: renaming it would touch
the app-wide exception handler and every call site for no behavioural gain.
The framework's own tests move alongside the code they cover. The fastapi
removed-name guard in test_common.py now globs both packages, so budgets.py
and spend_logs.py stay covered after leaving the framework's directory.
Pure move, no behaviour change: the 179 tests across both packages pass
unchanged.
* feat(proxy): add paginated GET /public/v1/model_hub
The public Model Hub page loads every public model group in one call.
Measured on a live proxy with 300 published groups, /public/model_hub
answers with 328 KB in a single response and the page renders all 300 rows
into the DOM. At a few thousand models that is multiple megabytes and a
page that stops responding, which is what a customer reported.
Adds GET /public/v1/model_hub, the first resource on the unauthenticated
/public/v1 surface. It is built on the shared list framework, so it gets
the {data, meta, links} envelope, RFC 9457 problems, strict unknown and
duplicate query parameter rejection, and sort validation without
reimplementing any of it. Sorting covers model_group, mode, the token
limits and the per-token costs, `q` searches model_group, and the filters
are the ones the page actually offers: mode and providers. Default sort is
alphabetical, which is what a browse list wants and what these rows can
support: they carry no creation timestamp.
/public/model_hub is untouched. The shipped UI still calls it and its
migration is a separate change, so this is purely additive alongside it.
Model hub rows are computed off the running router rather than read from a
table, so this adds InMemoryListExecutor: the same QueryPlan applied in
Python instead of rendered to SQL. It matches the SQL executors where it
counts, NULLS LAST in both sort directions and NULL satisfying no
comparison, so a filter means the same thing on either. The other three
public hubs have the same shape and can reuse it as is.
The fix itself is ordering. The endpoint being superseded reads every
latest health check and joins it against the whole model list, so paging
the response alone would have changed nothing. Here the health lookup is
an injected dependency the executor calls on the page slice, after the
filter and the sort, so it resolves health for the rows being served and
no others. PrismaClient gains a bounded read for that, next to the
unbounded one it mirrors. The regression test pins the ordering by
asserting which model groups the lookup is asked about, and fails against
an enrich-then-slice implementation.
* fix(proxy): address self-review findings on the public model hub list
Five adversarial review passes over the branch. What they found:
`is_null` was the one predicate in the in-memory executor that read a
repeated field's container instead of its elements, so a field holding only
nulls was indistinguishable from a populated one. It now lifts over elements
like every other predicate does. Not reachable through this endpoint, whose
only repeated field grants `contains` alone, but the executor is written to
be reused by the other three hubs and the inconsistency was a trap for them.
The fastapi removed-name guard globbed the framework packages but not
`public_endpoints/public_v1`, which `proxy_server` also imports unguarded at
module level, so the new package had none of the protection the test claims
to give. It now covers all three.
Regenerates the dashboard's API types, which the OpenAPI sync check requires
whenever the proxy's route surface moves. The diff is the 65 generated lines
for the new operation and nothing else; no dashboard code changes here.
Also trims comments and docstrings that argued for a decision or restated a
signature rather than explaining code, and wraps a docstring line that ran
past 120 characters.
* ci: run the relocated list framework tests in the proxy-endpoints shard
The framework's tests moved from tests/test_litellm/proxy/management_endpoints,
which the proxy-endpoints shard claims, into a new tests/test_litellm/proxy/list_api
that no shard named. Both coverage guards caught it: the semantic shards have no
catch-all bucket, so the directory would have run nowhere.
Claims it alongside management_endpoints, where the same tests ran before.
* docs(proxy): stop restating the list spec in the model hub route docstring
The docstring listed every sortable field, the page-size cap and the filter
set, all of which already live in MODEL_HUB_LIST_SPEC and all of which the
endpoint hands back in the allowed array of a rejected request. Two copies of
one spec is a prose update owed on every change to the real one.
Keeps what a caller cannot derive from the endpoint itself: what the resource
is, that it needs no authentication, and a working example. Regenerates the
dashboard types, which carry the docstring as the operation description.
* fix(proxy): reject a repeated sort field instead of sorting by it twice
sort took any number of comma-separated keys, and the in-memory executor runs
one full sorted() pass per key before slicing. Naming one allowed field N times
therefore bought N passes over every published model group, synchronously on the
event loop, from a route that needs no credentials. Measured on 300 groups:
0.001s for one key, 0.034s for a thousand, 0.166s for five thousand, and it
grows with the catalogue this endpoint exists to make large.
A repeated field cannot change the ordering, so rejecting repeats costs a caller
nothing and bounds the passes at len(sortable), a number the spec author picks
rather than the caller. That beats an arbitrary cap: no magic number, and the
bound holds for every resource built on the framework.
The tiebreaker is appended after parsing, so sorting explicitly by it stays legal.
Budgets renders one ORDER BY in SQL and never had the amplification, but the
check belongs with the rest of the sort validation rather than in one executor.
* fix(proxy): make the search disjunction one level deep by type
Two CI gates, one cause. AnyOf declared its clauses as Predicate, so both
consumers had to recurse to evaluate one: the SQL renderer through
_render/_render_all, and the in-memory executor through _holds. The recursion
detector flags the latter, and its reason is the same one this PR already ran
into once, a caller-controlled cost that shows up as CPU.
Nothing actually builds a nested AnyOf. _search_predicate is its only producer
anywhere in the repo and it emits Compare leaves, in every call site and every
test. Declaring clauses as tuple[Compare, ...] makes that a fact the type
checker keeps rather than a comment, and _holds then evaluates a disjunction of
leaves with no recursion at all.
Also marks the new health read's broad except, which the strict gate counts,
and covers the ordering comparison operators. The endpoint exposes only
eq/in/contains, so gt/gte/lt/lte were live code no test evaluated.
* fix(proxy): keep the new health read inside the type-discipline ceiling
The bounded health query added ten LIT002 violations, which pushed the
codebase total past its budget. The gate counts across the tree and compares
to the merge base, so a file already carrying debt does not absorb new
violations.
Returns an empty tuple rather than an empty list on the two no-result paths:
the signature already promises a Sequence, so that is a free two-violation
reduction and a better type. Builds prisma's order argument from a tuple of
pairs, which turns four literals into one. The three that remain are prisma's
own API shape and each carries its reason.
Both budget gates now pass against the merge base.
* fix(proxy): clear the two basedpyright errors the new route added
The type-check budget is over its ceiling on the base already, so the gate
blames any increase: reportArgumentType 2574/2564 and reportPrivateUsage
1815/1808, one each, both from this file.
fastapi types a route's tags as list[str | Enum], so the tuple was an argument
error; budgets.py has the same one and it is part of what put the rule over.
Passing a list is what the signature asks for, marked because an inline list
is a construction the discipline gate counts.
_get_model_group_info is private by name but is the shared reader the endpoint
this supersedes imports the same way, so the import carries a rule-scoped
ignore with that reason rather than a copy of the function.
basedpyright now reports zero errors across both new modules, and all three
budget gates pass against the merge base.
* feat(ui): edit the auto-router tier set with custom classifier-defined tiers
The editor over the model layer beneath it. An Edit tiers button turns the tier
list into an editor: a tier takes a name, a classifier definition, and models,
between two and eight rows. Restore defaults resets to the built-in four rather
than stacking them on top. Keyword rules follow a rename, an orphaned rule
blocks the save, and both forms dry-run the exact payload against the backend
validator before writing.
The edit modal hydrates a stored custom set into rows, and an untouched
open-and-save round-trips byte-identically, per-model reasoning efforts
included. A form that never opens the editor submits the same bytes as before.
The cost-optimization tier chart renders arbitrary tier names: the guard that
returned no models for a non-built-in name is gone, and the fixed four-color
array gives way to the shared cycle.
* refactor(ui): extract tier editor sections to clear new lint warnings
* test(ui): drop narration comments per repo convention
* fix(ui): default editingTiers so the build's type check passes
* fix(ui): restore the mid-dry-run submit guard and its regression tests
Every test in the file seeds, reads and deletes the same fixed group and team
ids, and auth_ui_unit_tests runs pytest with -n 2. Two tests landing on the two
workers at once tread on each other: one worker's _clean_db DELETE wipes rows
the other just seeded, and its sync writes land in the other's read.
Both shapes showed up on 13a0976bb6, a commit that renames a passthrough test
and nothing else. test_reconcile_is_idempotent... read back an empty table, and
test_reconcile_handles_a_null_array_column read the idempotent test's team on
its own second group.
Scoping the ids to PYTEST_XDIST_WORKER keeps each worker in its own rows. Tests
on one worker still run in sequence, so no isolation is lost.
Reproduced against a local Postgres: -n 2 failed 6 out of 6 runs before, passed
6 out of 6 after, and serial runs are green either way. Stripping the COALESCE
guard from the mirror's SQL still fails the suite, so the ids are all that
changed.
Greptile flagged this test as coupled to OpenAI's availability. The coupling was
not the status assertion it pointed at, and it predates this PR: the test it
replaced called /v1/assistants live the same way, and pass_through_endpoints gates
success logging on `response.status_code < 400`, so an upstream outage has always
meant no log fires and the payload assertions fail regardless.
The target is now a local HTTP server on an ephemeral port, so the test is offline
either way. It still exercises the generic passthrough handler, since
_is_supported_openai_endpoint does not claim a 127.0.0.1 URL any more than it
claimed /v1/moderations, and it now also asserts what the upstream actually
received rather than only what came back.
respx was the obvious approach and does not work here: it patches httpx transports,
and the passthrough issues its request through the custom aiohttp transport, so the
call went to the real api.openai.com and returned 401 while respx sat unused.
Mutation checked: gating off the success enqueue fails the test, and tampering with
the logged response body fails it.
`_count_content_list` accepted text, image_url, tool_use, tool_result,
thinking and tool_reference, and raised on anything else, so an
Anthropic-native `{"type": "image", "source": {...}}` block aborted the
whole count. That is the documented Anthropic image format and exactly
what /v1/messages receives.
Three user-visible effects. /v1/messages/count_tokens and
/utils/token_counter return 500, and the router's context-window
pre-call check swallows the ValueError and returns every deployment
unfiltered, so an oversized prompt carrying an image is dispatched to
the provider instead of being rejected locally with a 400.
Prices the block through the existing image path: a base64 source
becomes a data URI, a url source passes through, and a file source
falls back to the default image token count. Blocks nested inside
tool_result.content are covered too, because _count_anthropic_content
recurses back into _count_content_list.
Fixes#36604
batch_completion collects per-request failures into its result list rather than
raising them; its own source says "return exceptions if any". So the test's
`except Timeout` and `except litellm.InternalServerError` arms could never fire for
the case they were written for. An upstream 500 instead reached
`response.choices`, raised AttributeError on the exception object, and fell through
to the bare `except Exception` that calls pytest.fail. That is what CircleCI hit.
The tolerance now reads the returned values, which is where the failures actually
are. The same two exception types are tolerated as before, nothing broader.
Checked against four injected outcomes: three InternalServerErrors pass, three
Timeouts pass, an AuthenticationError fails, and a response whose content is None
fails. So it is not tolerating its way to a vacuous green.
* feat(spend): report prompt caching savings as total and gateway-attributed
`prompt_caching_savings_spend` credited every cached request, including caching a
client asked for with its own `cache_control` and caching a provider does implicitly,
so the number overstated what the gateway had any hand in.
Gating that column in place would have fixed the overstatement by changing what the
column means, leaving rows written before the change saying "all caching savings" and
rows after saying "gateway-injected only" with nothing to tell them apart, and forcing
a decision about rewriting history. It also breaks the cache-leakage estimate on the
dashboard, whose numerator would be gated while its denominator, the cached token
counts, would not, so the rate it extrapolates from would be quietly diluted.
Report both instead. `prompt_caching_savings_spend` keeps meaning every net dollar
caching saved, which is what a customer means by "what did caching save me", and the
new `gateway_injected_caching_savings_spend` carries the subset litellm caused by
injecting the breakpoints itself. Both are derived from the same marker, so this
changes what is done with it rather than how it is obtained.
The attributed figure is normally the smaller of the two, being a subset of the same
requests, but not always: a request that writes cache it never reads has negative net
savings, and excluding such a request can lift the attributed figure above the total.
Also stops the marker riding into a fallback leg. The fallback rebuild spread the
failed attempt's metadata forward, so a deployment that injected nothing inherited the
marker and was credited anyway, which silently restored the very overstatement this
separates out.
* fix(bedrock): credit gateway caching where the tool cachePoint is placed (#38478)
The savings marker records breakpoints litellm placed, and a tool_config
injection point becomes one only in the converse transform, and only when the
request carries tools. The prompt hook cannot see either condition, so marking
on the point's presence credited request shapes that cached nothing, while
Bedrock tool caching the gateway did cause went uncredited.
Record it at the placement site instead. The marker's reader also resolves its
bucket by value now: litellm_params declares litellm_metadata as None on every
request, so asking the shared name resolver named a bucket that was not there
and the mark was dropped.
Both anchoring tests read the trigger's box before the click and the popup's box the
instant it turns visible. Base UI places the popup asynchronously and opening it can
shift the trigger, so both boxes could be sampled before the layout settled. The run
on 1eedaa3a43 missed by 4.2px (expected >= 446.015, got 441.799) on a tree with no UI
changes at all, having passed on 21092d633b, which differs only in a deleted python
test and a budget json.
Each assertion now re-reads both boxes under expect.poll. The conditions themselves
are unchanged: the popup must sit at or below the trigger's bottom edge in the first
test and must not overlap it in the second. Polling cannot mask a genuinely misplaced
popup, since one that never lands correctly still fails when the poll times out.
/v1/batches now rejects a missing input_file_id with a 400 through
raise_if_required_body_param_missing, so the contract negative that was
skipped for "500s instead of 400" passes as written. Verified against a
live proxy.
The three Datadog MCP tests were skipped because each one sent a
`telemetry` argument that search_datadog_logs rejects with "unexpected
additional properties". That argument was never a documented Datadog
parameter and no assertion reads it, so it is dropped and the tests run
again unchanged otherwise.
Azure now answers the create call with 410 and code assistants_api_deprecated:
"The Assistants API has been retired. Follow the migration guide to update your
workloads." Microsoft retired it on the same day OpenAI retired theirs, which is
why this landed with the OpenAI ones rather than before them.
This job runs pytest with -x, so the test was also hiding everything after it in
tests/pass_through_tests.
test_pass_through_file_operations stays. It only asks /v1/files for
purpose="assistants", which still answers 200 on both upload and delete.
test_opik_logging_http_request asserted "nothing has been POSTed yet" roughly one
second into a window governed by OpikLogger's 5-second periodic flush. On a loaded
CI worker the five preceding acompletion calls eat that budget, the periodic flush
fires, and the assertion flips. Reproduced with no product changes at all: letting
5.5 seconds pass before the assertion drains the queue and sets mock_post.called,
which is exactly the failure CircleCI reports.
The test now pins flush_interval past anything the test can reach, so the two
batching assertions measure batching instead of wall clock, and drives the flush
path explicitly at the end rather than sleeping the interval. That last phase used
to be near-vacuous, since the size-triggered flush had already emptied the queue.
Assertions now match only calls to Opik's own /traces/batch and /spans/batch.
get_async_httpx_client caches one client per special provider, so the mock is
process-wide and any other logging callback's POST would otherwise count.
Dropped the teardown that closed that shared client, which broke every later test
in the same worker that logs through it, and the try/except that turned assertion
failures into a pytest.fail with no traceback.
Mutation checked: flushing on every event and never flushing on size both fail the
test.
Two unrelated causes, both leaving staging red with tests that no longer describe
anything true.
#38264 gave LangFuseLogger a langfuse_environment argument and started carrying it
in the credentials dict. The handler test's fake logger did not accept the new
keyword, so constructing it raised TypeError, and four cases in
test_langfuse_unit_tests rebuilt the cache key by hand from three fields and missed
on the four-field key production now writes. Caching itself was never broken: the
handler sets and gets with the same dict. The fake now takes the argument and
asserts it is forwarded, and the cache assertion issues a second identical request
and expects the same logger back, which is the behaviour that matters and cannot
rot the next time a credential field is added.
OpenAI has retired the Assistants API. /v1/assistants and /v1/threads both answer
404 with a valid key, where every live route answers 401, so nothing calling them
can pass again. test_custom_logger_passthrough covered generic passthrough logging
and only used assistants because it is a route with no provider-specific handler;
it moves to /v1/moderations, which is still unclaimed by
_is_supported_openai_endpoint, so the same generic branch is exercised. The two
tests there asserted the same thing against different dead routes, so they collapse
into one. The Ruby suite existed solely to drive assistants, threads, messages and
runs, so it goes along with the RVM and bundler steps that were installed only to
run it, and the two dead OpenAI assistants cases leave
test_openai_assistants_passthrough.
The Azure assistants case in that file stays. Azure runs its own lifecycle and I
could not reach the CI deployment to check whether that API is still there.
Resolve tests/llm_translation/test_together_ai.py in favor of staging:
bcb6a0a998 already landed the fail-open assertion for models missing from
the registry, so both models now list response_format and tools. This
branch's narrower gating of response_format no longer matches behavior.
A coding agent names each conversation by quoting the whole session and asking
for a title. The classifier rated the quoted session rather than the request, so
the cheapest call the client makes routed to the most expensive tier: 11 of 17
title generations in one day of real traffic came back COMPLEX.
Recognize those prompts by literal sentinel on the newest ask and route them to
the cheapest configured tier without classifying them, so the call costs nothing
to route. The placement is scoped to the one request that carries the sentinel:
it never displaces an operator's classifier plugin, the bandit cannot reach above
the tier as raised, it never becomes the session pin, and the sentinel that
matched is recorded on the routing decision. Detection reads the newest ask
alone, so a title request quoted into a later turn cannot cheapen the work that
follows it, and a keyword rule, an escalation keyword or the plan-mode floor all
still decide over it.
Regenerates the lazy OpenAPI snapshot, which was already stale on the base for an
unrelated Presidio guardrail field and failed the schema check on every PR.
Resolves LIT-6349
* fix(anthropic): handle per-level reasoning_effort flags without supports_reasoning
When a model has only per-level flags (e.g. supports_minimal_reasoning_effort: true)
but no explicit supports_reasoning flag, treat it as implicitly reasoning-capable.
This fixes gpt-5-search-api which declares minimal support but was incorrectly
degraded to low/minimal floor due to missing explicit supports_reasoning flag.
Test: verify per-level flag enables resolution path even without supports_reasoning.
Note: This change indirectly causes 20 azure deployments to forward max/xhigh
instead of degrading to high when requested, as these models now correctly
resolve their supported efforts through declared capability flags. This is
intended behavior (avoiding unnecessary degradation) but silent; operators
seeing increased latency/cost should check reasoning effort changes in logs.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(anthropic): explicit supports_reasoning=False wins over per-level flags
Greptile P1: the implicit-True branch bypassed the operator's explicit
supports_reasoning: false escape hatch when per-level flags were present
or inherited through the bare-twin lookup. Return () first on explicit
False, then apply the per-level implication only when the flag is unset.
Also drops a test comment that restated the test name (P2).
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(anthropic): drop and self-heal empty thinking blocks on /v1/messages
* test(anthropic): pin early-signature carry across the blank thinking chunk skip
#38318 taught exception_type to map upstream status codes for providers with
no branch of their own. It reads the status code off the exception, but
_handle_error stamps 500 onto every failure that never carried one, so a
refused connection reached the mapper wearing a status code nothing upstream
had sent, and came back as InternalServerError instead of APIConnectionError.
The two are not interchangeable to a caller: a 5xx says the provider answered
and failed, which the router treats as a reason to cool the deployment down,
while a connection error says the request never landed.
BaseLLMException now records whether its status code was received or
synthesized, _handle_error sets that when it invents the 500, and the status
mapper declines to act on a code litellm made up, so those failures fall
through to the APIConnectionError the branch was always meant to produce.
Genuine upstream 5xx responses are untouched, which the second test pins.
The search transformation assertion #38318 had loosened to InternalServerError
goes back to APIConnectionError for the same reason.
The Blog trigger carried `bg-transparent!`, which emits an important
background-color and so beat the non-important `hover:bg-accent` the shared
product-link class supplies. Docs lit up on hover and Blog stayed flat, the
same Docs/Blog inconsistency this branch is about on a different axis.
Dropping the override lets the shared hover through. `border-0!` stays, since
it keeps the trigger's box identical to the plain Docs anchor. Verified in the
browser: both now paint lab(96.1596 -0.0823438 -1.13575) on hover at 36px tall.
Docs went from a ghost Button to a plain anchor, which dropped the focus
treatment the Button was supplying, so tabbing to Docs showed nothing while
tabbing to Blog showed a ring. The ring now lives on the shared class both
sides use, matching the Button primitive's values.
Kept Docs as a real anchor rather than routing it back through Button:
nativeButton={false} stamps role="button" onto the element, so the old
DashboardHeader markup announced Docs as a button and lost its link
semantics. Tests pin both the ring and the link role.
Project auth expands all-proxy-models, * patterns, and access-group names
to many concrete models, but the rate limiter looks quotas up by the exact
requested model name, so a quota keyed on one of those entries is never
applied. Fail loudly with a 400 instead of storing an unenforceable quota
* fix(moonshot, together_ai): send the reasoning effort Kimi K3 accepts
Moonshot documents reasoning_effort as a top-level chat completions field for its reasoning
models, and defaults it to max, but MoonshotChatConfig builds its supported params by
subtracting from the OpenAI base list, which never carried that param. An explicit level
raised UnsupportedParamsError before the request left the proxy, so low and high were
unreachable and every call ran at the provider default
Together accepts low, high and max on Kimi K3. The per-model clamp added for the gpt-oss
family folds max down to high for every model except deepseek-ai/DeepSeek-V4-Pro, so a caller
asking for max silently got roughly half the reasoning budget they paid for
Moonshot now offers reasoning_effort whenever the registry says the model reasons. Together
sends a level the map entry declares unchanged, and keeps its existing table for every level
an entry does not name, so the only value that moves is Kimi K3 at max
* fix(moonshot): unwrap the bridges' effort object to the level string
* feat(ui): the model and wire layer for operator-defined auto-router tier sets
The data half of the custom tier set editor, with no visible UI change: the
editor lands separately on top of it.
One reader, activeTierRows, mints built-in rows with the canonical tier key as
their id, so the fallback pointer, the plan-mode floor and the per-model params
are row ids in both modes and nothing downstream branches on the mode. One
restrictions table carries each forbidden setting beside the reason shown for
it, so the greyed control and the omitted payload key cannot disagree. The
tier-set writes live in applyTierSetAction, where the fallback re-point and the
floor turn-off happen in one commit, unit-tested without a render.
buildComplexityRouterConfig emits tiers, tier_definitions and fallback_tier from
the rows, forces the LLM classifier, and strips what the backend rejects beside
tier_definitions. A payload built without a custom tier set is byte-identical to
what the form sends today.
* fix(ui): resolve frontend-lint failures on the tier-set model layer
* test(ui): drop a redundant explanatory comment per repo convention
* fix(ui): keyword rules follow their tier row through every tier-set action
