Commit graph

13809 commits

Author SHA1 Message Date
ryan-crabbe-berri
ae263fdcfc fix(ui): type the patch response's JSON columns as the shapes the details panel reads
The generated UserItem calls metadata and model_max_budget free-form objects, which is true
of the columns and not assignable to what the details panel reads, so the response type now
restates them as the shapes /v2/user/info already claims.
2026-08-28 13:37:15 -07:00
ryan-crabbe-berri
7080d51c39 feat(ui): clear user settings from the Admin UI and edit TPM/RPM there
The user edit form now saves through PATCH /management/v1/users/{user_id}
instead of /user/update, so emptying a control actually clears the setting
rather than saving as a no-op. TPM and RPM limits are editable and shown on
the details panel, and /v2/user/info returns them so the form seeds correctly.
2026-08-28 13:37:15 -07:00
ryan-crabbe-berri
423563ed9c test: patch the prisma client once per user test instead of once per collaborator
The management_v1 fixture reached into UserRepository.table and the spend-counter helper to
stand the route up. Seeding prisma_client.db is enough, since the repository reads the table
off it, and the other two patches were doing nothing: the admin-name patch set the value the
constant already had, and the counter only fires when the payload carries spend.

The user_info_v2 tests each built their own prisma double, so a fixture now seeds the rows the
group reads and the tests just say which users exist.
2026-08-28 13:18:01 -07:00
ryan-crabbe-berri
460d02ab2f feat(management-v1): add PATCH /management/v1/users/{user_id} with merge-patch semantics
POST /user/update cannot clear a user setting. Its field resolution drops any null
(and any [] or {}), so a request asking to remove a tpm limit gets a 200 back with
the old value still in the database and nothing to say the clear was discarded.

The filter is a leftover. It was written when the request was serialized with
data.json(), which emitted every unset field at its non-None default (models=[],
metadata={}, spend=0), so without it every update wiped them. exclude_unset=True
has done that job since #10993, leaving the filter to swallow deliberate clears
only, which is why max_budget needed its own fields_set carve-out to become
clearable at all.

Rather than change what /user/update means for existing callers, the clearable
contract goes on a new control-plane route. An omitted field is left alone, an
explicit null clears the setting, unknown body keys are refused with a 422 problem
instead of ignored, and a user id that does not exist is a 404 rather than the
silent create the underlying upsert would otherwise do.

Only field resolution is new. Authorization, the self-escalation guard, metadata
merging, the entitlement upsert, audit logging and cache invalidation stay in the
one shared write path, now taking the resolver as a parameter, so the two surfaces
cannot drift on the parts that matter. That path's role guard also moves from
"user_role is not None" to presence, since a merge-patch caller clearing their own
role to null would otherwise have slipped past it.

Also adds the mutation machinery the surface was missing: an ItemResponse envelope,
problem_responses() for declaring RFC 9457 failures in OpenAPI, and a 422 branch in
the control plane's validation handler so a rejected body is no longer reported as
a query-parameter fault.
2026-08-28 13:16:12 -07:00
Mateo Wang
d4f6b4491d
Merge pull request #38656 from aaaaaandrew/litellm_preserve_stream_selected_model
fix(streaming): preserve provider model for cost calculation
2026-08-28 13:00:35 -07:00
Mateo Wang
18830667b2
Merge pull request #38691 from BerriAI/litellm_messages_cost_alias_pricing
fix(proxy): price the /v1/messages cost header by the deployment model, not the client alias
2026-08-28 12:49:50 -07:00
tin-berri
1e86532c3d
feat(auto_router): write and preview the classifier prompt an edited tier set sends (#38605)
* feat(auto_router): write and preview the classifier prompt an edited tier set sends

An edited tier set replaces the whole rubric, so the built-in prompt editor is
refused there and the operator had no way to steer the classifier or add
calibration examples of their own. classification_prompt has always been
accepted beside tier_definitions as the rubric's opening; the dashboard just
never exposed it.

Custom mode gets its own Edit prompt dialog bound to that field. The dialog
previews the assembled prompt from the proxy, debounced against the draft, so a
built-in tier that leaves its description blank shows the shipped criteria it
inherits. The preview and the live classifier both call
custom_tier_classification_prompt, verified byte-identical against a running
proxy, so the preview cannot drift from what the router sends.

The preview POSTs on the same path as the shipped GET, because the prompt is the
operator's own text and must not reach access logs through a URL. The path joins
admin_viewer_routes so a role that may call the GET is not refused the POST, and
the request model applies the write gate's own strip and cap so the preview
refuses what the save would refuse.

* fix(ui): name the orphaned keyword rules inside the tier editor before Done

* fix(ui): drop stale classifier prompt preview responses

* style(ui): format the stale preview regression test
2026-08-28 12:36:52 -07:00
Mateo Wang
2896bbd5f4
Merge pull request #38653 from BerriAI/litellm_deflake_20260828
fix(tests): drain the global logging worker in RAG aquery billing tests
2026-08-28 12:19:27 -07:00
mateo-berri
c11a1f0bc1 fix(cost_calculator): scope region_name to response-derived model names
The unconditional region read let a base_model or custom pricing
deployment resolve to the regional cost-map key: a bedrock kimi
base_model shifted to regional rates and vertex claude-opus-5 with a
us-east5 key priced 0.0. Region now applies only when the model name
comes from the provider response (provider_response_model or the
response's own model), matching the base branch. Restores the #38069
regression test and adds region-on-provider-model and base-model-free
cases
2026-08-28 12:19:14 -07:00
mateo-berri
3daf7a3095 fix(proxy): price the /v1/messages cost header by the deployment model, not the client alias
The x-litellm-response-cost header on non-streaming /v1/messages responses is
recomputed from the response body because the Anthropic TypedDict cannot carry
hidden params. That recompute ran after the body's model field had already been
restamped to the client-facing alias, so the cost calculator priced the alias
(for example together_ai/muse-glimmer-30b) instead of the deployment model that
spend logging uses. On Together AI that alias is unregistered and falls into the
parameter-size bucket, so the header overbilled cold requests by about 2.3x and
priced cache reads at zero on warm ones while recorded spend stayed correct.

Move the restamp after every cost read of the response so the header and the
spend logs price the same model, and add a regression test that pins the header
to the provider-reported model while the body still returns the alias.
2026-08-28 12:09:25 -07:00
yucheng-berri
2ef77f30e3
fix: enforce MCP toolsets attached to a team, org, or internal user (#38488)
* fix: enforce MCP toolsets attached to a team, org, or internal user

object_permission.mcp_toolsets was resolved into servers and tools only at
the key level; every other principal read mcp_tool_permissions and silently
ignored its toolsets. A team/org/user toolset alongside a server grant was
inert (all tools callable), a toolset alone granted nothing, and an inert
team toolset let the org server list substitute for the empty team result,
handing the caller every org server.

Resolve toolsets at each level that resolves mcp_tool_permissions, union
their servers into that level's granted server set, and count a declared
key/team toolset toward has_lower_level_mcp_restrictions so the org list
can only cap, never substitute, even when the toolset resolves empty.

Resolves LIT-5749

* fix: deny when a team's declared MCP toolset cannot be resolved

The team server resolver swallowed UnloadableEntitlementError into an empty
list, so a dangling team toolset dropped the team ceiling instead of denying,
unlike the org and user paths. Re-raise it so the top-level resolver denies.

Also anchor the test-quality suppression comments on the patch opener lines
the gate reads, with per-seam reasons.
2026-08-28 12:06:57 -07:00
yuneng-jiang
4d9025c2bf
Merge pull request #38677 from BerriAI/litellm_/mutmut-test-coverage-gaps-b4dd83
test: close mutation-testing gaps in container, skills and openai-like config factories
2026-08-28 10:58:16 -07:00
yucheng-berri
c5cdd2e8e1
fix(logging): preserve null end user in callbacks (#38642) 2026-08-28 10:54:53 -07:00
Mateo Wang
5bcd494e88
Merge pull request #38560 from BerriAI/devin/1787857843-registry-audit-rolling
fix(registry): add Gemini Omni 1.1 Flash, xAI grok-imagine image models, Mistral cache-read pricing, GLM 5.3 Flash + Kimi K2.7 Code entries
2026-08-28 10:18:28 -07:00
Yuneng Jiang
5928556c16
test: assert endpoint config freshness by identity instead of mutating it
The previous check proved _load_endpoints_config returns a fresh object by
clearing the first result and reloading. That mutates shared state and only
works while the loader happens not to cache, so a future cache would corrupt
every later test rather than fail this one.

Compare the two loads by identity and equality instead. Verified red-before-green:
adding a module-level cache to the loader fails this test, removing it passes.
2026-08-28 10:11:55 -07:00
Mateo Wang
3a52ae0a3f
Merge pull request #38615 from BerriAI/litellm_mantle_gpt5_cost_tiers
fix(model_prices): add bedrock_mantle gpt-5.5/5.4 272K tiers, align sol with AWS invoice
2026-08-28 10:09:52 -07:00
Mateo Wang
e1cc96eaa7
Merge pull request #38609 from BerriAI/litellm_a2a_agent_semantic_search
feat(a2a): semantic search over the agent registry via GET /v1/agents?query and an agent_search MCP tool
2026-08-28 10:08:05 -07:00
Yuneng Jiang
0bba7f8869
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/mutmut-test-coverage-gaps-b4dd83 2026-08-28 10:05:50 -07:00
yuneng-jiang
0eb7c3ad05
feat(proxy): add paginated GET /public/v1/model_hub (#38636)
* refactor(proxy): move the shared list framework to a surface-neutral package

The list framework and its RFC 9457 problem machinery sat under
management_endpoints/management_v1/, which was the right home while
/management/v1 was its only consumer. The public surface is about to build
on the same framework, and a control-plane package is the wrong thing for a
public route to import.

Moves list_framework.py in full, plus everything in common.py except
MANAGEMENT_V1_PREFIX, to litellm/proxy/list_api/. Every importer is updated
directly instead of leaving re-export shims, so each symbol keeps exactly
one import path. ManagementProblem keeps its name: renaming it would touch
the app-wide exception handler and every call site for no behavioural gain.

The framework's own tests move alongside the code they cover. The fastapi
removed-name guard in test_common.py now globs both packages, so budgets.py
and spend_logs.py stay covered after leaving the framework's directory.

Pure move, no behaviour change: the 179 tests across both packages pass
unchanged.

* feat(proxy): add paginated GET /public/v1/model_hub

The public Model Hub page loads every public model group in one call.
Measured on a live proxy with 300 published groups, /public/model_hub
answers with 328 KB in a single response and the page renders all 300 rows
into the DOM. At a few thousand models that is multiple megabytes and a
page that stops responding, which is what a customer reported.

Adds GET /public/v1/model_hub, the first resource on the unauthenticated
/public/v1 surface. It is built on the shared list framework, so it gets
the {data, meta, links} envelope, RFC 9457 problems, strict unknown and
duplicate query parameter rejection, and sort validation without
reimplementing any of it. Sorting covers model_group, mode, the token
limits and the per-token costs, `q` searches model_group, and the filters
are the ones the page actually offers: mode and providers. Default sort is
alphabetical, which is what a browse list wants and what these rows can
support: they carry no creation timestamp.

/public/model_hub is untouched. The shipped UI still calls it and its
migration is a separate change, so this is purely additive alongside it.

Model hub rows are computed off the running router rather than read from a
table, so this adds InMemoryListExecutor: the same QueryPlan applied in
Python instead of rendered to SQL. It matches the SQL executors where it
counts, NULLS LAST in both sort directions and NULL satisfying no
comparison, so a filter means the same thing on either. The other three
public hubs have the same shape and can reuse it as is.

The fix itself is ordering. The endpoint being superseded reads every
latest health check and joins it against the whole model list, so paging
the response alone would have changed nothing. Here the health lookup is
an injected dependency the executor calls on the page slice, after the
filter and the sort, so it resolves health for the rows being served and
no others. PrismaClient gains a bounded read for that, next to the
unbounded one it mirrors. The regression test pins the ordering by
asserting which model groups the lookup is asked about, and fails against
an enrich-then-slice implementation.

* fix(proxy): address self-review findings on the public model hub list

Five adversarial review passes over the branch. What they found:

`is_null` was the one predicate in the in-memory executor that read a
repeated field's container instead of its elements, so a field holding only
nulls was indistinguishable from a populated one. It now lifts over elements
like every other predicate does. Not reachable through this endpoint, whose
only repeated field grants `contains` alone, but the executor is written to
be reused by the other three hubs and the inconsistency was a trap for them.

The fastapi removed-name guard globbed the framework packages but not
`public_endpoints/public_v1`, which `proxy_server` also imports unguarded at
module level, so the new package had none of the protection the test claims
to give. It now covers all three.

Regenerates the dashboard's API types, which the OpenAPI sync check requires
whenever the proxy's route surface moves. The diff is the 65 generated lines
for the new operation and nothing else; no dashboard code changes here.

Also trims comments and docstrings that argued for a decision or restated a
signature rather than explaining code, and wraps a docstring line that ran
past 120 characters.

* ci: run the relocated list framework tests in the proxy-endpoints shard

The framework's tests moved from tests/test_litellm/proxy/management_endpoints,
which the proxy-endpoints shard claims, into a new tests/test_litellm/proxy/list_api
that no shard named. Both coverage guards caught it: the semantic shards have no
catch-all bucket, so the directory would have run nowhere.

Claims it alongside management_endpoints, where the same tests ran before.

* docs(proxy): stop restating the list spec in the model hub route docstring

The docstring listed every sortable field, the page-size cap and the filter
set, all of which already live in MODEL_HUB_LIST_SPEC and all of which the
endpoint hands back in the allowed array of a rejected request. Two copies of
one spec is a prose update owed on every change to the real one.

Keeps what a caller cannot derive from the endpoint itself: what the resource
is, that it needs no authentication, and a working example. Regenerates the
dashboard types, which carry the docstring as the operation description.

* fix(proxy): reject a repeated sort field instead of sorting by it twice

sort took any number of comma-separated keys, and the in-memory executor runs
one full sorted() pass per key before slicing. Naming one allowed field N times
therefore bought N passes over every published model group, synchronously on the
event loop, from a route that needs no credentials. Measured on 300 groups:
0.001s for one key, 0.034s for a thousand, 0.166s for five thousand, and it
grows with the catalogue this endpoint exists to make large.

A repeated field cannot change the ordering, so rejecting repeats costs a caller
nothing and bounds the passes at len(sortable), a number the spec author picks
rather than the caller. That beats an arbitrary cap: no magic number, and the
bound holds for every resource built on the framework.

The tiebreaker is appended after parsing, so sorting explicitly by it stays legal.
Budgets renders one ORDER BY in SQL and never had the amplification, but the
check belongs with the rest of the sort validation rather than in one executor.

* fix(proxy): make the search disjunction one level deep by type

Two CI gates, one cause. AnyOf declared its clauses as Predicate, so both
consumers had to recurse to evaluate one: the SQL renderer through
_render/_render_all, and the in-memory executor through _holds. The recursion
detector flags the latter, and its reason is the same one this PR already ran
into once, a caller-controlled cost that shows up as CPU.

Nothing actually builds a nested AnyOf. _search_predicate is its only producer
anywhere in the repo and it emits Compare leaves, in every call site and every
test. Declaring clauses as tuple[Compare, ...] makes that a fact the type
checker keeps rather than a comment, and _holds then evaluates a disjunction of
leaves with no recursion at all.

Also marks the new health read's broad except, which the strict gate counts,
and covers the ordering comparison operators. The endpoint exposes only
eq/in/contains, so gt/gte/lt/lte were live code no test evaluated.

* fix(proxy): keep the new health read inside the type-discipline ceiling

The bounded health query added ten LIT002 violations, which pushed the
codebase total past its budget. The gate counts across the tree and compares
to the merge base, so a file already carrying debt does not absorb new
violations.

Returns an empty tuple rather than an empty list on the two no-result paths:
the signature already promises a Sequence, so that is a free two-violation
reduction and a better type. Builds prisma's order argument from a tuple of
pairs, which turns four literals into one. The three that remain are prisma's
own API shape and each carries its reason.

Both budget gates now pass against the merge base.

* fix(proxy): clear the two basedpyright errors the new route added

The type-check budget is over its ceiling on the base already, so the gate
blames any increase: reportArgumentType 2574/2564 and reportPrivateUsage
1815/1808, one each, both from this file.

fastapi types a route's tags as list[str | Enum], so the tuple was an argument
error; budgets.py has the same one and it is part of what put the rule over.
Passing a list is what the signature asks for, marked because an inline list
is a construction the discipline gate counts.

_get_model_group_info is private by name but is the shared reader the endpoint
this supersedes imports the same way, so the import carries a rule-scoped
ignore with that reason rather than a copy of the function.

basedpyright now reports zero errors across both new modules, and all three
budget gates pass against the merge base.
2026-08-28 10:02:59 -07:00
yuneng-jiang
dec3bca26e
Merge pull request #38640 from BerriAI/litellm_/e2e-skipped-tests-triage-2048ac
test(e2e): unskip four tests whose blockers no longer hold
2026-08-28 10:02:30 -07:00
yuneng-jiang
a4a0386717
Merge pull request #38637 from BerriAI/litellm_/red-tests-review-fbb718
test: refresh the suites that drifted from langfuse and OpenAI's retired Assistants API
2026-08-28 09:58:44 -07:00
mateo-berri
b8f91235a6 fix(cost): only let a WxH request size drive image pricing 2026-08-28 09:52:31 -07:00
Mateo Wang
ecf84a2c2a
Merge pull request #38657 from BerriAI/litellm_count_tokens_fallback_tools_system
fix(proxy): count tools, system, and Anthropic image and document blocks in the count_tokens fallback (internal copy of #36671)
2026-08-28 09:52:14 -07:00
mateo-berri
bf047148fc fix(cost): price image generations from the requested quality when the response omits it 2026-08-28 09:38:21 -07:00
Yuneng Jiang
2b0932f930
test(access-groups): give each xdist worker its own fixture ids
Every test in the file seeds, reads and deletes the same fixed group and team
ids, and auth_ui_unit_tests runs pytest with -n 2. Two tests landing on the two
workers at once tread on each other: one worker's _clean_db DELETE wipes rows
the other just seeded, and its sync writes land in the other's read.

Both shapes showed up on 13a0976bb6, a commit that renames a passthrough test
and nothing else. test_reconcile_is_idempotent... read back an empty table, and
test_reconcile_handles_a_null_array_column read the idempotent test's team on
its own second group.

Scoping the ids to PYTEST_XDIST_WORKER keeps each worker in its own rows. Tests
on one worker still run in sequence, so no isolation is lost.

Reproduced against a local Postgres: -n 2 failed 6 out of 6 runs before, passed
6 out of 6 after, and serial runs are green either way. Stripping the COALESCE
guard from the mirror's SQL still fails the suite, so the ids are all that
changed.
2026-08-28 09:35:12 -07:00
mateo-berri
b05ac5fefd test(registry): allow /v1beta/interactions in the supported_endpoints schema 2026-08-28 09:29:09 -07:00
Yuneng Jiang
0ec619e7d1
test: close mutation-testing gaps in container, skills and openai-like config factories
Mutation testing surfaced three factory functions whose tests ran against them
but asserted nothing that a mutation could break, so every planted bug survived.

- litellm/llms/litellm_proxy/skills/code_execution.py: the OpenAI and Anthropic
  tool schemas were unpinned (the Anthropic one was not reached by any test at
  all) and the handler's default fallbacks were unchecked
- litellm/containers/endpoint_factory.py: the endpoints.json contract, the
  generated sync/async function set and the response-type mapping were unpinned
- litellm/llms/openai_like/dynamic_config.py: the generated Responses API config
  class had no coverage of auth header, URL resolution or the store override

The openai_like tests clear _responses_config_cache around each test. Without
that, the module-level cache hands back a class built before the mutation and
the tests pass against mutated code.

Verified by re-running mutmut per scope:
  llms/litellm_proxy  45.2% -> 62.8%  (70 mutants newly killed)
  containers          36.8% -> 84.3%  (45 mutants newly killed)
  llms/openai_like    55.7% -> 66.9%  (34 mutants newly killed)
2026-08-28 09:28:14 -07:00
Yuneng Jiang
13a0976bb6
test(passthrough): name the test for what it now covers
The target is a local server, so the OpenAI host check already excludes it
and the docstring's claim about the route path no longer holds.
2026-08-28 09:11:48 -07:00
Yuneng Jiang
021e03fe90
test(passthrough): serve the passthrough target locally instead of calling OpenAI
Greptile flagged this test as coupled to OpenAI's availability. The coupling was
not the status assertion it pointed at, and it predates this PR: the test it
replaced called /v1/assistants live the same way, and pass_through_endpoints gates
success logging on `response.status_code < 400`, so an upstream outage has always
meant no log fires and the payload assertions fail regardless.

The target is now a local HTTP server on an ephemeral port, so the test is offline
either way. It still exercises the generic passthrough handler, since
_is_supported_openai_endpoint does not claim a 127.0.0.1 URL any more than it
claimed /v1/moderations, and it now also asserts what the upstream actually
received rather than only what came back.

respx was the obvious approach and does not work here: it patches httpx transports,
and the passthrough issues its request through the custom aiohttp transport, so the
call went to the real api.openai.com and returned 401 while respx sat unused.

Mutation checked: gating off the success enqueue fails the test, and tampering with
the logged response body fails it.
2026-08-28 09:03:57 -07:00
yassin
418b820af4 fix(proxy): let llm_api virtual keys read /model_group/info
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-28 14:42:22 +00:00
mateo-berri
24d226c6c2 chore(token_counter): drop docstrings and test prose that restated the count_tokens branches 2026-08-28 06:28:21 -07:00
Andrew Mattie
f5fbde9151 fix(streaming): align assembled provider model 2026-08-28 08:13:14 -05:00
mateo-berri
83ab87091b fix(proxy): only attach tools to the count_tokens fallback when counting messages 2026-08-28 06:10:30 -07:00
mateo-berri
70ba0bb973 fix(proxy): count tools, system, and Anthropic document blocks in the count_tokens fallback 2026-08-28 05:41:38 -07:00
Fazeel Usmani
7cd3a27d60 update test signature for Anthropic image block handling 2026-08-28 05:35:06 -07:00
Fazeel Usmani
fe28781dfd fix(token-counter): enhance handling of Anthropic image blocks in token counting 2026-08-28 05:35:05 -07:00
Fazeel Usmani
1cce589aa0 fix(token-counter): count Anthropic native image content blocks
`_count_content_list` accepted text, image_url, tool_use, tool_result,
thinking and tool_reference, and raised on anything else, so an
Anthropic-native `{"type": "image", "source": {...}}` block aborted the
whole count. That is the documented Anthropic image format and exactly
what /v1/messages receives.

Three user-visible effects. /v1/messages/count_tokens and
/utils/token_counter return 500, and the router's context-window
pre-call check swallows the ValueError and returns every deployment
unfiltered, so an oversized prompt carrying an image is dispatched to
the provider instead of being rejected locally with a 400.

Prices the block through the existing image path: a base64 source
becomes a data URI, a url source passes through, and a file source
falls back to the default image token count. Blocks nested inside
tool_result.content are covered too, because _count_anthropic_content
recurses back into _count_content_list.

Fixes #36604
2026-08-28 05:35:05 -07:00
Devin AI
cb70941660 fix(tests): drain the global logging worker in RAG aquery billing tests instead of polling
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-28 09:58:22 +00:00
Yuneng Jiang
e8b9f3675b
test(batch): make the upstream-failure tolerance actually reachable
batch_completion collects per-request failures into its result list rather than
raising them; its own source says "return exceptions if any". So the test's
`except Timeout` and `except litellm.InternalServerError` arms could never fire for
the case they were written for. An upstream 500 instead reached
`response.choices`, raised AttributeError on the exception object, and fell through
to the bare `except Exception` that calls pytest.fail. That is what CircleCI hit.

The tolerance now reads the returned values, which is where the failures actually
are. The same two exception types are tolerated as before, nothing broader.

Checked against four injected outcomes: three InternalServerErrors pass, three
Timeouts pass, an AuthenticationError fails, and a response whose content is None
fails. So it is not tolerating its way to a vacuous green.
2026-08-28 00:25:53 -07:00
tin-berri
ca0b951a43
feat(spend): report prompt caching savings as total and gateway-attributed (#38134)
* feat(spend): report prompt caching savings as total and gateway-attributed

`prompt_caching_savings_spend` credited every cached request, including caching a
client asked for with its own `cache_control` and caching a provider does implicitly,
so the number overstated what the gateway had any hand in.

Gating that column in place would have fixed the overstatement by changing what the
column means, leaving rows written before the change saying "all caching savings" and
rows after saying "gateway-injected only" with nothing to tell them apart, and forcing
a decision about rewriting history. It also breaks the cache-leakage estimate on the
dashboard, whose numerator would be gated while its denominator, the cached token
counts, would not, so the rate it extrapolates from would be quietly diluted.

Report both instead. `prompt_caching_savings_spend` keeps meaning every net dollar
caching saved, which is what a customer means by "what did caching save me", and the
new `gateway_injected_caching_savings_spend` carries the subset litellm caused by
injecting the breakpoints itself. Both are derived from the same marker, so this
changes what is done with it rather than how it is obtained.

The attributed figure is normally the smaller of the two, being a subset of the same
requests, but not always: a request that writes cache it never reads has negative net
savings, and excluding such a request can lift the attributed figure above the total.

Also stops the marker riding into a fallback leg. The fallback rebuild spread the
failed attempt's metadata forward, so a deployment that injected nothing inherited the
marker and was credited anyway, which silently restored the very overstatement this
separates out.

* fix(bedrock): credit gateway caching where the tool cachePoint is placed (#38478)

The savings marker records breakpoints litellm placed, and a tool_config
injection point becomes one only in the converse transform, and only when the
request carries tools. The prompt hook cannot see either condition, so marking
on the point's presence credited request shapes that cached nothing, while
Bedrock tool caching the gateway did cause went uncredited.

Record it at the placement site instead. The marker's reader also resolves its
bucket by value now: litellm_params declares litellm_metadata as None on every
request, so asking the shared name resolver named a bucket that was not there
and the mark was dropped.
2026-08-28 00:19:06 -07:00
Yuneng Jiang
d8679508d4
test(e2e): measure the select popup after it settles instead of mid-flight
Both anchoring tests read the trigger's box before the click and the popup's box the
instant it turns visible. Base UI places the popup asynchronously and opening it can
shift the trigger, so both boxes could be sampled before the layout settled. The run
on 1eedaa3a43 missed by 4.2px (expected >= 446.015, got 441.799) on a tree with no UI
changes at all, having passed on 21092d633b, which differs only in a deleted python
test and a budget json.

Each assertion now re-reads both boxes under expect.poll. The conditions themselves
are unchanged: the popup must sit at or below the trigger's bottom edge in the first
test and must not overlap it in the second. Polling cannot mask a genuinely misplaced
popup, since one that never lands correctly still fails when the poll times out.
2026-08-28 00:15:02 -07:00
Yuneng Jiang
0c5c96dbf6
test(e2e): unskip four tests whose blockers no longer hold
/v1/batches now rejects a missing input_file_id with a 400 through
raise_if_required_body_param_missing, so the contract negative that was
skipped for "500s instead of 400" passes as written. Verified against a
live proxy.

The three Datadog MCP tests were skipped because each one sent a
`telemetry` argument that search_datadog_logs rejects with "unexpected
additional properties". That argument was never a documented Datadog
parameter and no assertion reads it, so it is dropped and the tests run
again unchanged otherwise.
2026-08-28 00:01:51 -07:00
Yuneng Jiang
1eedaa3a43
test(passthrough): drop the Azure assistants test, retired upstream on 2026-08-26
Azure now answers the create call with 410 and code assistants_api_deprecated:
"The Assistants API has been retired. Follow the migration guide to update your
workloads." Microsoft retired it on the same day OpenAI retired theirs, which is
why this landed with the OpenAI ones rather than before them.

This job runs pytest with -x, so the test was also hiding everything after it in
tests/pass_through_tests.

test_pass_through_file_operations stays. It only asks /v1/files for
purpose="assistants", which still answers 200 on both upload and delete.
2026-08-27 23:58:23 -07:00
Yuneng Jiang
21092d633b
test(opik): stop the batching test racing its own 5-second flush timer
test_opik_logging_http_request asserted "nothing has been POSTed yet" roughly one
second into a window governed by OpikLogger's 5-second periodic flush. On a loaded
CI worker the five preceding acompletion calls eat that budget, the periodic flush
fires, and the assertion flips. Reproduced with no product changes at all: letting
5.5 seconds pass before the assertion drains the queue and sets mock_post.called,
which is exactly the failure CircleCI reports.

The test now pins flush_interval past anything the test can reach, so the two
batching assertions measure batching instead of wall clock, and drives the flush
path explicitly at the end rather than sleeping the interval. That last phase used
to be near-vacuous, since the size-triggered flush had already emptied the queue.

Assertions now match only calls to Opik's own /traces/batch and /spans/batch.
get_async_httpx_client caches one client per special provider, so the mock is
process-wide and any other logging callback's POST would otherwise count.

Dropped the teardown that closed that shared client, which broke every later test
in the same worker that logs through it, and the try/except that turned assertion
failures into a pytest.fail with no traceback.

Mutation checked: flushing on every event and never flushing on size both fail the
test.
2026-08-27 23:43:38 -07:00
Yuneng Jiang
a47c2f76ab
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/red-tests-review-fbb718 2026-08-27 23:37:37 -07:00
Yuneng Jiang
feffb62266
test: refresh the suites that drifted from langfuse and OpenAI's retired Assistants API
Two unrelated causes, both leaving staging red with tests that no longer describe
anything true.

#38264 gave LangFuseLogger a langfuse_environment argument and started carrying it
in the credentials dict. The handler test's fake logger did not accept the new
keyword, so constructing it raised TypeError, and four cases in
test_langfuse_unit_tests rebuilt the cache key by hand from three fields and missed
on the four-field key production now writes. Caching itself was never broken: the
handler sets and gets with the same dict. The fake now takes the argument and
asserts it is forwarded, and the cache assertion issues a second identical request
and expects the same logger back, which is the behaviour that matters and cannot
rot the next time a credential field is added.

OpenAI has retired the Assistants API. /v1/assistants and /v1/threads both answer
404 with a valid key, where every live route answers 401, so nothing calling them
can pass again. test_custom_logger_passthrough covered generic passthrough logging
and only used assistants because it is a route with no provider-specific handler;
it moves to /v1/moderations, which is still unclaimed by
_is_supported_openai_endpoint, so the same generic branch is exercised. The two
tests there asserted the same thing against different dead routes, so they collapse
into one. The Ruby suite existed solely to drive assistants, threads, messages and
runs, so it goes along with the RVM and bundler steps that were installed only to
run it, and the two dead OpenAI assistants cases leave
test_openai_assistants_passthrough.

The Azure assistants case in that file stays. Azure runs its own lifecycle and I
could not reach the CI deployment to check whether that API is still there.
2026-08-27 23:24:36 -07:00
Yuneng Jiang
74bc1efe65
Merge branch 'litellm_internal_staging' into litellm_fix_staging_ci_regressions
Resolve tests/llm_translation/test_together_ai.py in favor of staging:
bcb6a0a998 already landed the fail-open assertion for models missing from
the registry, so both models now list response_format and tools. This
branch's narrower gating of response_format no longer matches behavior.
2026-08-27 23:12:42 -07:00
tin-berri
02c1c45b53
fix(complexity_router): route client housekeeping calls to the cheapest tier (#38598)
A coding agent names each conversation by quoting the whole session and asking
for a title. The classifier rated the quoted session rather than the request, so
the cheapest call the client makes routed to the most expensive tier: 11 of 17
title generations in one day of real traffic came back COMPLEX.

Recognize those prompts by literal sentinel on the newest ask and route them to
the cheapest configured tier without classifying them, so the call costs nothing
to route. The placement is scoped to the one request that carries the sentinel:
it never displaces an operator's classifier plugin, the bandit cannot reach above
the tier as raised, it never becomes the session pin, and the sentinel that
matched is recorded on the routing decision. Detection reads the newest ask
alone, so a title request quoted into a later turn cannot cheapen the work that
follows it, and a keyword rule, an escalation keyword or the plan-mode floor all
still decide over it.

Regenerates the lazy OpenAPI snapshot, which was already stale on the base for an
unrelated Presidio guardrail field and failed the schema check on every PR.

Resolves LIT-6349
2026-08-27 22:55:09 -07:00
tin-berri
b3322e30a7
fix(anthropic): handle per-level reasoning_effort flags without supports_reasoning (#38618)
Some checks failed
Unit Tests / caching-local (push) Has been cancelled
Unit Tests / core-utils (push) Has been cancelled
Unit Tests / enterprise-package (push) Has been cancelled
Unit Tests / enterprise-routing (push) Has been cancelled
Unit Tests / integrations (push) Has been cancelled
Unit Tests / All Other Providers (push) Has been cancelled
Unit Tests / Vertex AI (push) Has been cancelled
Unit Tests / misc (push) Has been cancelled
Unit Tests / proxy-auth (push) Has been cancelled
Unit Tests / proxy-endpoints (push) Has been cancelled
Unit Tests / proxy-extras (push) Has been cancelled
Unit Tests / proxy-infra (push) Has been cancelled
Unit Tests / proxy-server (push) Has been cancelled
Unit Tests / responses-caching-types (push) Has been cancelled
GitHub Actions Security Analysis / zizmor (push) Has been cancelled
Postgres Tests / schema-migration (push) Has been cancelled
Postgres Tests / proxy-behavior (push) Has been cancelled
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Has been cancelled
Unit Tests: Proxy DB Operations / auth-checks (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-utils (push) Has been cancelled
Unit Tests: Proxy DB Operations / budgets (push) Has been cancelled
Unit Tests: Proxy DB Operations / custom-logging (push) Has been cancelled
Unit Tests: Proxy DB Operations / db-and-spend (push) Has been cancelled
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Has been cancelled
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Has been cancelled
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Has been cancelled
Unit Tests: Proxy DB Operations / key-generation (push) Has been cancelled
Unit Tests: Proxy DB Operations / logging-misc (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-runtime (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-server-core (push) Has been cancelled
* fix(anthropic): handle per-level reasoning_effort flags without supports_reasoning

When a model has only per-level flags (e.g. supports_minimal_reasoning_effort: true)
but no explicit supports_reasoning flag, treat it as implicitly reasoning-capable.
This fixes gpt-5-search-api which declares minimal support but was incorrectly
degraded to low/minimal floor due to missing explicit supports_reasoning flag.

Test: verify per-level flag enables resolution path even without supports_reasoning.

Note: This change indirectly causes 20 azure deployments to forward max/xhigh
instead of degrading to high when requested, as these models now correctly
resolve their supported efforts through declared capability flags. This is
intended behavior (avoiding unnecessary degradation) but silent; operators
seeing increased latency/cost should check reasoning effort changes in logs.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(anthropic): explicit supports_reasoning=False wins over per-level flags

Greptile P1: the implicit-True branch bypassed the operator's explicit
supports_reasoning: false escape hatch when per-level flags were present
or inherited through the bare-twin lookup. Return () first on explicit
False, then apply the per-level implication only when the flag is unset.

Also drops a test comment that restated the test name (P2).

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-27 22:19:22 -07:00
tin-berri
39e5b0c2d1
fix(anthropic): drop and self-heal empty thinking blocks on /v1/messages (#38625)
* fix(anthropic): drop and self-heal empty thinking blocks on /v1/messages

* test(anthropic): pin early-signature carry across the blank thinking chunk skip
2026-08-27 21:41:56 -07:00