mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-09 03:18:44 +00:00
5 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
tin-berri
|
58de920921
|
feat(mcp): bound outbound tool-call concurrency per MCP server (#31641)
Add an optional per-server max_concurrent_requests that caps how many tool calls LiteLLM sends to one MCP server at once, so batch-processing backends are not overwhelmed by unbounded parallel dispatch. Excess calls queue on a per-server asyncio.Semaphore instead of being rejected. Unset or non-positive means unlimited, preserving existing behavior. Resolves LIT-2749 |
||
|
Krrish Dholakia
|
cca71a07c2
|
feat(mcp): add mcp_tool_search virtual tools for large tool catalogs (#31777)
* feat(mcp): add tool search virtual tools for large catalogs
When mcp_tool_search_enabled is set on a key's object_permission,
tools/list returns only mcp_tool_search and mcp_tool_call instead of
the full catalog. The LLM searches by keyword then calls discovered
tools by name, avoiding context bloat with 100+ tool deployments.
* fix(mcp): persist mcp_tool_search_enabled and route tool_call by name
The mcp_tool_search_enabled flag existed on the Pydantic models but the
Prisma schema lacked the column, so keys generated with the flag never
persisted it and tools/list kept returning the full catalog. Add the
column across all three schema.prisma copies plus a migration.
handle_mcp_tool_call passed server_name="" into call_tool, which built a
malformed prefixed name ("-<tool>") and failed to resolve the server.
Resolve the caller's allowed servers and dispatch through execute_mcp_tool
instead, matching how the normal /tools/call path routes.
* fix(mcp): filter list_tools to virtual tools on the protocol path
The REST surface (/mcp-rest/tools/list) returned only the two virtual
tools when mcp_tool_search_enabled was set, but the MCP protocol handler
(handle_list_tools, used by real MCP clients over streamable-http/SSE)
still returned the full catalog. Apply the same early return there so an
actual MCP client sees mcp_tool_search and mcp_tool_call instead of every
tool. call_tool was already intercepted on this path.
* fix(mcp): enforce IP + server filtering on virtual tool search/call
Review flagged that the virtual mcp_tool_search/mcp_tool_call path skipped
access controls the normal MCP flow applies. mcp_tool_call resolved allowed
servers from key permissions only, never applying IP filtering, so a caller
on a public IP could invoke a tool on a server marked
available_on_public_internet: false. mcp_tool_search listed the raw catalog
via global_mcp_server_manager.list_tools, exposing tool names/schemas that
/tools/list would hide and ignoring per-key/per-server tool filters.
Route both virtual handlers through the same filtered paths used by the
normal MCP flow: search now calls _list_mcp_tools and call resolves servers
via _get_allowed_mcp_servers, both threaded with the request client IP so
filter_server_ids_by_ip applies. execute_mcp_tool then enforces the server
allowlist and per-key tool permissions. Thread client_ip through
_list_mcp_tools/_get_tools_from_mcp_servers and pass it from the REST and
SSE call sites.
* fix(ci): ruff format server.py and sync dashboard API types
ruff format normalizes the list_tools client_ip changes in server.py, and
schema.d.ts gains the mcp_tool_search_enabled object-permission field so the
generated dashboard types match the proxy OpenAPI spec.
* style(mcp): drop quoted annotations and sort imports
Clears UP037 on the virtual tool handler signatures (redundant with
from __future__ import annotations) and I001 on the list_tools import block.
* refactor(mcp): extract virtual-tool dispatch and host progress capture
Pulls the mcp_tool_search/mcp_tool_call interception and the host
progress-callback setup out of mcp_server_tool_call into helpers, keeping
that handler under the strict cyclomatic-complexity ceiling after the
client_ip threading. No behavior change.
* test(mcp): cover SSE virtual-tool dispatch and host progress helpers
Adds unit tests for _dispatch_virtual_mcp_tool (non-virtual passthrough,
flag-disabled rejection, search/call routing with client_ip),
_capture_host_progress_callback, and the protocol list_tools virtual
early-return, covering the new server.py paths.
* fix(mcp): forward per-request auth headers through virtual tool handlers
The virtual mcp_tool_search/mcp_tool_call path intercepted the request
before the normal header extraction ran, so client-supplied per-request
auth (Authorization for upstream pass-through, x-mcp-auth-<alias>) was
dropped and execute_mcp_tool/_list_mcp_tools received None. Thread
mcp_auth_header, mcp_server_auth_headers, oauth2_headers, and raw_headers
from both the REST and SSE call sites through the handlers so upstream MCP
servers that require pass-through auth can be listed and called.
* fix(mcp): preserve requested server scope in virtual tool calls
A scoped MCP session (/mcp/<server>/ or header-scoped) carries an
mcp_servers scope that the normal call path passes into routing so the
session can only reach that server. The virtual-tool branch dropped it and
resolved with mcp_servers=None, letting a scoped session call mcp_tool_call
for any server the key can access. Thread the context mcp_servers scope
through _dispatch_virtual_mcp_tool into both handlers so search and call
resolve against the same scoped server set.
* fix(mcp): convert virtual tool errors to isError on the protocol path
The virtual-tool dispatch ran before the protocol handler's HTTPException
and guardrail handling, so a rejected virtual call (e.g. an out-of-scope
403 from execute_mcp_tool) raised out of mcp_server_tool_call and broke the
MCP JSON-RPC stream instead of returning an isError CallToolResult. Move
the dispatch inside the same try that wraps call_mcp_tool so virtual-tool
errors get the same isError conversion as normal tool calls.
* fix(mcp): spend-log virtual tool calls on the REST path
The REST virtual-tool branch returned before common_processing_pre_call_logic,
so execute_mcp_tool ran without a litellm_logging_obj and virtual mcp_tool_call
invocations were not spend-logged or guardrail-checked like normal calls. Run
the same pre-call pipeline in the call branch and thread the resulting
litellm_logging_obj through handle_mcp_tool_call into execute_mcp_tool.
* fix(mcp): reject virtual tool call when key has no accessible servers
handle_mcp_tool_call passed an empty allowed_mcp_servers list into
execute_mcp_tool; an unprefixed local tool name then fell through to the
local registry, which has no server permission check, so a key with only
mcp_tool_search_enabled and no server grants could run operator-configured
local tools by name. Reject with 403 before dispatch when no servers are
accessible, matching call_mcp_tool.
* docs(mcp): document virtual tool_search module and parity rule in AGENTS.md
* style(mcp): apply ruff format at repo line-length (120)
* fix(mcp): add mcp_tool_search_enabled to ObjectPermissionDict and customer test fixture
* chore: trigger CI
* fix(mcp): mirror pre-call pipeline, guard imports, coerce top_k, honor include_disabled_tools
- SSE mcp_tool_call now runs common_processing_pre_call_logic so it spend-logs and runs guardrails like the REST path (P1)
- coerce_top_k avoids ValueError on non-integer top_k from clients (both REST and SSE)
- guard mcp.types import in tool_search behind runtime/TYPE_CHECKING per package convention
- admin list with include_disabled_tools returns the real catalog even when mcp_tool_search_enabled is set
|
||
|
ryan-crabbe-berri
|
3dce3daff6
|
feat(proxy): type Customer Management response_model for OpenAPI coverage (#31043)
* feat(proxy): type Customer Management response_model for OpenAPI coverage Add response_model to the five remaining untyped /customer operations (block, unblock, new, update, delete) so the generated OpenAPI schema documents a concrete response body. new/update reuse the canonical LiteLLM_EndUserTable (matching info/list); block, unblock, and delete get small dedicated models in litellm/types/proxy/management_endpoints/customer_endpoints.py. Together with the already-typed info/list/daily-activity routes this brings the Customer Management group to full response_model coverage. Regression tests assert each public /customer/* route declares the expected response_model and that /customer/new surfaces a typed schema in app.openapi(), so dropping a response_model fails CI. * fix(proxy): keep budget_id in typed customer responses Address review feedback on the Customer Management response_model typing. Greptile flagged that response_model=LiteLLM_EndUserTable on /customer/new and /customer/update silently drops fields the raw Prisma model_dump() echoed. Checking the schema, budget_id is the only such scalar column that was missing from the Pydantic model (created_at/updated_at/tpm_limit do not exist on litellm_endusertable), so add budget_id to LiteLLM_EndUserTable. This restores budget_id on new/update and also fixes the pre-existing gap where /customer/info and /customer/list (already typed) dropped it, which the UI Customer type expects. A regression test pins budget_id surviving the response_model filter on /customer/update. Also document UnblockUsersResponse.blocked_users via a Field description: it holds the users that remain blocked after the call. The key name predates this PR and is kept to avoid a backwards-incompatible rename on a beta route. * fix(proxy): keep nested budget fields in customer responses response_model=LiteLLM_EndUserTable nests the budget as the narrow write allowlist LiteLLM_BudgetTable, which silently drops the server-managed fields the customer endpoints used to return (budget_reset_at, created_at). Introduce CustomerResponse, a thin response model that nests LiteLLM_BudgetTableFull (the repo's budget response model), and apply it on /customer/new, /customer/update, /customer/info and /customer/list. list also builds CustomerResponse so its budget isn't narrowed at construction time. created_by/updated_at/updated_by remain omitted, matching how budgets are returned elsewhere. The shared LiteLLM_EndUserTable is left untouched: it's constructed in many places that pass narrow budget instances, and pydantic v2 won't coerce a budget instance into a wider nested model. Typing only at the response boundary (where the handler hands FastAPI a dict) sidesteps that. A regression test pins budget_reset_at + created_at through the filter and asserts the internal audit fields stay out. * test(proxy): add golden-master characterization tests for customer responses Lock the exact JSON body each customer-object endpoint (info/list/new/update) and delete return today, so the upcoming type-safety refactor of the handlers is only allowed to land if it reproduces these byte for byte. Pins null-field inclusion, the nested budget shape (server fields kept, audit fields dropped), and object_permission reverse-relation stripping. Green against current code. * refactor(proxy): make the customer response flow type-safe Replace the untyped dict + bolt-on response_model pattern on the customer object endpoints with explicit typed construction. A single mapper, _to_customer_response, validates a DB row into CustomerResponse at one Any -> typed seam; new/update/info/list now return it (or a list of it) and carry real -> CustomerResponse / -> List[CustomerResponse] return annotations, and delete returns DeleteCustomersResponse. basedpyright now verifies the handlers' return shapes instead of a runtime filter doing it silently. This also deletes the four copy-pasted object_permission reverse-relation cleanup loops: pydantic's extra=ignore drops those undeclared fields during validation, so the loops were dead code (proven by the golden-master tests, which stay byte-for-byte green). basedpyright errors on the file drop from 140 to 116, all from removed dict plumbing. CustomerResponse stays a thin subclass of LiteLLM_EndUserTable so it inherits the existing validators/config unchanged (behavior preservation); only the nested budget type is widened. * refactor(proxy): annotate customer response mapper param as BaseModel Address review nit: the mapper's untyped `record` added an ANN001 violation. The incoming rows are pydantic v2 models, so type the param as BaseModel rather than object (object has no model_dump, which would just move the problem to basedpyright). This clears the ANN001 and also drops three basedpyright unknown-type violations the untyped param was adding. * style(test): ruff format customer endpoint tests * test(proxy): give customer budget test update mocks a valid model_dump The type-safe response refactor validates the update result via _to_customer_response (CustomerResponse.model_validate(record.model_dump())). These budget tests mocked the end-user update to return a bare MagicMock, so model_dump() yielded a MagicMock that fails validation. Give each update mock a minimal valid dict; the tests assert on the prisma calls, not the body. * chore(ui): regenerate API types from proxy OpenAPI spec * fix(ui): make generated API types stable across Python versions Python 3.13 strips a docstring's common leading indentation at compile time while 3.12 keeps it, so app.openapi() emits differently-indented description strings depending on the interpreter. The dashboard type generator ran locally on 3.13 and in CI on 3.12, so schema.d.ts drifted and the "Verify schema.d.ts matches the proxy OpenAPI spec" check failed Normalize every description through inspect.cleandoc in the spec dump so the output is identical regardless of interpreter, then regenerate |
||
|
Mateo Wang
|
48b5a5a0cc
|
style: unify ruff format width on 120 (#31518)
The repo linted at 120 (E501, isort) but ran ruff format at 88 via a --line-length 88 override in the Makefile and CI, leaving the formatter and the linter disagreeing on wrap width. Drop the override so ruff.toml's line-length = 120 is the single source of truth and reformat the tree to match. |
||
|
Yassin Kortam
|
5e2db7eee4
|
feat(litellm): add models and repository layers (#29686) |