26 commits

Author	SHA1	Message	Date
quietjules	8801a17f0d	fix(proxy)!: bind admin-owned team keys to their team's MCP servers (#49161) ... * test(proxy): pin admin-owned team keys to their team's MCP servers and toolsets A virtual key on a real team owned by a proxy_admin or proxy_admin_viewer, with no MCP list of its own, currently reaches every MCP server in the registry and any toolset, ignoring its team. These tests expect it to resolve only what its team grants, like the same key owned by an internal user, while the admin dashboard session keeps the whole registry. * fix(proxy)!: bind admin-owned team keys to their team's MCP servers A virtual key on a real team now resolves MCP servers and toolsets through its team even when its owner is a proxy_admin or proxy_admin_viewer, the way team model access already binds it. The unscoped admin view stays for the master key, admin keys with no team, the admin dashboard session, and admitted user subjects. The toolset grant checks on /toolset/{name}/mcp and on the Responses litellm_proxy MCP tool use the same rule. BREAKING CHANGE: a team key owned by a proxy admin or admin viewer no longer reaches MCP servers or toolsets its team does not grant. Grant them to the team or use a key with no team. * fix(proxy): answer 403 for a toolset the key cannot use on LLM routes A litellm_proxy MCP tool on /v1/responses, /v1/chat/completions or /v1/messages that named a toolset the key's team was never granted was skipped, which left the server filter empty and handed the model every tool the key could reach. The denied name now stays in the filter, so the request gets the same 403 an unknown server name gets. Admin credentials on a real team stay bound to it whatever carries them, a lite login session included, the way the team already bounds their model access. The tests pin that and drop an unused patch. * fix(proxy): keep a denied toolset out of the scoped key beside a granted one * test(responses): cover toolset name fallbacks and align MCP admin docstrings * test(mcp): cover no-team admins' toolset access on LLM routes and drop a dead admin-check clause * fix(responses): keep a whole-gateway MCP entry serving the key's tools beside a denied toolset * test(responses): check the tools a whole-gateway request lists beside a denied toolset * test(responses): keep the toolset lookup real in the whole-gateway listing test * test(responses): run the toolset gateway tests on a real MCP server manager * fix(responses): keep denied toolset names out of the MCP server filter A denied toolset named like a server the key holds was matched to that server by the loose server-name lookup. Denied names now never reach the filter, and a request naming only denied toolsets answers 403 directly, each name once * fix(responses): let a held MCP access group outrank a same-named ungranted toolset and give denied toolsets the generic 403 * test(responses): pin access group and granted toolset precedence for LLM-route MCP names --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 22:07:32 -07:00
simonp9	160a0b72cf	fix(mcp): bill both calls of a streamed chat completions MCP auto-execute (#48899) ... * fix(mcp): bill both rounds of a streamed chat completions MCP auto-execute A streamed /v1/chat/completions request with an MCP tool set to require_approval "never" runs two model calls on one logging object, and the final-stream dedupe kept only the first round's spend row. The final usage chunk also showed only the second round. The MCP stream iterator now holds the first round's assembled response, prices it, and logs one success row that carries the usage and cost of both rounds, with the fixed margin charged once. The final usage chunk shows the same summed usage. A single-round stream still logs its one row when the answer finishes. When the follow-up round fails, the first round is logged from a copy of the logging object taken before the follow-up opened, and the failure row carries only the follow-up's partial usage with no second fixed margin. With proxy post-call guardrails armed, only (assembled, cache_hit) tuples are parked for the proxy, and the deferred log is armed only while the iterator reads, so the proxy's orphaned-callback cleanup never sees it after the Router's first-chunk read * fix(mcp): send the usage chunk of a streamed chat MCP request that ends in one round When a streamed chat completions MCP request ends in its first round (the model answers without a tool call, or no tool results come back), the iterator drained the inner stream to fire its spend log and dropped the usage chunk that stream produced, so a client that asked for include_usage got no usage chunk at all. The drained usage chunk is now held and sent to the client after the final content chunk and before [DONE], the same place a plain stream sends it, so the last usage chunk matches the billed row. A two-round request still drops the first round's usage chunk and sends the merged one from the follow-up, and a stream without include_usage still sends none * fix(mcp): hold round one's log for the guardrail, bill the chunk cost once, and keep round one's id When the follow-up call of a streamed chat MCP auto-execute fails, round one's success log now goes through the proxy's deferred guardrail release instead of straight to the handlers, so a post-call guardrail runs on it and the row carries it. Without a proxy release it logs directly as before The last usage chunk now takes its cost from the combined row's billed total, so a fixed cost margin is counted once in the chunk, as it is on the row. The follow-up's usage chunk is held until the follow-up ends to make that possible The combined row now keeps round one's id, the stream's first chunk id, which matches how a single-round stream is logged * fix(mcp): bill every chat MCP stream round once on failure, hang-up, and sync drop When the follow-up call fails, round 1's success row is now logged on a copy of the logging object that does not share the request's guardrail results, so a post-call guardrail that reports on the failed request counts once instead of on both rows A client hang-up during the follow-up now bills round 1 plus the follow-up's partial answer on one success row with round 1's id and the fixed margin counted once, the same way a plain stream's hang-up bills its partial answer The last usage chunk now carries round 1's usage and cost when only round 1 is billed, and a synchronous reader that drops the stream mid-answer bills round 1 on the loop it read with instead of discarding it The deferred release's logged_on argument is typed and every release closure accepts it, and the parked dispatch constants and protocols both MCP billing paths use now live in one module. The unit tests wait on a logging signal instead of a fixed delay and check the billed row through the proxy stream hook instead of private callback state * test(mcp): drive the unassembled follow-up through an upstream audio delta that is not base64 The test replaced litellm.stream_chunk_builder with a stub that raises. Round 2 now streams an audio delta whose data is not valid base64, so the real builder raises, the stream parks only best-effort usage, and the client's last usage chunk must carry round 1's tokens and cost * fix(mcp): count the fixed margin once in the combined stream row's usage cost * fix(mcp): bill round 2's partial answer when a client hangs up on a routed stream On the proxy the Router wraps the MCP stream, and when a client hangs up during round 2 the router's own cleanup closes that stream before the proxy's disconnect cleanup runs. The close logged round 1 alone, so round 2's partial answer was never billed. The MCP stream now bills round 1 plus round 2's partial answer as soon as the hang-up's cancellation reaches its read, before anything closes it * fix(mcp): count the follow-up's own messages in round 2's partial prompt The follow-up stream shares the request's logger, so it took the client's original messages as its prompt. When round 2 ended early, from a hang-up or a broken upstream, its partial usage counted only those messages and left out the assistant tool call and the tool results the follow-up actually sent. The follow-up stream now counts the messages the follow-up call sent * fix(mcp): bill round 2's partial when a routed stream's client hangs up between reads On a proxy the stream the disconnect cleanup holds is the router's fallback wrapper, so the cleanup never found the MCP stream's in-flight billing and released round 1 alone when the client hung up while the proxy sat at a yield. The round billing now registers its in-flight biller on the request's logging object, next to the parked release, and the cleanup looks it up there, so any wrapper chain bills round 1 plus round 2's partial once * fix(mcp): send the MCP stream usage chunk under the billed row id and count the partial off the loop A streamed chat completion that runs a LiteLLM-hosted MCP tool bills both model calls on one row under round 1's chatcmpl id, but the usage chunk it sent the client was round 2's held chunk and kept round 2's id, so a caller could not join the usage chunk to its row. The usage chunk now carries round 1's id in every path that sends it, with or without a deferred post-call guardrail, while its usage and cost and every content chunk's id stay as they were When the client hangs up during round 2, assembling round 2's partial answer counts the follow-up messages, tool result included, with the local tokenizer. That ran on the event loop and stalled every other request on the worker for about a second on a large tool result. The assembly now runs on a worker thread through asyncify from both the reader's cancellation and the proxy's disconnect cleanup, shielded from cancellation, and round 1 is checked again after the await so a release that happened meanwhile is never billed twice * test(mcp): wait on the loop with a hang guard instead of a 300 ms deadline * fix(mcp): give a proxy-rejected follow-up's failure row round 1's own cost breakdown * refactor(streaming): fold the partial response's empty guards into one check --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 07:34:10 -07:00
simonp9	594f862bf3	fix(streaming): rebuild each choice of an n>1 chat stream on its own (#49002) ... * fix(streaming): rebuild each choice of an n>1 chat stream on its own stream_chunk_builder folded every choice of an n>1 chat stream into choice 0, so post-call guardrails such as tool_permission re-emitted one choice holding every choice's tool calls, and spend logs and logging callbacks kept only the first choice's text. Each choice is now assembled from its own chunks, while usage, cost, and hidden params still read the whole stream. Cisco AI Defense compares each choice's streamed text on its own so safe n>1 streams still pass through unchanged. * fix(streaming): keep bridged reasoning summaries on choice 0 and infer an unreported choice finish reason * fix(streaming): match MCP follow-ups and bridged summary text to the rebuilt choices MCP auto-execute now answers only the first choice's tool calls, since its follow-up turn continues that choice alone. The Responses bridge streams a space before each reasoning summary part after the first, matching the non-streamed reply. build_base_response now takes the per-choice streams instead of having its choices replaced afterwards * fix(streaming): space every later reasoning summary part, deny per judged call, and rebuild in one text pass * fix(guardrails): key tool_permission denials for custom tool calls too, and type the stream rebuild's messages * fix(mcp): continue the first choice that called a tool when an n>1 reply auto-executes MCP tools * fix(noma): scan each choice of a multi-choice reply on its own * fix(noma): keep masked choices when a later choice's scan fails open on a stream --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 18:29:14 -07:00
Mateo Wang	dfd9ad4103	fix(responses)!: bill partial usage on mid-stream failures and report bridged /v1/messages failures to the proxy (#47537) ... A Responses API stream that fails partway now bills the usage it already consumed, ends with an error instead of a clean stop, and honors stream_timeout between chunks. Bridged /v1/messages calls report those failures to the proxy's failure hooks, so spend, budgets and logs see them	2026-10-28 13:32:09 -07:00
Shivam Rawat	f1795e6d44	fix(responses): keep every MCP tool round in streaming follow-up input (#47698) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>	2026-10-28 00:51:10 -07:00
tin-berri	c17faa1a27	fix(mcp): refuse calls to guardrail-hidden tools (#48669)	2026-10-27 17:13:40 -07:00
Mateo Wang	0a1dbc0fb4	refactor(mcp): drop the experimental label from the MCP server and client paths (#48513) ... * refactor(mcp): drop the experimental label from the MCP server and client paths Move litellm/proxy/_experimental/mcp_server to litellm/proxy/mcp_server and litellm/experimental_mcp_client to litellm/mcp_client, with the mirrored tests moved under tests/unit/proxy/mcp_server and tests/unit/mcp_client. Both old packages stay as shims that warn with DeprecationWarning, re-export the new package's public names and alias every submodule in sys.modules so deep imports keep resolving to the same objects. The OpenAPI docs gate now puts the merge base's enterprise package on PYTHONPATH so its base counts do not mix the checked-out enterprise code with the base litellm tree Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci(mcp): keep tests/unit/proxy/_experimental as a sharded root for the shim tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): alias namespace subpackages and bind attributes in the deprecated MCP shims pkgutil.walk_packages skipped mcp_server/auth because it has no __init__.py, so old imports of litellm.proxy._experimental.mcp_server.auth.* failed, and sys.modules entries alone left litellm.experimental_mcp_client.tools unreachable as an attribute. Walk the package on disk and setattr each alias on its old parent. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-27 17:47:50 +00:00
Mateo Wang	14193435b8	refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages (#48510) ... * refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages Move the unified /v1/messages code out of a directory called pass_through, since it is not the passthrough feature. The inner messages sub-package becomes native, and the Vertex AI partner model experimental_pass_through package becomes messages. The old packages stay as import shims that warn once and alias every submodule in sys.modules, so old deep imports still resolve Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): keep Final variables out of loops in the rename shim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): match the warning text in the rename shim test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): resolve old pass_through imports lazily through a meta path finder Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): mark the alias loader spec restore for the type discipline gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(anthropic): format the renamed package shim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(anthropic): keep the rebind-ok marker on the spec restore line Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): carry the new package search path on alias specs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): declare the shim test locals Final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-27 10:08:25 -07:00
Shivam Rawat	172aa72ad2	feat(responses): honor max_agentic_loops for the streamed MCP auto-execute round cap (#47688) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 15:12:07 -07:00
Mateo Wang	820725661e	fix(router): log each dropped /v1/messages attempt through its own logging copy and open the retry without its usage (#45194) ... * fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure A /v1/messages stream the router retries before any content shares one logging object across both attempts. The superseded attempt's end-of-stream logging ran as a success (a usage-only or placeholder result) and claimed the request's one-shot success log, so the recovered stream's spend row was either missing or written under the superseded attempt's message id. The passthrough logging path now logs an Anthropic stream whose frames carry an `event: error` as a failure with whatever usage the provider reported, the way a dropped stream already does, and the router closes the superseded stream before it opens the retry so that logging is queued first. * fix(proxy): log a provider error frame through an attempt-scoped logging copy The error-frame failure stashed the superseded attempt's usage on the logging object the retry shares, so the retry's success log carried the failed attempt's token counts. The failure now logs through a copy of the logging object with its own model_call_details, and the helper moves into the Anthropic passthrough logging handler so the proxy streaming handler only routes by endpoint type. * fix(proxy): scope the error-frame failure log to streams closed at the frame A stream that ran to its end at the provider's error frame kept logging through the success route before this PR, and the attempt-scoped failure log is now taken only for a stream the router closed at that frame, with the copy made at the close so the retry's deployment never reaches it * fix(router): log each dropped /v1/messages attempt through its own logging copy and open the retry without its usage A /v1/messages stream dropped after message_start and retried in its model group shares one Logging object across attempts, so the retry's success log inherited the dropped attempt's stashed usage and exception, and a second drop never reached the failure handlers because the first had already marked the object as logged. The dropped attempt now dispatches its failure handlers through an attempt-scoped copy, and the router discards the stashed usage right before it opens the next attempt, on same-group retries and on every fallback hop, so a request that runs out of attempts still bills the last attempt's partial usage. * fix(router): dispatch chat stream failures through the attempt-scoped logging copy The router now clears a dropped attempt's stashed usage before every fallback hop, so the chat stream wrapper's queued failure log must read that usage from its own copy of the logging object, like the /v1/messages path already does. The stash test asserts through a real success log, the cooldown test reads a real Router's cooldown list, and the integration helpers carry full types. * test(vertex): give the stream wrapper's logging stub an attempt_scoped_copy * test: route the pass-through logging mock's attempt copy to itself and read the callback sink through a reader The pass-through failure test stubbed dispatch_failure_handlers on a spec'd MagicMock, which the attempt-scoped copy no longer reached, so the logging worker never finished. The integration tests kept earlier sink batches in a caller-owned list the helper mutated; a small reader keeps them instead. * test(router): use real Logging in fallback cooldown test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): preserve superseded retry usage for billing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): bill retry input after prior stream usage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): track retry dispatch for terminal billing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): use deterministic retry timestamp Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): avoid real clock in fallback tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tests): remove duplicate streaming logger import Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): retain error-frame usage for exhausted retries * test(proxy): fix retry billing expectations and helper annotations * fix(proxy): preserve failure estimate without carried usage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): audit retry helpers and fixtures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): fix retry fixture timestamps Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): observe retry stash at attempt boundary Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): keep terminal failure attribution when dropped attempts log on copies Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): publish a dropped attempt's payload only while it is still the current attempt Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover spend-log encoding error propagation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): skip adapter failure dispatch when the wrapped stream already logged it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): cool down the final fallback attempt when its failure callbacks were blocked Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): import Final in adapter regression Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drop spend-log encoding expectation superseded by main's poison isolation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): finish the wrapped stream's failure log before surfacing the error Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): wait for the wrapped stream's failure log to finish like the inline dispatch did Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): wait for the executor-run sync failure callback separately from the async caller boundary Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): pin pass-through stream fixture start times Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): publish failed attempt payload before awaiting failure callbacks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): publish failed attempt payload from the failure handler after baseline invalidation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(logging): drop unused discard_partial_usage_for_failure Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-20 21:46:45 -07:00
Mateo Wang	0e24bbb8a2	fix(responses): bill every round of a streamed MCP auto-execute on one spend row (#47445) ... * test(responses): cover billing every round of a streamed MCP auto-execute * fix(responses): bill every round of a streamed MCP auto-execute on one spend row * fix(responses): bill every round of a streamed MCP auto-execute on bridged providers A streamed /v1/responses MCP auto-execute on a provider bridged through chat completions (for example Anthropic) billed only round 1, because each round's chat stream dispatched on the shared logging object and the first dispatch won. Bridged rounds now go through the same one-row billing as native rounds. A round is billed as soon as its chat stream ends, before its closing events reach the client. The row carries the final response, usage and cost summed over every round, and the fixed margin once. A round served from the response cache is billed at zero. A follow-up chat stream that breaks bills the paid rounds on its failure row. A single-round stream logs exactly as before. * fix(responses): charge one fixed margin and keep bridged usage fields on streamed MCP rounds A follow-up round that fails after earlier rounds were billed now charges the per-request fixed margin once, and its failure row carries no cost breakdown that would disagree with the summed cost. A bridged multi-round stream logs a plain cache_hit flag like a single-round stream does, and its usage keeps the summed provider usage fields of every round. A new test covers a stream dropped after its event loop closed, which must release its parked log without an unawaited coroutine warning. * fix(responses): price each streamed MCP round on its own and bill every round when a guardrail blocks Each round of a streamed MCP auto-execute is now priced from its own uncached cost, so a provider that states its cost (xAI) charges the stated cost of every round plus the fixed margin once, and the cost breakdown always agrees with the billed cost. Summed usage goes through a JSON-safe dump, so a nested provider field such as xAI's context_details no longer breaks a failed stream's response.failed event. A post-call guardrail that blocks a multi-round bridged stream now drops the parked success log and bills every round on the failure row, and a failed follow-up round's failure row carries the summed breakdown of both rounds. * fix(responses): keep earlier MCP rounds on their success row when a later round fails When a later round of a streamed MCP auto-execute fails, the rounds before it are now logged on their success row, and the failure row bills only the failed round with no second fixed margin. That failure log runs on its own copy of the request's logging state, so its usage and cost never leak into the earlier rounds' row. A post-call guardrail that blocks a native multi-round stream keeps the summed success row beside the $0 failure row, as at the merge base, while a bridged stream still bills every round on its failure row. * fix(responses): keep a failed later MCP round on its own failure row A streamed MCP auto-execute whose later round failed wrote only the earlier rounds' success row. The failure was logged on a copy of the logging state, while the proxy builds its failure row from the request's own state, which by then held the success payload, so the failure row took the success row's id and was dropped. Now the failure is logged on the request's own state as at the merge base, and once a later round starts, the success row for the rounds before it is logged from a copy of the state taken at that moment, so the two rows never mix. * fix(proxy): let an MCP tool call reuse its request's max_parallel_requests slot * test(proxy): create the other-request task in a way Python 3.10 supports and mark its bindings Final * fix(proxy): keep a reused max_parallel slot until the request's last round An MCP tool call that reuses its request's max_parallel_requests slot no longer loses it to the first round's late success callback, so the key stays counted for the rounds still to run. The reused call also stops replacing the request's rate limit headers. * fix(responses): keep an unpriced streamed MCP round unpriced instead of billing it as free * fix(responses): log a bridged MCP stream as a chat completion with every round's usage	2026-10-21 02:56:38 +00:00
Yassin Kortam	2f139a70ae	fix(mcp)!: never forward the admission LiteLLM key upstream for any auth type (#46932) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-20 14:17:29 -07:00
Mateo Wang	daa133da5c	fix(mcp)!: bill both model calls of a chat completions MCP auto-execute on one spend row (#47422) ... * test(mcp): cover one spend row billing both rounds of a chat completions MCP auto-execute * fix(mcp): bill both model calls of a chat completions MCP auto-execute on one spend row * fix(mcp): keep provider usage fields when summing both chat rounds of an MCP auto-execute * fix(mcp): log each auto-executed MCP tool call under its own spend-log id A chat MCP auto-execute logged its tool calls under the request's call id, which is also the id of the request's failure row. When the follow-up call failed, the database kept the tool row and dropped the failure row that bills the paid first call * test(mcp): cover the margin, cache, failure row and parallel tool rows of MCP auto-execute billing	2026-10-20 04:37:31 +00:00
MFJG	783cf06f31	fix(sdk): decide MCP gateway routing without importing the proxy handler (#45961) ... * fix(mcp): keep the gateway handler off the SDK path when no tool is an MCP tool completion(), responses() and anthropic messages imported the MCP gateway handler whenever tools were passed, and responses() did so on every call. Since the handler began importing the MCP catalog at module level in #42568, that import needs the optional mcp package, so a base SDK install failed with ModuleNotFoundError on any function-tool call and on every responses() call A new import-safe pre-check skips the handler unless some tool is a dict of type mcp, which is the only shape the handler's gateway decision can accept. MCP gateway tools still reach the handler as before Adds in-process regression tests for the three entry points and a function-tool completion to the base SDK install check * fix(mcp): leave one-shot tool iterables unread in the gateway pre-check responses() declares tools as an Iterable and the handler scans the same object a second time, so a pre-check that read a generator would hand the handler one tool fewer. The pre-check now scans only collections and reports anything else as possibly naming the gateway, which keeps the previous path The Anthropic messages regression test now mocks the httpx layer with respx and turns off the aiohttp transport, instead of replacing AsyncHTTPHandler * fix(mcp): scan only lists and tuples in the gateway pre-check The pre-check read any Collection, but a Collection that yields its items only once would still be exhausted by the handler's second scan. Only the list and tuple builtins are guaranteed to iterate the same way twice, so they are the only inputs read now. Everything else keeps the previous path and is reported as possibly naming the gateway The unread-input test now covers a generator, an iterator, a one-shot collection and a deque * fix(mcp): scan only exact list and tuple types in the gateway pre-check isinstance accepted list subclasses, and a subclass that iterates only once would still be consumed by the handler's second scan. The fast path now reads only objects whose type is exactly list or tuple. Every other input, subclasses included, keeps the previous path and is reported as possibly naming the gateway The unread-input test gains a one-shot list subclass and a tuple subclass * fix(sdk): decide MCP gateway routing without importing the proxy handler The pre-check now holds the handler's exact gateway test, so remote MCP tools, mcp 1.x installs and any tools iterable stay off the handler, and the base SDK check calls responses() with a remote MCP tool. * fix(sdk): send /mcp/<name> remote MCP URLs to the provider when the gateway registry cannot load A path-only /mcp/<name> match now counts as a gateway tool only when the MCP server registry imports, so SDK installs without the proxy extras pass remote MCP servers like Zapier to the provider. Also drops a dead test patch. * test(sdk): send a /mcp/<name> remote MCP tool in the base SDK install check * style(sdk): drop docstrings from the MCP gateway pre-check and its tests * test(mcp): cover /mcp/<name> gateway routing on the proxy and Messages tools on a base SDK install --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-20 01:44:54 +00:00
devin-ai-integration[bot]	86a91f35de	fix(responses): log and bill every round of non-stream MCP auto-execute (#43667) ... * fix(responses): log and bill every round of non-stream MCP auto-execute Both model rounds run as internal sub-calls on the request's logging object, and the outer call logs the response the client received with usage, cost, and cost breakdown summed across the rounds. The internal-call gate now also covers the sync success callbacks, so an inner round no longer takes their first-wins slot * fix(caching): let the parent call log and bill an internal sub-call's cache hit A round of non-stream MCP auto-execute that the response cache served fired the cache-hit callbacks on the shared logging object, so the spend log kept round 1's function_call row and dropped the billed row for the answer. Internal sub-call cache hits now fire no callbacks and carry a zero cost, and the summed cost breakdown leaves out rounds served from the cache * fix(logging): keep a parent row unique when its response came from the response cache A RAG query or an MCP turn whose final round is served from the cache hands back the cached response's id, which already belongs to the row that cached it, so SpendLogs dropped the new row as a duplicate. Such a row now gets the cache-hit id suffix, and it is logged as a cache hit when it cost nothing. * fix(logging): skip the zero-cost warning for a response served from the cache at no charge A non-stream MCP auto-execute request whose rounds the response cache served is priced at $0 before the parent marks it a cache hit, so it logged a false 'priced at $0' warning and counted in litellm_zero_cost_requests_total. The diagnostic now treats a response the cache served at no charge the same as a marked cache hit * fix(responses): bill the paid first MCP round when a later step of the request fails * fix(responses): log each MCP tool call under its own id so a failed follow-up keeps its spend row The non-stream auto-execute path passed the request's call id to the tool call, so the tool's spend row and the request's failure row shared one request_id and the failure row was dropped on insert, leaving the paid first round unbilled. * fix(responses): bill round 1 on a client disconnect, keep MCP tool rows in the request's session, and charge a fixed margin once per request * fix(responses): keep bridged MCP auto-execute in one session and trace and continue its conversation The bridged follow-up no longer chains on round 1, so a thinking model keeps its reasoning and the request stays in one session. The tool call log continues the request trace without renaming it, the tools are fetched once, and a later turn finds the MCP turn under the id the client got, skipping the tool rows. * fix(responses): log the non-stream MCP tool listing in the request's session and trace * fix(responses): find a bridged MCP turn by the id the client holds and log its listing in the bridged trace * fix(responses): derive the MCP tool row id from the request's call id The tool call's spend row now uses <call id>_<tool call id>, so it stays linked to the request without colliding with the request's failure row. Also tighten the internal cache-hit test to a fixed clock and typed params. * fix(responses): bill a retried MCP attempt for its own rounds A router retry or fallback reuses the request's logging object, so a failed attempt's stashed round 1 usage made the next attempt's success row log stale tokens. Each non-stream MCP attempt now starts without that stash. * fix(responses): bill every failed MCP attempt's paid first round on the request failure A retry or fallback now sets a failed attempt's stashed round 1 aside instead of dropping it, and the proxy's failure row adds it to the last attempt's own, so a request whose later attempt fails on its first call still bills the round the provider charged. A cached final round also keeps the paid first round's cost breakdown. * fix(responses): keep MCP tool catalog and results out of the bridged next-turn replay * fix(responses): keep MCP tool spend, client body keys, and Langfuse traces working on auto-execute Count the MCP tools an auto-executed Responses API call ran in the tool spend rollup, accept a client body key named previous_response_client_ids without a 500, and point MCP listing and tool spans at the trace Langfuse gives the parent request, including x-litellm-session-id requests * fix(responses): join MCP spans to the Langfuse trace when the request sends body metadata * test(integration): cover non-stream MCP auto-execute billing, failure rows, and bridged follow-ups --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 17:37:33 +00:00
Mateo Wang	2c4ec9c6fa	fix(responses): open faked stream items empty so tool arguments and text arrive once (#46533) ... * fix(responses): open faked stream items empty so tool arguments and text arrive once * fix(responses): open faked stream items in progress with a fresh empty list, and open mid-stream guardrail blocks empty * fix(responses): open faked refusals, reasoning summaries, custom tool calls and MCP calls empty so each value arrives once * test(integration): cover the empty-open Responses stream items on every fake, replay and block path Integration cells for the faked /v1/responses stream, the chat and messages bridges, cache replay, guardrail block events and the MCP gateway call. Each opened item starts empty and each value arrives once, checked with raw httpx, the OpenAI SDK and the Anthropic SDK, plus upstream errors, 5 KB values, a client drop, a dropped upstream connection and a worker SIGKILL	2026-10-15 09:35:12 -07:00
devin-ai-integration[bot]	102619d47a	test: migrate proxy security, schema migration and MCP legacy tests (#45937) ... * test: migrate proxy security, schema migration and MCP legacy tests * test: run the MCP proxy e2e upstream under the SDK1 peer in integration-mcp * test: warm the baseline store before timing the locked projection * test(e2e): hold the stack exclusively while the semantic filter flag is flipped * test(mcp): scope the in-process proxy e2e fixtures to their module The session-scoped autouse fixture deleted DATABASE_URL and overrode LITELLM_MASTER_KEY for the rest of the worker, which broke every later test in the integration mcp suite * test(e2e): assert the semantic filter keeps listed tools instead of a ranking Which Datadog tool an OpenAI embedding ranks first is not something litellm owns, so the test now checks that the kept tools are a non-empty subset of the key's listing within top_k * test(mcp): keep the in-process proxy e2e tests on one worker loadgroup spread the file's tests across workers and interleaved them with other files, so the module-scoped in-process proxy was torn down and restarted between tests and requests timed out * test(mcp): name the upstream requests that carried the wrong token The bare all() assertion hid which request broke the forwarded token, which made an intermittent CircleCI failure undiagnosable --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-11 16:17:24 -07:00
yuneng-jiang	9b53ee878f	perf(ci): isolate and shard integration suites (#44146) ... * perf(ci): distribute unit cases and reuse security sweep connections * revert(ci): reject first speed trial after new failures * perf(ci): avoid repeated test discovery and parsing * perf(ci): isolate parallel security workers * revert(ci): restore measured security execution isolation * perf(tests): reuse source scans and deduplicate cache cleanup * perf(ci): restore parallel security with safe log batching * test(ci): isolate fixture state and schema comparisons * test(ci): make schema and scheduler fixtures deterministic * test(ci): drain queued logging before fixture teardown * test(ci): preserve logging delivery without idle worker restarts * test(ci): keep guardrail sinks available during parallel bursts * test(ci): isolate full proxy lifecycle from gateway imports * test(ci): preserve first failure responses and sweep evidence * test(ci): restore documentation URL settings between cases * test(ci): isolate open MCP tools in private gateways * test(ci): group private MCP cases and handle exited workers * test(ci): retain proxy exits and MCP failure details * test(ci): repair merge interactions with main Key the schema shard's full clone on artifact-name, since main removed the unit-flag input. Give each owned Redis its own log file again. Hand the realtime transcription tests main added a logging object whose dispatch handlers are awaitable, so the per-test logging drain no longer waits on a MagicMock task forever * perf(ci): shard owned integration cases and close lifecycle leaks * fix(ci): qualify shard execution and harden interrupted cleanup * fix(ci): bound worker supervision and own fault traffic * perf(ci): reuse unparametrized fixtures across bounded case groups * fix(ci): preserve boot fault allowances and complete exit records * test(ci): tolerate startup retries and retain readiness details * perf(ci): balance integration shards by recorded duration * fix(ci): load migration database helpers only for workers	2026-10-10 11:15:19 -07:00
Mateo Wang	58258409c9	refactor(proxy): expose public names for private proxy helpers (#45170) ... * refactor(proxy): expose public names for private proxy helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep original class names behind public aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve internal callback filtering Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep _PROXY_ class names for managed files hooks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names in package exports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve recursive auth helper name Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): align MCP limiter tests with server enforcement Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): match main's MCP limiter tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names bound in importing modules Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve compatibility imports through strict lint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): use exact pyright suppression in password helper test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): add reasons to compatibility import noqa comments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): update IN-list baseline for renamed helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-07 18:59:27 -07:00
joshua-berri	950da7c2ec	fix(mcp): keep worker MCP configurations consistent via a catalog revision (#42568) ... * fix(mcp): refresh server catalog for each gateway operation * fix(mcp): reject configuration changes during scoped dispatch * fix(mcp): refresh shared catalog state for each operation * fix(mcp): refresh catalog before native alias routing * docs(mcp): clarify native route resolution order * test(mcp): align streaming fixture and generated API documentation * fix(mcp): preserve discovery published during catalog refresh * fix(mcp): coalesce queued catalog reads without a stale window * fix(mcp): reconcile concurrent route changes when publishing catalog * test(mcp): provide catalog scope in post-call hook fixtures * fix(mcp): retain valid live routes and handlers during refresh * fix(mcp): keep refreshed OpenAPI operation membership authoritative * test(mcp): preserve logging fixtures after catalog integration * fix(mcp): isolate transport test admission state and exhaust OAuth outcomes * refactor(mcp): narrow catalog consistency change to ticket scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(mcp): gate catalog refresh on a database revision marker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): refresh waiters that observed a newer catalog revision Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): awaitable catalog revision doubles in prisma mocks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(mcp): satisfy catalog refresh lint and type gates * test(mcp): provide catalog scope in toolset fixtures * test(mcp): exercise temporary OAuth through catalog operations * fix(mcp): share active catalog snapshots across discovery tasks * fix(mcp): retain discovered routes across cached catalog operations * fix(mcp): preserve local tool ownership across discovery * refactor(mcp): satisfy tightened immutability lint ceiling Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): authorize local handlers by registered server ownership * fix(mcp): check registered handler freshness and isolate fixtures * fix(mcp): read catalog revisions and snapshots from the writer * fix(mcp): resolve access groups from the operation catalog * fix(mcp): preserve empty access group restrictions * fix(mcp): retain rediscovered routes across concurrent updates * fix(mcp): enforce registered ownership for local tool dispatch * fix(mcp): preserve issuer discovery across catalog snapshots --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-06 18:21:29 -07:00
devin-ai-integration[bot]	c3a23fe499	refactor: expose core private helpers under public names (#44871) ... * refactor: expose core private symbols with compatibility aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * types: narrow core migration diagnostics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve runtime behavior in core symbol migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve core private usage migration behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve private value rebinding compatibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore optional imports and cover public helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: exempt router property from call coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: align recursive detector ignore names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-07 00:05:27 +00:00
devin-ai-integration[bot]	0b74ae9c5c	feat(mcp): hand listed-tool description and input schema to pre-call hooks per caller (#41162) ... * feat(mcp): hand listed-tool metadata to pre-call hooks with per-caller catalog identity Track the tools each MCP server listed per caller identity so pre_mcp_call and during_mcp_call hooks receive the tool description and input schema the client saw. Servers with no caller-dependent inputs share one slot; user identity, forwarded headers, stdio env, relayed bearers, and server-specific auth get their own. Local registry and OpenAPI paths pass the registered metadata and admin description overrides. The Agent 365 guardrail reads the new fields into its evaluate payload. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(mcp): drop the listed-tools empty sentinel and routine test docstrings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): mark the listed-tools cache digest as a non-security hash Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): key the listed-tools cache by the OBO subject token token_exchange servers list upstream with the caller's own Entra bearer, so two callers on one LiteLLM key with different subjects were sharing a catalog slot Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): resolve the BYOK credential before keying the listed-tools slot Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): drop the OAuth discovery cache when a server definition changes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(mcp): drop a diff-narrating comment Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): never validate a supplied header on the tools/list BYOK path The pre-listing resolver ran the tool-call byok_auth_required check even when the caller already supplied x-mcp-auth, and it ran outside the per-server error boundary, so a single deprecated-header caller dropped the server from the aggregate list. Listing now returns a supplied header unchanged and falls back to the stored credential without raising Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): assert the BYOK listing lands in the caller's listed-tool slot Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): cover the deprecated string x-mcp-auth header on a BYOK tools/list Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): key the per-caller listed-tool slot by the hashed token Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): key discovery cache by the hashed token Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): key discovery caches per caller correctly and drop stale caches on server updates Discovery-list cache identity now uses the hashed token instead of the raw api_key and treats MCPJWTSigner-signed servers as per caller. Server definition changes also drop the cached upstream OAuth metadata. OpenAPI listings look tools up under the normalized registry prefix with the separator, so an overlapping sibling prefix no longer leaks into the list. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): keep the discovery cache digest call unchanged so CodeQL matches the existing alert Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(mcp): derive the listed-tool caller identity from the discovery cache key Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): guard OAuth metadata cache writes with a per-server generation and drop unproven per-caller discovery keys An upstream metadata fetch that started before a server edit could store its stale reply after invalidate_oauth_metadata_cache ran. Invalidation now bumps a per-server generation and the fetch only stores when the generation it captured before I/O is unchanged. The MCPJWTSigner-based per-caller discovery classification and the api_key to token key change had no reproduction (the signer only injects on tools/list, and UserAPIKeyAuth hashes api_key in place), so both go back to the merge-base behavior. Integration coverage under tests/integration/mcp: overlapping OpenAPI aliases, a config-declared server name with a space, OAuth metadata refetch after a save, and the in-flight stale-write race Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): keep OAuth metadata generations only while a fetch is in flight Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): count queued OAuth metadata fetchers so invalidation survives lock handoff Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): keep a held OAuth metadata lock registered even when no fetcher slot claims it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): prove a peer worker drops stale upstream OAuth metadata after a save elsewhere Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): return one masked text per scanned string in the selected-guardrail REST test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): fold the signed caller into the discovery digest instead of a second key hash Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): satisfy type discipline gate on listed-tool identity Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): hand tools/call hooks the exact catalog entry tools/list served get_listed_tool re-applied the admin description override on top of the cached listing, so a guardrail-masked description was restored to its original wording at call time, and the OpenAPI / local-registry call path built its metadata from the registry instead of the guarded caller catalog. Both paths now return the cached entry as served, falling back to the registry only when no listing was recorded Adds tests/integration/mcp/test_mcp_listed_tool_metadata.py (red on the prior head for the two regressions, red on the merge base for the feature, green on this head) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): key OpenAPI listed-tool entries per caller so tools/call reads its own guarded listing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): align listed-tool slot tests with per-caller keying Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): keep oauth2 listing on the minted or signed credential, not the stored BYOK secret The listing helper that keys the per-caller catalog by the stored BYOK credential also handed that credential to the upstream client, which on an oauth2 server short-circuited the client_credentials mint and the MCPJWTSigner gate. Split the two: the catalog identity keeps the stored credential so tools/call finds the caller's slot, while an oauth2 server's tools/list sends only the per-request header, letting the M2M mint or signed JWT proceed as on main Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): oauth2 BYOK listing sends the minted token, not the stored secret, through the real proxy Integration cell for the listing fix: a client_credentials BYOK server with a stored user credential, one tools/list as that user, the peer must see a live minted bearer and one /token mint. Red at the pre-fix tip (zero mints, stored secret upstream), green at the fixed head and at the merge base Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): keep the stored BYOK credential for catalog identity only on tools/list Listing used the resolved stored credential both to key the caller's catalog slot and as the upstream transport header, so REST api_key and bearer_token listings sent the user's secret instead of the server's static token and the MCPJWTSigner gate went quiet. The upstream client and the signer gate now read the caller-supplied mcp_auth_header for every auth type, exactly as before the catalog existed, and the stored credential only names the slot tools/call reads Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): type the listed-tool metadata read from pre-call kwargs for the basedpyright gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): hand never-listed tools/call hooks name and arguments only The local-registry call path fell back to the registry entry with the admin description override when no tools/list had been recorded for the caller, so a pre_mcp_call guardrail scanned a description the caller was never served and blocked OpenAPI calls that passed before, and base's own selected-guardrail REST test failed on the two-text redaction. _registered_tool_metadata now returns the listed entry or None, so a tools/call with no prior listing sends name and arguments only as promised, and that REST test double goes back to its base shape * fix(mcp): keep during_mcp_call hooks on name and arguments only call_tool handed the caller's listed entry to the during-hook task as well, so during_mcp_call guardrails scanned the description line and schema leaves of any listed tool after the upstream call had already run, blocking calls that passed before whenever the policy matched the description, returned a fixed-length texts list, or hit the depth guard on a deep schema. The listed entry is only disclosed for pre_mcp_call, so the during task no longer receives it and its request object carries no description or schema, as before * fix(mcp): key the BYOK catalog slot by the client's header, not the stored credential tools/list resolved the stored BYOK credential to pick the caller's catalog slot, which read the credential store before the classified try block. With Postgres down and a cold per-worker cache that made every REST tools/list on an is_byok server fail with tools=[] and no upstream call, and the read seeded the per-worker cache (including a negative entry), so a tools/call on another worker after a store, rotate or revoke on this one kept using the stale value. The slot is now keyed by what the client supplied plus the caller's hashed key, on both sides. _get_tools_from_server and call_tool take a keyword-only catalog_auth_header that defaults to mcp_auth_header as received (the default is the builtin Ellipsis so it survives a module reload). The /mcp fan-out and execute_mcp_tool, which swap the resolved credential into mcp_auth_header, pass the client's value explicitly. What goes upstream is unchanged. _byok_catalog_auth_header is gone. * fix(mcp): drop a listed catalog recorded across a server save _record_listed_tools ran after the awaited upstream fetch, so a PUT /v1/mcp/server that landed mid-fetch had its invalidation undone when the fetch completed: hooks then saw the pre-save description next to the post-save definition until the next listing, instead of name and arguments only. The manager now keeps a per-server listed-tools generation, bumped by _invalidate_server_definition_caches. _get_tools_from_server reads it before the fetch and _record_listed_tools skips the write when it moved; the next listing records normally. * fix(mcp): drop the catalog again once a saved OpenAPI server's registry is rebuilt add_server and update_server publish the saved definition before the OpenAPI registry entries are rebuilt from the spec, so a listing recorded during that fetch held the pre-save entries under the new generation. The generation is bumped a second time after the registry refresh. The during-hook task no longer accepts a listed entry, the one-line wrapper over get_listed_tool is inlined at its two call sites, and the per-server generation map is a plain dict. * fix(mcp): keep discovery and OAuth metadata caches across an OpenAPI spec re-read add_server and update_server ran the full server-definition invalidation a second time after the awaited OpenAPI spec fetch, which also dropped the prompts/resources/templates discovery entries and the OAuth protected-resource metadata filled under the already-published definition, so the next request went upstream again. Only the listed-tool catalog recorded during the fetch holds pre-save entries, so the post-fetch pass now drops just that catalog and bumps its generation via the new _drop_listed_tools helper, which the full invalidation also calls. * fix(mcp): look a called tool up in the listed catalog by its bare name only get_listed_tool stripped the server prefix a second time when the exact name was absent from the caller's listing, so a never-listed upstream tool whose bare name starts with the server prefix resolved to the listed sibling and that sibling's description and input schema reached the pre-call hooks for a call to a different tool. Every caller already passes the once-stripped bare name, so the lookup is now exact. Tests that looked the catalog up by a prefixed name now use the bare name the callers pass; two new tests pin the never-listed sibling case at the manager and at the tools/call path. * fix(mcp): record a listed-tool catalog only for a listing the caller is served _get_tools_from_server now records the catalog into the caller's listed-tools slot only when asked (record_listing=True), which the served listings pass: the /mcp and Responses API tools/list handlers via _get_tools_from_mcp_servers, MCPServerManager.list_tools, and the REST listing via _list_server_tools. Four internal listings stop recording, so a later tools/call hands pre_mcp_call hooks name and arguments only, as on main: - _list_tools_before_first_call, the implicit listing inside tools/call when this worker does not yet expose the tool - fetch_pinnable_tool_catalog, the admin pin snapshot listed without the catalog guard and without description overrides - _initialize_tool_name_to_mcp_server_name_mapping, the startup fill - get_tools_for_server, used by the semantic tool filter _create_prefixed_tools returns to its tool-name mapping job only; the record follows it in _get_tools_from_server. * fix(mcp): opt every listing out of catalog recording unless it is served The aggregate listing and _list_mcp_tools now default to record_listing=False, so a catalog fetched inside a tools/call no longer fills the caller's listed-tools slot. The /mcp/proxy meta-tools (call_tool, search_tools, get_tool_schema) and the tool-search virtual tool stop recording: /mcp/proxy serves only the meta-tools and the search serves only its hits, so a later pre_mcp_call hook was reading a description the caller never listed. The tools/list handler, the Responses MCP handler and the /v1/mcp/tools management listing opt in with record_listing=True, since each serves the catalog to the caller. * fix(mcp): key the listed-tool slot by the caller's admission identity and forwarded bearer The slot a tools/list records for a later tools/call was keyed by (user_id, api_key) only, so every team-only JWT caller shared one slot and one JWT user acting in two teams shared a slot; a tools/call then handed pre_mcp_call hooks a description another caller was served. The slot is now keyed by the hashed key, user, team and organization, plus the admission credential of a caller admitted with neither a key nor a user. The caller bearer split the slot only on client-forwarded-token and token-exchange servers; a legacy delegated oauth2 server (delegate_auth_to_upstream without client credentials) also forwards it upstream and served a different catalog per bearer into one slot. The bearer now splits the slot on every server whose egress forwards it (_consumes_caller_authorization) or exchanges it. * fix(mcp): record only tools served by the bridge Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): keep bridge tool metadata request-local * refactor(mcp): centralize listed catalog recording guard * fix(mcp): preserve base TPM reservations for listed tool calls * fix(mcp): preserve project token reservations for listed calls * fix(mcp): record served catalogs and preserve call message bytes * test(mcp): align listing expectation with deferred recording * test(mcp): audit listed metadata across callers and bridge lifecycles * test(mcp): preserve guardrail fixture worker affinity * refactor(mcp): expose listed catalog recording API * feat(mcp): pass served_tools through the anthropic messages bridge /v1/messages auto-execution now hands this request's resolved tool definitions to _execute_tool_calls, matching the Responses and chat completions bridges: the pre_mcp_call hook receives the description and input schema the model was shown for that call. Request-local only; the shared listed-tools catalog is untouched. * test(mcp): pin served_tools handoff on the anthropic messages bridge Mirrors the credentials-forwarding test: the request's resolved tool definitions must reach _execute_tool_calls under served_tools so pre_mcp_call hooks judge the call on the description and input schema the model was shown. Fails without the previous commit's one-liner. * style(mcp): sort the local import block ruff flagged * fix(mcp): keep the admin include_disabled_tools view off the listed-tools catalog GET /mcp-rest/tools/list?include_disabled_tools=true is the admin-only configuration view: apply_tool_filters is False, so it serves the full server catalog. Recording that response into the caller's listed-tools slot warmed tools/call metadata no runtime listing ever served, breaking the only-a-served-listing-records invariant (Bugbot). The record is now gated on apply_tool_filters; disabled tools stay unreachable (the call-time allowlist 403 fires before hooks), so the observable fix is the slot no longer warming from a settings view. Verified live: the new test fails on the unfixed head and passes here, and the rest of the listed-tool-metadata suite is unchanged. --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-04 00:53:58 -07:00
devin-ai-integration[bot]	d7e6154863	fix(mcp): resolve team-granted toolsets for non-admin keys and dashboard sessions (#43908) ... * fix(mcp): expand team and dashboard grants when listing and serving toolsets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): honor team toolset grants on the responses gateway path and expose a public team permission lookup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(mcp): drop the toolset route docstring tweak so the OpenAPI snapshot stays unchanged Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): scope inherited toolset grants by the key's own MCP ceiling A key that declares any MCP grant of its own keeps only its own toolsets, one that declares none inherits its team's, and require_key_mcp_access_defined stops a virtual key inheriting while dashboard sessions and admitted users still do. Adds the direct, no-grant, admin and key-ceiling integration cases and makes the LLM gateway toolset case discriminate a scoped toolset from the aggregate grant Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): resolve toolset grants per admitted source and enforce the live team roster Dashboard sessions and gateway-admitted users now expand into the admitted subject's per-team sources when resolving toolset grants, so a team-granted toolset is not capped by the user's own MCP row and is reachable on the namespaced route. A cached team id no longer grants a toolset unless the live roster still lists the user, a team lookup fault only drops that team's inherited grant, and /team/member_add evicts the cached team object so the new member is authoritative immediately Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): read a key's named object permission before letting it inherit team toolsets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(mcp): inject the toolset grant resolver into scope helpers so tests stop patching MCPRequestHandler Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): drop the duplicate admitted_subject_sources wrapper after merging main and follow its renamed resolvers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): honour fresh policy and the session resource scope on pinned toolsets A pinned toolset scope now reads the toolset through the writer when the admitted session requires fresh policy, so a tool revoked from the toolset is gone on the next request. A gateway bearer scoped to one server can only open a toolset that names that server Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): keep operator-open servers out of toolset gateway urls Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): audit cells for team-granted toolsets across every surface Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): bound toolset edit convergence by both cache layers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): drop toolset integration cells that test behavior this PR does not change Repeat-read byte identity, 20 concurrent calls, a stopped peer and a killed worker are covered generically by test_mcp_resilience.py and test_mcp_user_env_vars.py. The toolset edit cell asserts pre-existing cache propagation and flaked locally with connection resets while polling * chore(mcp): drop mutable-ok suppressions that main's LIT013 now flags as unused * chore(mcp): keep the require_key_mcp_access_defined read from adding an unknown-argument type error --------- Co-authored-by: ryan <ryan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-01 14:15:06 -07:00
devin-ai-integration[bot]	ce25856424	feat(mcp): scan and pin upstream tool descriptions (#43283) ... * feat(mcp): scan and pin upstream tool descriptions Run every discovered MCP tool's description and input schema through the pre_mcp_call guardrails before a listing reaches the client, drop the tools a guardrail blocks, and serve the guardrail's masked text otherwise. Add POST and DELETE /v1/mcp/server/{server_id}/pin so an admin can freeze a server's tool names and descriptions; the gateway serves the pinned catalog and raises a Slack alert with the diff when the upstream drifts. * chore: sync schema.prisma copies from root * fix(mcp): pin input schemas, scan before pinning, admin-only pin writes * fix(mcp): apply overrides and the pin before the discovery scan, dedupe alerts before sending The guardrail scan now runs on the text the client is about to see: description overrides are applied first, the pinned catalog next, and the scan last, so a masked pinned or override description is served masked and a pinned tool keeps serving its pinned text while the upstream's text is poisoned. The alert signature is recorded before the send and dropped only when that send fails, so a recovery during a slow send is never undone. A tool whose scan payload cannot be built is hidden alone instead of failing the listing. apply_tool_overrides shrinks to apply_display_name_overrides and the MagicMock servers in the MCP tests carry pinned_tools=None. * fix(mcp): snapshot the pin through the REST module's unpinned catalog helper * fix(mcp): pin the raw upstream catalog so an override never hides upstream description drift * refactor(mcp): trim the tool catalog guard docstrings to one line * test(mcp): cover guarded discovery boundaries and response definitions * fix(mcp): bound discovery guardrail concurrency per catalog * fix(mcp): scan tool catalogs in bounded parallel batches * fix(mcp): hide pinned catalogs from restricted management views --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-09-28 18:38:49 -07:00
yuneng-jiang	a11a93f44a	test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit (#43199) ... * ci: run the unit_selection.sh shard files on every event instead of only fork pull requests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: rename fork-flag to unit-flag now that it applies on every event * test: move tests/test_litellm root and small trees into tests/unit Pure renames, no content changes. Follow-up commits in this PR fix references, merge the three files that already existed in tests/unit, keep live-provider tests in tests/test_litellm and wire CI. * test: carry tests/test_litellm conftest isolation into tests/unit Callback lists, routing fallbacks, cached HTTP clients, logger state, AWS, proxy-URL and keychain env, and session-end client cleanup now reset for unit tests too. The environment isolation owns its MonkeyPatch so a test's own monkeypatch is undone before the model-cost teardown runs. * test: merge, split and prune the moved root and small-tree tests Merge batches/test_batch_utils.py and the chat_completions and messages dispatch tests into the files that already existed in tests/unit. Keep the live Gemini interactions tests, the async image-fetch format test and the OpenAI embedding scorer test in tests/test_litellm since they need real network or keys. Put test_router.py under tests/unit/test_router so the existing package no longer shadows it. Delete eight tests the audit found superseded by stronger ones kept in this move. * ci: run the moved root and small-tree tests under their legacy flags Add the misc and responses-caching-types flags to unit_selection.sh and CircleCI, extend enterprise-routing and mcp-integration, and point the legacy GHA shards, Makefile, redis-compat workflow, merge smoke manifest and change classifier at the new paths. * test: make the new tests/unit directories packages tests/unit/test_package_layout.py requires every directory to carry an __init__.py, and without one the moved and retained test_litellm_responses_bridge.py modules collide on import. * test: scope the unit socket block to tests/unit in shared sessions The GHA shards collect the legacy test-path and the unit selection in one pytest session. The unit conftest's loopback-only block leaked into legacy modules that reach the network at import. The legacy conftest now lifts the restriction at collect and setup time, and the unit conftest re-applies it when collecting its own modules. * test: move tests/test_litellm/llms into tests/unit/llms Rename-only. Moves the provider tests and the fine-tuning fixtures they load, mirroring the old paths. Follow-up commits merge, split and wire them. * test: merge, split and prune the moved llms tests Merges the Databricks chat transformation tests into the existing unit file, keeps the tests that need real keys or the network in tests/test_litellm, deletes the audited tests a stronger unit test already covers, and points imports at tests.unit.llms. * ci: run the moved llms tests under their legacy flags The Vertex AI and All Other Providers shards keep their legacy test-path for the retained files and add the llm-vertex-ai and llm-other-providers unit selections. CircleCI gets matching unit jobs. * test: make the tests/unit/llms directories packages Adds __init__.py to the moved dirs and drops the legacy ones whose directories no longer hold tests. * test: drop script runners and path hacks the llms split left dangling The __main__ runners in the split openai_like files and the Databricks e2e runner called tests that now live in the other half of the split or were deleted. The retained legacy halves also no longer need sys.path edits. * test: give the shard-script tests their own GITHUB_OUTPUT They only passed where the runner set it. The CircleCI unit job's env allowlist drops it, so the script's redirect failed there. * test: point the router and module-deletion checks at tests/unit router_code_coverage and code_qa_check_tests only searched tests/test_litellm, so the moved router tests no longer counted. The two silent-experiment tests the audit deleted were the only direct callers of those methods; they are replaced with tests that assert the forwarded shadow request and the recursion guard. * test: move tests/test_litellm integrations and secret_managers into tests/unit Rename-only. Mirrors the old paths, including the directory conftests and the prompt and JSON fixtures. Follow-up commits prune and wire them. * test: prune and repoint the moved integrations tests Deletes the 7 audited tests a stronger test in the same tree already covers, imports the TLS sink helpers from their new conftest path, and restores os.environ after each integrations test. Some presets write OTEL_EXPORTER_OTLP_HEADERS straight into os.environ, and without the legacy tree's test ordering that header leaked into the AgentOps tests. * ci: run the moved integrations tests under their legacy flag The integrations GHA shard and a new CircleCI job run the integrations unit selection. secret_managers joins the misc selection. * docs: point integrations and secret_managers references at tests/unit * test: make the moved integrations directories packages * test: keep the Databricks manual e2e runner and fix the SageMaker Nova run path The Databricks e2e file is a manual script whose main() calls the tests that were pruned, so pruning them broke the documented run. It is back to its main version. The SageMaker Nova docstring now points at the file's real location in tests/local_testing. * test: move tests/test_litellm core utils, routing, responses, caching and rust_bridge into tests/unit Rename-only. Mirrors the old paths, including fixtures, the stubtest config and the native-route wheel script. Two files that collide with existing unit files are merged in a follow-up commit. * test: merge, prune and repoint the moved core, routing, responses, caching and rust_bridge tests Merges the two files that collided with existing unit files, folding the legacy extra case into test_is_chat_completion_cached_dict, and deletes the 9 audited tests a stronger test in the same file already covers. Keeps what needs the network in tests/test_litellm: test_tokenizers pulls a tokenizer from the Hugging Face hub, and the gpt2 and r50k_base tokenizer cases download their BPE files. The unit core_utils conftest points TIKTOKEN_CACHE_DIR at litellm's bundled encodings so the rest never depend on import order to stay offline, and FakeSecretVault moves to a shared module so both trees can build it. * ci: run the moved core, routing, responses, caching and rust_bridge tests under their flags core_utils gets a core-utils flag and CircleCI job, and its GHA shard keeps the legacy path for the retained network tests. router_utils and router_strategy join enterprise-routing, responses joins responses-caching-types (minus responses/mcp, which mcp-integration owns), caching joins caching-local and rust_bridge joins misc. The redis-compat, test-rust, stubtest and merge-smoke paths follow the move. * docs: point the Rust crate references at tests/unit * test: make the moved core, routing and rust_bridge directories packages * test: keep the no-loop DualCache batch_get_cache regression test It runs the sync path outside any event loop, which the inside-loop test cannot, so a change that picks the Redis client by loop state would only show up there. * test: keep the job's UNIT_FLAG out of the shard-script tests * fix(url_utils): block 192.0.0.0/24 on every Python patch release * test: move the new budget limiter tests into tests/unit/router_strategy * test: move the new sentry scrubbing tests into tests/unit/litellm_core_utils * test: move the new zerobus tests into tests/unit/integrations * test: make tests/unit/integrations/zerobus a package * test: load litellm's own tiktoken cache setup once instead of resetting it per test --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-09-25 17:10:13 -07:00
devin-ai-integration[bot]	ba77646991	ci: move provider-independent MCP tests into tests/unit and run mcp-integration from litellm-tests (#42904) ... * ci: fix the litellm-tests unit job with sysmon coverage, an env allowlist and coverage upload on failure * test: replace key-dependent proxy, enterprise and mcp unit tests with synthetic values and integration and e2e coverage * test: drop key reads at the legacy proxy, enterprise and mcp paths and wire the gemini pass-through split * ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags * ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags * ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests * ci: move provider-independent MCP tests into tests/unit and run mcp-integration from litellm-tests * ci: fail the unit shard when circleci tests split errors * test: drop restating comments from the gemini pass-through split * build: point the local proxy unit targets at the nested tests/unit/proxy tree * ci: exit the unit shard cleanly when circleci tests split assigns it no files --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-09-24 23:07:48 +00:00