mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-14 23:21:35 +00:00
test(e2e): expand MCP coverage registry from product feature map
Document MCP gateway surfaces and customer-reported gaps, grow mcp.yaml and mgmt MCP cells so skipped coverage is honest, and mark the OAuth chat path as covering the chat_completion auto-exec cell
This commit is contained in:
parent
bbc6e3feea
commit
8b888a0666
5 changed files with 847 additions and 38 deletions
|
|
@ -119,14 +119,36 @@ mgmt.<endpoint>.<assertion>
|
|||
mgmt.key.generate.happy_path (surface=ui)
|
||||
```
|
||||
|
||||
MCPs - endpoint features with the protocol op as the variant
|
||||
MCPs - endpoint features with the protocol op as the variant. Feature map and full
|
||||
vocabulary: `coverage_registry/MCP_FEATURES.md`. `auth_family` is how the *client*
|
||||
authenticates to LiteLLM; upstream server auth is an assertion (e.g. `upstream_sigv4`)
|
||||
|
||||
```
|
||||
mcp.<operation>.<auth_family>.<assertion>
|
||||
operation : list_tools | call_tool | list_resources | read_resource | list_prompts | get_prompt
|
||||
operation : list_tools | call_tool | list_resources | read_resource | list_resource_templates
|
||||
| list_prompts | get_prompt | tool_search | tool_call_virtual | test_connection
|
||||
| sampling | elicitation | chat_completion | responses | messages
|
||||
| auth | permission | oauth | toolset | health
|
||||
auth_family : none | api_key | bearer | oauth
|
||||
assertion : succeeds | denied_without_permission
|
||||
assertion : succeeds | denied_without_permission | access_group_scoped | allowed_tools_scoped
|
||||
| disallowed_tools_blocked | params_filtered | toolset_scoped
|
||||
| namespaced_multi_server | partial_on_upstream_fault | progress_forwarded
|
||||
| cost_logged | concurrent_limit | forwards_extra_headers | resolves_user_env_vars
|
||||
| auto_executes_tools | stream_auto_executes_tools | semantic_filter_narrows
|
||||
| enforces_model_access
|
||||
| upstream_static_auth | upstream_oauth2_client_credentials
|
||||
| upstream_oauth2_authorization_code | delegate_auth_upstream | oauth_passthrough
|
||||
| upstream_true_passthrough | upstream_token_exchange | upstream_id_jag
|
||||
| upstream_oauth_delegate | upstream_sigv4
|
||||
| transport_http | transport_sse | transport_stdio | openapi_tools
|
||||
| returns_401_not_500 | returns_429_on_budget | empty_intersection_denies
|
||||
| dangling_grant_errors | stable_id_survives_repoint | per_server_route
|
||||
| sees_newly_added_server | trailing_slash_parity | resource_metadata_public_https
|
||||
| prefix_separator_honored | byok_not_false_unhealthy
|
||||
| unregistered_server_blocked | acting_user_oauth_token
|
||||
e.g. mcp.call_tool.oauth.succeeds
|
||||
mcp.messages.api_key.auto_executes_tools
|
||||
mcp.chat_completion.api_key.auto_executes_tools
|
||||
```
|
||||
|
||||
Reliability & Performance - behavior features (no route; endpoint is exercised_on)
|
||||
|
|
|
|||
301
tests/e2e/coverage_registry/MCP_FEATURES.md
Normal file
301
tests/e2e/coverage_registry/MCP_FEATURES.md
Normal file
|
|
@ -0,0 +1,301 @@
|
|||
# LiteLLM MCP gateway feature map
|
||||
|
||||
Inventory of MCP features in the codebase, mapped to the e2e coverage registry
|
||||
(`mcp.yaml`, plus related `mgmt` / `guardrail` cells). Cross-checked against
|
||||
customer open-MCP issues (anon export, 2026-08).
|
||||
|
||||
Every customer-noticeable behavior should become a registry cell, then a
|
||||
`@pytest.mark.covers(...)` test.
|
||||
|
||||
**Primary code:** `litellm/proxy/_experimental/mcp_server/`
|
||||
**Admin API:** `litellm/proxy/management_endpoints/mcp_management_endpoints.py`
|
||||
**Types:** `litellm/types/mcp.py`, `litellm/types/mcp_server/`
|
||||
**LLM bridges:**
|
||||
- Chat completions: `litellm/responses/mcp/chat_completions_handler.py` (`acompletion_with_mcp`)
|
||||
- Responses API: `litellm/responses/main.py` (`aresponses_api_with_mcp`), `mcp_streaming_iterator.py`
|
||||
- **Messages API:** `litellm/llms/anthropic/experimental_pass_through/messages/mcp_handler.py` (`anthropic_messages_with_mcp`) wired from `messages/handler.py`
|
||||
- Shared expansion: `litellm/responses/mcp/litellm_proxy_mcp_handler.py`
|
||||
**Guardrails:** `guardrail_hooks/mcp_*`, `guardrail_translation/`
|
||||
|
||||
Registry grammar (MCP module):
|
||||
|
||||
```
|
||||
mcp.<operation>.<auth_family>.<assertion>
|
||||
auth_family : none | api_key | bearer | oauth # how the *client* authenticates to LiteLLM
|
||||
```
|
||||
|
||||
`auth_family` is **not** the upstream MCP server auth type. Upstream auth
|
||||
(oauth2, sigv4, basic, …) is expressed in the assertion or operation variant.
|
||||
|
||||
---
|
||||
|
||||
## 1. Protocol operations (MCP server surface)
|
||||
|
||||
Exposed via MCP protocol handlers in `server.py` and REST mirrors in
|
||||
`rest_endpoints.py` (`/tools/list`, `/tools/call`, test helpers). Aggregate
|
||||
route is `/mcp/` (trailing slash matters for some clients); per-server and
|
||||
toolset routes exist via `dynamic_mcp_route` / `toolset_mcp_route`.
|
||||
|
||||
| Feature | Code | Registry cell(s) | E2E today |
|
||||
| --- | --- | --- | --- |
|
||||
| list_tools | `server.py` handle_list_tools; REST `/tools/list` | `mcp.list_tools.{api_key,bearer,oauth,none}.succeeds` | api_key often skipped LIT-5052; oauth skipif Linear session |
|
||||
| call_tool | `server.py` mcp_server_tool_call; REST `/tools/call` | `mcp.call_tool.{api_key,bearer,oauth,none}.succeeds` | same |
|
||||
| list_tools denied without key scope | permission path | `mcp.list_tools.api_key.denied_without_permission` | yes |
|
||||
| call_tool denied without permission | rest/manager | `mcp.call_tool.api_key.denied_without_permission` | skipped LIT-5052 |
|
||||
| list_prompts / get_prompt | `server.py` | `mcp.list_prompts` / `mcp.get_prompt` | missing |
|
||||
| list_resources / read_resource | `server.py` | `mcp.list_resources` / `mcp.read_resource` | missing (**customer pain:** Atlassian / Claude Code ListResources) |
|
||||
| list_resource_templates | `server.py` | `mcp.list_resource_templates.api_key.succeeds` | missing |
|
||||
| Progress on call_tool | `server.py` forward_progress | `mcp.call_tool.api_key.progress_forwarded` | missing |
|
||||
| Multi-server tool namespace / prefix | short_prefix / `MCP_TOOL_PREFIX_SEPARATOR` | `mcp.list_tools.api_key.namespaced_multi_server` | missing |
|
||||
| Toolset tool FQ names + separator | toolset UI/DB + prefix | `mcp.toolset.api_key.prefix_separator_honored` | missing (customer PR #34559) |
|
||||
| Virtual tools: mcp_tool_search / mcp_tool_call | `tool_search.py` | `mcp.tool_search` / `mcp.tool_call_virtual` | missing |
|
||||
| Partial list on upstream failure | `faults/` | `mcp.list_tools.api_key.partial_on_upstream_fault` | missing |
|
||||
| Per-server MCP path (`/mcp/{alias}/…`) | dynamic route | `mcp.list_tools.api_key.per_server_route` | missing (customer open) |
|
||||
| Aggregate `/mcp/` picks up new servers without restart | registry reload | `mcp.list_tools.api_key.sees_newly_added_server` | missing (customer open) |
|
||||
| Trailing slash parity `/mcp` vs `/mcp/` | server routing | `mcp.list_tools.api_key.trailing_slash_parity` | partial historical fix |
|
||||
| REST test connection / test list tools | `rest_endpoints.py` | `mcp.test_connection.api_key.succeeds` | missing |
|
||||
| BYOK health without user token | health + is_byok | `mcp.health.api_key.byok_not_false_unhealthy` | missing (LIT-4896/5136) |
|
||||
|
||||
---
|
||||
|
||||
## 2. Client auth to the gateway (`auth_family`)
|
||||
|
||||
| Family | Meaning | Registry |
|
||||
| --- | --- | --- |
|
||||
| `api_key` | Virtual key / master key | P0 list/call + deny |
|
||||
| `bearer` | Bearer token (incl. OAuth access token as Bearer) | P1 list/call |
|
||||
| `oauth` | Interactive OAuth2; gateway-managed per-user tokens | P1 + chat/messages bridges |
|
||||
| `none` | Anonymous / public / `delegate_auth_to_upstream` | P1 list/call |
|
||||
|
||||
| Related behavior | Code | Registry | Customer issue |
|
||||
| --- | --- | --- | --- |
|
||||
| Auth fail → 401 + WWW-Authenticate (not 500) | `extract_mcp_auth_context`, ProxyException mapping | `mcp.auth.api_key.returns_401_not_500` | #2 (PR #31011 claimed) |
|
||||
| Budget exceeded on /mcp → 429 | same path | `mcp.auth.api_key.returns_429_on_budget` | #2 |
|
||||
| Pre-emptive 401 for unauthenticated OAuth servers | `server.py` | oauth cells | #15 |
|
||||
| Stateful session auth contexts + cap | `server.py` session managers | P2 | — |
|
||||
|
||||
---
|
||||
|
||||
## 3. Upstream server auth (`MCPAuth`)
|
||||
|
||||
| `auth_type` / flag | Behavior | Cell | E2E / issues |
|
||||
| --- | --- | --- | --- |
|
||||
| `none` | No upstream auth | generic succeeds | partial |
|
||||
| static (`api_key`, `bearer_token`, `basic`, `authorization`, `token`) | Inject static headers | `upstream_static_auth` | missing dedicated |
|
||||
| `oauth2` + `authorization_code` | Per-user vault | oauth cells | Linear e2e; UI flaky (#15) |
|
||||
| `oauth2` + `client_credentials` | M2M | `upstream_oauth2_client_credentials` | missing |
|
||||
| `delegate_auth_to_upstream` | Client PKCE with upstream | `delegate_auth_upstream` | missing |
|
||||
| `oauth_passthrough` | Proxy metadata + 401 challenges | `oauth_passthrough` | unit |
|
||||
| `true_passthrough` | Forward client Authorization | `upstream_true_passthrough` | missing |
|
||||
| `oauth2_token_exchange` / OBO | RFC 8693 / entra_obo | `upstream_token_exchange` | missing |
|
||||
| `oauth2_id_jag` | ID-JAG | `upstream_id_jag` | missing |
|
||||
| `oauth_delegate` | Bridge / SSO assertion | `upstream_oauth_delegate` | missing |
|
||||
| `aws_sigv4` | SigV4 (AgentCore etc.) | `upstream_sigv4` | unit; UI onboard flaky (#18) |
|
||||
| Session handshake (Databricks-style mint) | **not implemented** as full handshake | product gap | #5 open |
|
||||
| Resource metadata `https` behind TLS terminator | `oauth_utils` X-Forwarded-Proto / public base | `mcp.oauth.api_key.resource_metadata_public_https` | #16 open |
|
||||
|
||||
---
|
||||
|
||||
## 4. Transports and MCP spec versions
|
||||
|
||||
| Feature | Values | Notes / issues |
|
||||
| --- | --- | --- |
|
||||
| Transport | `sse`, `http`, `stdio` | stdio poorly e2e'd; customer #20 |
|
||||
| Spec version | 2024-11-05, 2025-03-26, 2025-06-18 | |
|
||||
| OpenAPI → tools | `openapi_to_mcp_generator.py` | backend exists; **UI missing** (#20) |
|
||||
| gRPC transport | **not supported** | FR #23 |
|
||||
| Admin transport allowlist | **not supported** | FR #22 |
|
||||
|
||||
---
|
||||
|
||||
## 5. Permission and multi-tenant safety
|
||||
|
||||
| Feature | Code | Registry | Customer / e2e |
|
||||
| --- | --- | --- | --- |
|
||||
| Key `object_permission.mcp_servers` | manager + auth_mcp | deny cells | partial |
|
||||
| Empty key∩team (or similar) intersection = **deny-all** | `user_api_key_auth_mcp.py` hierarchy | `mcp.list_tools.api_key.empty_intersection_denies` | **#3 open (A2A same class; MCP too)** |
|
||||
| MCP access groups | manager + e2e | `access_group_scoped` | list yes; call missing |
|
||||
| Key access group grants beyond team list | product gap / disputed | `mcp.list_tools.api_key.key_access_group_beyond_team` | #4 open |
|
||||
| Model-level MCP scoping | **not supported** | FR cell if added | #4 FR |
|
||||
| `allowed_tools` / `disallowed_tools` | MCPServer | allowed/disallowed cells | missing e2e |
|
||||
| Toolsets → permission expansion | toolset_db | toolset_scoped | missing; prefix bug #6 |
|
||||
| `allowed_params` | MCPServer | params_filtered | missing |
|
||||
| `allow_all_keys` | MCPServer | document + test | missing |
|
||||
| User-only / exclude service accounts | **FR** | `mcp.call_tool.api_key.user_scoped_only` when built | #8 open |
|
||||
| `require_key_mcp_access_defined` | auth hierarchy | cell | missing e2e |
|
||||
| End-user MCP permission guardrail | mcp_end_user_permission | guardrail | missing e2e |
|
||||
| Team-scoped server list | management | mgmt | missing |
|
||||
| Gateway allowlist: only registered MCPs reachable | product intent | `mcp.call_tool.api_key.unregistered_server_blocked` | #17 FR |
|
||||
| Invalid server name / tool prefix validation | utils / UI | `mcp.server.api_key.rejects_invalid_prefix` | #17 |
|
||||
|
||||
### Stable identity (config.yaml servers)
|
||||
|
||||
| Feature | Code | Issue |
|
||||
| --- | --- | --- |
|
||||
| `server_id` = hash(name, url, transport, auth_type, alias) | `_generate_stable_server_id` | **#1 OPEN:** rename/repoint URL changes ID; grants silently dangle |
|
||||
| Stable id OR migrate grants + loud dangling-grant error | needed | cell: `mcp.permission.api_key.dangling_grant_errors` + identity migration |
|
||||
|
||||
---
|
||||
|
||||
## 6. Admin / management API (`/v1/mcp/...`)
|
||||
|
||||
Registry module often `mgmt`, not `mcp`.
|
||||
|
||||
| Feature | Registry | Customer |
|
||||
| --- | --- | --- |
|
||||
| Create / update / delete / list / health | mgmt cells | UI onboard #18; health BYOK #12 |
|
||||
| Non-admin register + approve/reject | mgmt approve/reject | |
|
||||
| Temporary / session MCP | gap | |
|
||||
| User OAuth credential CRUD | mgmt + docs gap | **#9:** deposit API exists, docs + acting-user resolution unclear |
|
||||
| User env vars CRUD | mgmt | |
|
||||
| Toolsets CRUD | mgmt | prefix #6 |
|
||||
| Make public / discover / registry | gap | local registry FR #19 |
|
||||
| Multi-pod: UI save visible on all processes | reliability / pub-sub | **#7** (v1.96 Redis push claimed) |
|
||||
| Key update must not wipe MCP toolsets/servers | key management | **#10 OPEN** |
|
||||
| Remove stale deleted MCP IDs from key (UI) | key UI | **#11 LIT-3278** |
|
||||
| OpenAPI converter in UI | UI | #20 |
|
||||
| First-party LiteLLM admin MCP server | **FR product** | #21 |
|
||||
|
||||
---
|
||||
|
||||
## 7. LLM bridges (MCP tools inside model APIs)
|
||||
|
||||
All three share `LiteLLM_Proxy_MCP_Handler` for `litellm_proxy` / `litellm_proxy/mcp/...` tool references: expand tools under the caller's credentials, optional auto-execute loop.
|
||||
|
||||
| Surface | Entry | Registry | E2E |
|
||||
| --- | --- | --- | --- |
|
||||
| **Chat completions** `/v1/chat/completions` | `acompletion_with_mcp` via `main.py` | `mcp.chat_completion.{api_key,oauth}.auto_executes_tools` | oauth Linear (skipif); api_key missing |
|
||||
| **Responses** `/v1/responses` | `aresponses_api_with_mcp` | `mcp.responses.api_key.auto_executes_tools` | missing |
|
||||
| **Messages** `/v1/messages` | `anthropic_messages_with_mcp` via experimental pass-through handler | `mcp.messages.{api_key,oauth}.auto_executes_tools` | **was missing from map; no e2e** |
|
||||
| Semantic tool filter on chat tools | `semantic_tool_filter` + hook | `mcp.chat_completion.api_key.semantic_filter_narrows` | missing |
|
||||
| Provider-native Anthropic `mcp_servers` tool | `AnthropicMcpServerTool` / beta header | **not gateway**; provider-side MCP | separate from litellm_proxy bridge |
|
||||
| Playground auto-execution | UI → same bridges | exercises chat/messages paths | customer #12 |
|
||||
|
||||
Claude Code / Desktop often use **messages** + OAuth + resources; that is why resources + messages bridge + OAuth metadata are P0 for gateway maturity.
|
||||
|
||||
---
|
||||
|
||||
## 8. Sampling and elicitation
|
||||
|
||||
| Feature | Flag / code | Registry |
|
||||
| --- | --- | --- |
|
||||
| Sampling createMessage → completion | `allow_sampling`, `sampling_handler.py` | `mcp.sampling.api_key.succeeds` |
|
||||
| Sampling model access + budget | same | `mcp.sampling.api_key.enforces_model_access` |
|
||||
| Elicitation relay | `allow_elicitation` | `mcp.elicitation.api_key.succeeds` |
|
||||
|
||||
---
|
||||
|
||||
## 9. Cost, spend, concurrency, headers
|
||||
|
||||
| Feature | Registry | Customer |
|
||||
| --- | --- | --- |
|
||||
| Per-server / per-tool cost on call_tool | `mcp.call_tool.api_key.cost_logged` | |
|
||||
| list_tools spend log | `mcp.list_tools.api_key.cost_logged` | |
|
||||
| UI MCP Server Activity: tool invocations not tokens | product/UI | **#13 LIT-4897** (tokens always 0) |
|
||||
| max_concurrent_requests | `concurrent_limit` | |
|
||||
| extra_headers / static_headers / user env vars | forward / resolve cells | |
|
||||
| Timeout per server | reliability-adjacent | |
|
||||
|
||||
---
|
||||
|
||||
## 10. Guardrails on MCP
|
||||
|
||||
| Feature | Registry | E2E |
|
||||
| --- | --- | --- |
|
||||
| Content filter `pre_mcp_call` | `guardrail.litellm_content_filter.pre_mcp_call.blocks` | skipped LIT-5052 |
|
||||
| MCP security hook | `guardrail.mcp_security.pre_call.blocks` | missing |
|
||||
| MCP JWT signer | gap | unit |
|
||||
| End-user permission | gap | missing |
|
||||
|
||||
---
|
||||
|
||||
## 11. Config load / multi-pod / lifecycle
|
||||
|
||||
| Feature | Notes | Issue |
|
||||
| --- | --- | --- |
|
||||
| YAML `mcp_servers` load | manager | #1 identity |
|
||||
| DB store + reload / Redis pub-sub config sync | proxy config sync | #7 multi-process |
|
||||
| Temporary server Redis cache | management | |
|
||||
| Discovery well-known OAuth | discoverable + byok | #15/#16 |
|
||||
| DCR | gateway_dcr_flow | |
|
||||
| Operator open servers / allow_all union | manager | |
|
||||
|
||||
---
|
||||
|
||||
## 12. Customer open-issue → product/registry map
|
||||
|
||||
Source: open MCP issues export (26 rows, anon). Status abbreviated.
|
||||
|
||||
| # | Theme | Product status | Registry / e2e action |
|
||||
| --- | --- | --- | --- |
|
||||
| 1 | Stable server_id vs rename/URL; dangling grants silent | OPEN | `mcp.permission.api_key.stable_id_survives_url_change` or migration + `dangling_grant_errors` |
|
||||
| 2 | /mcp auth 500 vs 401/429 | Claimed fixed #31011 | `returns_401_not_500`, `returns_429_on_budget` — regression e2e |
|
||||
| 3 | Empty permission intersection allow-all (A2A; same class MCP) | OPEN blocking | `empty_intersection_denies` (MCP + a2a suite) |
|
||||
| 4 | Model-level MCP scope; key AG beyond team | FR / open | FR cells when designed |
|
||||
| 5 | Central MCP inherit without per-key; Databricks handshake | OPEN | product; handshake not in codebase |
|
||||
| 6 | Toolset prefix separator | Claimed #34559 | `toolset.prefix_separator_honored` |
|
||||
| 7 | Multi-process UI save not sticky | Claimed v1.96 Redis | multi-pod e2e after release |
|
||||
| 8 | User-only MCP (exclude service accounts) | FR | when built |
|
||||
| 9 | OBO deposit API + acting user for agents | OPEN docs/behavior | mgmt credential + `call_tool` with acting user |
|
||||
| 10 | Key budget edit wipes MCP toolset | OPEN | mgmt key update regression |
|
||||
| 11 | Stale MCP IDs on key UI | LIT-3278 | UI/mgmt e2e |
|
||||
| 12 | BYOK health false unhealthy | LIT-4896/5136 | `byok_not_false_unhealthy` |
|
||||
| 13 | MCP activity Total Tokens = 0 | LIT-4897 | UI metric |
|
||||
| 14 | Per-server routes; aggregate tool list stale | OPEN | `per_server_route`, `sees_newly_added_server` |
|
||||
| 15 | OAuth E2E UI + token forward + clients | OPEN / churn | oauth + messages/chat bridges |
|
||||
| 16 | Resource metadata http behind TLS | OPEN | `resource_metadata_public_https` |
|
||||
| 17 | Gateway allowlist unregistered MCPs; invalid names | FR | allowlist + validation cells |
|
||||
| 18 | UI register / playground tools empty | OPEN/stale | mgmt + playground |
|
||||
| 19 | Local/dev MCP registry governance | FR | product |
|
||||
| 20 | OpenAPI UI; stdio/oauth maturity | OPEN / lost deal | openapi + transports |
|
||||
| 21 | First-party LiteLLM admin MCP | FR | product |
|
||||
| 22 | Transport allowlist setting | FR | product |
|
||||
| 23 | gRPC transport | FR low | product |
|
||||
| 24 | Resources + Atlassian OAuth via Claude Code | OPEN recurring | **resources + messages + oauth** cells |
|
||||
| 25–26 | Competitive losses (MCP immaturity) | LOST | treat as quality bar, not single cells |
|
||||
|
||||
---
|
||||
|
||||
## Coverage snapshot
|
||||
|
||||
| Bucket | In product | In registry | Live e2e |
|
||||
| --- | --- | --- | --- |
|
||||
| Core list/call api_key | yes | yes | blocked LIT-5052 |
|
||||
| Deny / access groups | yes | yes | partial |
|
||||
| bearer / oauth / none | yes | yes | oauth skipif |
|
||||
| prompts / **resources** | yes | yes | **none** (Atlassian #24) |
|
||||
| LLM bridges chat | yes | yes | oauth only |
|
||||
| LLM bridges **responses** | yes | yes | none |
|
||||
| LLM bridges **messages** | yes | **added** | **none** |
|
||||
| Auth status codes | claimed | **added** | need regression |
|
||||
| Stable id / dangling grants | bug | **added** | need |
|
||||
| Empty intersection deny | bug | **added** | need |
|
||||
| Toolsets / multi-pod / BYOK health | partial | partial | thin |
|
||||
| Model-level scope, user-only mode, gRPC, admin MCP | FR / missing | FR notes only | — |
|
||||
|
||||
Collector after registry expansion: **MCPs ~2/N live** until tests land; denominator is intentionally honest.
|
||||
|
||||
---
|
||||
|
||||
## Priority for new e2e (reliability / fewer regressions)
|
||||
|
||||
1. **P0** Unskip LIT-5052 (Datadog list/call/deny/guardrail).
|
||||
2. **P0** `/mcp` auth status codes 401/429 regression (#2).
|
||||
3. **P0** Empty permission intersection denies (#3).
|
||||
4. **P0** **messages** bridge auto-execute tools (Claude Code path) + **list_resources**.
|
||||
5. **P0** OAuth resource metadata public https (#16) + token used on tool call.
|
||||
6. **P1** Stable server_id / dangling grant loud failure (#1).
|
||||
7. **P1** allowed/disallowed tools; call_tool access groups.
|
||||
8. **P1** chat + responses auto-exec with api_key.
|
||||
9. **P1** Per-server route + sees newly added server (#14).
|
||||
10. **P1** Multi-pod MCP edit propagates (#7).
|
||||
11. **P1** Key update does not wipe MCP grants (#10).
|
||||
12. **P2** Toolset prefix, BYOK health UX, cost/invocation metrics, sampling.
|
||||
|
||||
---
|
||||
|
||||
## Related docs
|
||||
|
||||
- E2E MCP suite: `tests/e2e/CLAUDE.md` (real Datadog; Linear for OAuth)
|
||||
- Registry: `tests/e2e/coverage_registry/mcp.yaml`
|
||||
- MCP internal note: `litellm/proxy/_experimental/mcp_server/CLAUDE.md`
|
||||
|
|
@ -1,12 +1,19 @@
|
|||
# MCP module. Grounded in litellm/proxy/_experimental/mcp_server/. See tests/e2e/CLAUDE.md for the grammar.
|
||||
# MCP module. Grounded in litellm/proxy/_experimental/mcp_server/.
|
||||
# Feature map: tests/e2e/coverage_registry/MCP_FEATURES.md
|
||||
# Grammar: mcp.<operation>.<auth_family>.<assertion>
|
||||
# auth_family = how the *client* authenticates to LiteLLM (none|api_key|bearer|oauth)
|
||||
# upstream server auth is encoded in the assertion / operation variant, not auth_family
|
||||
# See tests/e2e/CLAUDE.md for the full grammar.
|
||||
|
||||
# --- Protocol: list_tools / call_tool by client auth family ---
|
||||
- id: mcp.list_tools.api_key.succeeds
|
||||
module: mcp
|
||||
tier: P0
|
||||
operation: list_tools
|
||||
auth_family: api_key
|
||||
assertions: [succeeds]
|
||||
source: "server.py:637"
|
||||
rationale: Core operation; most common auth path; high usage
|
||||
source: "server.py:handle_list_tools"
|
||||
rationale: "Core operation; most common auth path; high usage"
|
||||
- id: mcp.list_tools.api_key.access_group_scoped
|
||||
module: mcp
|
||||
tier: P1
|
||||
|
|
@ -14,55 +21,55 @@
|
|||
auth_family: api_key
|
||||
assertions: [access_group_scoped]
|
||||
source: "test_mcp_access_group_e2e.py"
|
||||
rationale: "A key granted an MCP access group sees the tagged server's tools; a key with a different group does not. Access-group-scoped tool selection at key creation"
|
||||
rationale: "A key granted an MCP access group sees the tagged server's tools; a key with a different group does not"
|
||||
- id: mcp.list_tools.api_key.denied_without_permission
|
||||
module: mcp
|
||||
tier: P0
|
||||
operation: list_tools
|
||||
auth_family: api_key
|
||||
assertions: [denied_without_permission]
|
||||
source: "mcp_server_manager.py:1409"
|
||||
rationale: Permission guard is high blast-radius; multi-tenant safety
|
||||
source: "mcp_server_manager.py:get_allowed_mcp_servers"
|
||||
rationale: "Permission guard is high blast-radius; multi-tenant safety"
|
||||
- id: mcp.call_tool.api_key.succeeds
|
||||
module: mcp
|
||||
tier: P0
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [succeeds]
|
||||
source: "server.py:849"
|
||||
rationale: Primary operation; customer-critical; high usage
|
||||
source: "server.py:mcp_server_tool_call"
|
||||
rationale: "Primary operation; customer-critical; high usage"
|
||||
- id: mcp.call_tool.api_key.denied_without_permission
|
||||
module: mcp
|
||||
tier: P0
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [denied_without_permission]
|
||||
source: "rest_endpoints.py:305-386"
|
||||
rationale: Tool-level permission guard; multi-tenant safety
|
||||
source: "rest_endpoints.py:call_tool_rest_api"
|
||||
rationale: "Tool-level permission guard; multi-tenant safety"
|
||||
- id: mcp.list_tools.bearer.succeeds
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: list_tools
|
||||
auth_family: bearer
|
||||
assertions: [succeeds]
|
||||
source: "server.py:662"
|
||||
rationale: OAuth/bearer token flow; upstream delegation
|
||||
source: "server.py:handle_list_tools"
|
||||
rationale: "OAuth/bearer token flow; upstream delegation"
|
||||
- id: mcp.call_tool.bearer.succeeds
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: bearer
|
||||
assertions: [succeeds]
|
||||
source: "server.py:886"
|
||||
rationale: Bearer token forwarding for tool invocation
|
||||
source: "server.py:mcp_server_tool_call"
|
||||
rationale: "Bearer token forwarding for tool invocation"
|
||||
- id: mcp.list_tools.oauth.succeeds
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: list_tools
|
||||
auth_family: oauth
|
||||
assertions: [succeeds]
|
||||
source: "rest_endpoints.py:138-188"
|
||||
rationale: Interactive OAuth2 flow; live token management
|
||||
source: "rest_endpoints.py; outbound_credentials/per_user_oauth_store"
|
||||
rationale: "Interactive OAuth2 flow; live token management"
|
||||
- id: mcp.call_tool.oauth.succeeds
|
||||
module: mcp
|
||||
tier: P1
|
||||
|
|
@ -70,52 +77,511 @@
|
|||
auth_family: oauth
|
||||
assertions: [succeeds]
|
||||
source: "db.py user_oauth_credential lookup"
|
||||
rationale: OAuth2 token passthrough; per-user credential storage
|
||||
rationale: "OAuth2 token passthrough; per-user credential storage"
|
||||
- id: mcp.list_tools.none.succeeds
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: list_tools
|
||||
auth_family: none
|
||||
assertions: [succeeds]
|
||||
source: "mcp_server_manager.py:1485-1492"
|
||||
rationale: Public/anonymous servers; delegate_auth_to_upstream
|
||||
source: "mcp_server_manager.py; delegate_auth_to_upstream"
|
||||
rationale: "Public/anonymous servers; delegate_auth_to_upstream"
|
||||
- id: mcp.call_tool.none.succeeds
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: none
|
||||
assertions: [succeeds]
|
||||
source: "rest_endpoints.py:305-334"
|
||||
rationale: No upstream auth required; demo servers
|
||||
source: "rest_endpoints.py"
|
||||
rationale: "No LiteLLM key required; demo or upstream-delegated servers"
|
||||
|
||||
# --- Protocol: prompts / resources ---
|
||||
- id: mcp.get_prompt.api_key.succeeds
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: get_prompt
|
||||
auth_family: api_key
|
||||
assertions: [succeeds]
|
||||
source: "server.py:1042"
|
||||
rationale: Prompt op; same auth stack as tools
|
||||
source: "server.py:get_prompt"
|
||||
rationale: "Prompt op; same auth stack as tools"
|
||||
- id: mcp.read_resource.api_key.succeeds
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: read_resource
|
||||
auth_family: api_key
|
||||
assertions: [succeeds]
|
||||
source: "server.py:1177"
|
||||
rationale: Resource op; same permission model as tools
|
||||
source: "server.py:read_resource"
|
||||
rationale: "Resource op; same permission model as tools"
|
||||
- id: mcp.list_prompts.api_key.succeeds
|
||||
module: mcp
|
||||
tier: P2
|
||||
operation: list_prompts
|
||||
auth_family: api_key
|
||||
assertions: [succeeds]
|
||||
source: "server.py:993"
|
||||
rationale: Smoke-level; same auth stack as list_tools
|
||||
source: "server.py:list_prompts"
|
||||
rationale: "Smoke-level; same auth stack as list_tools"
|
||||
- id: mcp.list_resources.api_key.succeeds
|
||||
module: mcp
|
||||
tier: P2
|
||||
operation: list_resources
|
||||
auth_family: api_key
|
||||
assertions: [succeeds]
|
||||
source: "server.py:1089"
|
||||
rationale: Smoke; rarely used; same auth model as tools
|
||||
source: "server.py:list_resources"
|
||||
rationale: "Smoke; rarely used; same auth model as tools"
|
||||
- id: mcp.list_resource_templates.api_key.succeeds
|
||||
module: mcp
|
||||
tier: P2
|
||||
operation: list_resource_templates
|
||||
auth_family: api_key
|
||||
assertions: [succeeds]
|
||||
source: "server.py:list_resource_templates"
|
||||
rationale: "Resource templates; same permission model as list_resources"
|
||||
|
||||
# --- Multi-tenant tool selection (server + key policy) ---
|
||||
- id: mcp.call_tool.api_key.access_group_scoped
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [access_group_scoped]
|
||||
source: "mcp_server_manager.py:get_allowed_mcp_servers"
|
||||
rationale: "Call path must honor the same MCP access groups as list_tools"
|
||||
- id: mcp.list_tools.api_key.allowed_tools_scoped
|
||||
module: mcp
|
||||
tier: P0
|
||||
operation: list_tools
|
||||
auth_family: api_key
|
||||
assertions: [allowed_tools_scoped]
|
||||
source: "types/mcp_server/mcp_server_manager.py:MCPServer.allowed_tools"
|
||||
rationale: "Server allowed_tools must hide disallowed tools from list_tools"
|
||||
- id: mcp.call_tool.api_key.allowed_tools_scoped
|
||||
module: mcp
|
||||
tier: P0
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [allowed_tools_scoped]
|
||||
source: "mcp_server_manager.py"
|
||||
rationale: "Calling a tool outside allowed_tools must be denied"
|
||||
- id: mcp.call_tool.api_key.disallowed_tools_blocked
|
||||
module: mcp
|
||||
tier: P0
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [disallowed_tools_blocked]
|
||||
source: "types/mcp_server/mcp_server_manager.py:MCPServer.disallowed_tools"
|
||||
rationale: "Explicit disallowed_tools block must win even if listed upstream"
|
||||
- id: mcp.call_tool.api_key.params_filtered
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [params_filtered]
|
||||
source: "types/mcp_server/mcp_server_manager.py:MCPServer.allowed_params"
|
||||
rationale: "allowed_params strips or rejects tool arguments outside the allow-list"
|
||||
- id: mcp.call_tool.api_key.toolset_scoped
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [toolset_scoped]
|
||||
source: "toolset_db.py; resolve_toolset_tool_permissions"
|
||||
rationale: "Key scoped to a toolset only sees/calls that toolset's tools"
|
||||
|
||||
# --- Namespacing, virtual tools, partial failure ---
|
||||
- id: mcp.list_tools.api_key.namespaced_multi_server
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: list_tools
|
||||
auth_family: api_key
|
||||
assertions: [namespaced_multi_server]
|
||||
source: "mcp_server_manager.py:_assign_unique_short_prefix"
|
||||
rationale: "Two servers' tools remain distinguishable (prefix/short_prefix) without collision"
|
||||
- id: mcp.tool_search.api_key.succeeds
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: tool_search
|
||||
auth_family: api_key
|
||||
assertions: [succeeds]
|
||||
source: "tool_search.py:handle_mcp_tool_search"
|
||||
rationale: "Virtual mcp_tool_search tool returns ranked tools when mcp_tool_search_enabled"
|
||||
- id: mcp.tool_call_virtual.api_key.succeeds
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: tool_call_virtual
|
||||
auth_family: api_key
|
||||
assertions: [succeeds]
|
||||
source: "tool_search.py:handle_mcp_tool_call"
|
||||
rationale: "Virtual mcp_tool_call invokes a real upstream tool by name after search"
|
||||
- id: mcp.list_tools.api_key.partial_on_upstream_fault
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: list_tools
|
||||
auth_family: api_key
|
||||
assertions: [partial_on_upstream_fault]
|
||||
source: "faults/list_outcomes.py"
|
||||
rationale: "One bad upstream must not blank the entire multi-server tool list"
|
||||
- id: mcp.call_tool.api_key.progress_forwarded
|
||||
module: mcp
|
||||
tier: P2
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [progress_forwarded]
|
||||
source: "server.py:forward_progress"
|
||||
rationale: "Upstream progress notifications reach the MCP client during long tool calls"
|
||||
|
||||
# --- Upstream auth variants (client still api_key unless noted) ---
|
||||
- id: mcp.call_tool.api_key.upstream_static_auth
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [upstream_static_auth]
|
||||
source: "types/mcp.py:MCPAuth api_key|bearer_token|basic|authorization|token"
|
||||
rationale: "Gateway injects static upstream credentials so call_tool succeeds"
|
||||
- id: mcp.call_tool.api_key.upstream_oauth2_client_credentials
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [upstream_oauth2_client_credentials]
|
||||
source: "MCPServer.has_client_credentials; outbound_credentials/client_credentials.py"
|
||||
rationale: "M2M oauth2_flow=client_credentials fetches and uses a client token"
|
||||
- id: mcp.call_tool.api_key.upstream_oauth2_authorization_code
|
||||
module: mcp
|
||||
tier: P0
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [upstream_oauth2_authorization_code]
|
||||
source: "outbound_credentials/per_user_oauth_store.py"
|
||||
rationale: "Per-user vaulted authorization_code token is used on call_tool after consent"
|
||||
- id: mcp.list_tools.none.delegate_auth_upstream
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: list_tools
|
||||
auth_family: none
|
||||
assertions: [delegate_auth_upstream]
|
||||
source: "MCPServer.delegate_auth_to_upstream"
|
||||
rationale: "oauth2 + delegate_auth_to_upstream allows anonymous PKCE against upstream"
|
||||
- id: mcp.call_tool.none.oauth_passthrough
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: none
|
||||
assertions: [oauth_passthrough]
|
||||
source: "MCPServer.oauth_passthrough"
|
||||
rationale: "Non-oauth2 servers with oauth_passthrough proxy metadata and 401 challenges"
|
||||
- id: mcp.call_tool.api_key.upstream_true_passthrough
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [upstream_true_passthrough]
|
||||
source: "types/mcp.py:MCPAuth.true_passthrough"
|
||||
rationale: "Client Authorization is forwarded unchanged to upstream"
|
||||
- id: mcp.call_tool.api_key.upstream_token_exchange
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [upstream_token_exchange]
|
||||
source: "types/mcp.py:MCPAuth.oauth2_token_exchange; outbound_credentials/token_exchanger.py"
|
||||
rationale: "OBO / RFC 8693 (or entra_obo) exchange yields a usable upstream token"
|
||||
- id: mcp.call_tool.api_key.upstream_id_jag
|
||||
module: mcp
|
||||
tier: P2
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [upstream_id_jag]
|
||||
source: "types/mcp.py:MCPAuth.oauth2_id_jag"
|
||||
rationale: "ID-JAG two-leg exchange issues a resource token for call_tool"
|
||||
- id: mcp.call_tool.api_key.upstream_oauth_delegate
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [upstream_oauth_delegate]
|
||||
source: "types/mcp.py:MCPAuth.oauth_delegate; bridge_token_flow.py"
|
||||
rationale: "SSO / bridge assertion path obtains upstream credentials"
|
||||
- id: mcp.call_tool.api_key.upstream_sigv4
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [upstream_sigv4]
|
||||
source: "types/mcp.py:MCPAuth.aws_sigv4"
|
||||
rationale: "SigV4-signed upstream MCP calls (e.g. Bedrock AgentCore)"
|
||||
|
||||
# --- Transports / OpenAPI tools ---
|
||||
- id: mcp.call_tool.api_key.transport_http
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [transport_http]
|
||||
source: "types/mcp.py:MCPTransport.http"
|
||||
rationale: "Streamable HTTP transport reaches a real upstream"
|
||||
- id: mcp.call_tool.api_key.transport_sse
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [transport_sse]
|
||||
source: "sse_transport.py; types/mcp.py:MCPTransport.sse"
|
||||
rationale: "SSE transport list+call works end-to-end"
|
||||
- id: mcp.call_tool.api_key.transport_stdio
|
||||
module: mcp
|
||||
tier: P2
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [transport_stdio]
|
||||
source: "types/mcp.py:MCPTransport.stdio"
|
||||
rationale: "Local stdio MCP server (command/args/env) is invocable through the gateway"
|
||||
- id: mcp.call_tool.api_key.openapi_tools
|
||||
module: mcp
|
||||
tier: P2
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [openapi_tools]
|
||||
source: "openapi_to_mcp_generator.py"
|
||||
rationale: "OpenAPI spec_path servers expose operations as tools and execute them"
|
||||
|
||||
# --- Cost, concurrency, headers, env ---
|
||||
- id: mcp.call_tool.api_key.cost_logged
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [cost_logged]
|
||||
source: "cost_calculator.py:MCPCostCalculator"
|
||||
rationale: "Spend logs record non-zero (or configured) cost for an MCP tool call"
|
||||
- id: mcp.list_tools.api_key.cost_logged
|
||||
module: mcp
|
||||
tier: P2
|
||||
operation: list_tools
|
||||
auth_family: api_key
|
||||
assertions: [cost_logged]
|
||||
source: "server.py:_get_tools_from_mcp_servers log_list_tools_to_spendlogs"
|
||||
rationale: "list_tools can write a spend row when logging is enabled"
|
||||
- id: mcp.call_tool.api_key.concurrent_limit
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [concurrent_limit]
|
||||
source: "mcp_server_manager.py:_get_call_semaphore"
|
||||
rationale: "max_concurrent_requests queues excess calls instead of unbounded fan-out"
|
||||
- id: mcp.call_tool.api_key.forwards_extra_headers
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [forwards_extra_headers]
|
||||
source: "MCPServer.extra_headers"
|
||||
rationale: "Client headers listed in extra_headers are forwarded upstream"
|
||||
- id: mcp.call_tool.api_key.resolves_user_env_vars
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [resolves_user_env_vars]
|
||||
source: "MCPServer.env_vars; management store_mcp_user_env_vars"
|
||||
rationale: "Per-user env vars interpolate into static_headers for the call"
|
||||
- id: mcp.test_connection.api_key.succeeds
|
||||
module: mcp
|
||||
tier: P2
|
||||
operation: test_connection
|
||||
auth_family: api_key
|
||||
assertions: [succeeds]
|
||||
source: "rest_endpoints.py:/test/connection"
|
||||
rationale: "Pre-save connection test against a candidate server config succeeds"
|
||||
|
||||
# --- Sampling / elicitation ---
|
||||
- id: mcp.sampling.api_key.succeeds
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: sampling
|
||||
auth_family: api_key
|
||||
assertions: [succeeds]
|
||||
source: "sampling_handler.py:handle_sampling_create_message"
|
||||
rationale: "allow_sampling servers can createMessage through the proxy LLM path"
|
||||
- id: mcp.sampling.api_key.enforces_model_access
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: sampling
|
||||
auth_family: api_key
|
||||
assertions: [enforces_model_access]
|
||||
source: "sampling_handler.py:_check_model_access"
|
||||
rationale: "Sampling refuses models the key cannot use"
|
||||
- id: mcp.elicitation.api_key.succeeds
|
||||
module: mcp
|
||||
tier: P2
|
||||
operation: elicitation
|
||||
auth_family: api_key
|
||||
assertions: [succeeds]
|
||||
source: "elicitation_handler.py:handle_elicitation_request"
|
||||
rationale: "allow_elicitation relays elicitation requests to the downstream client"
|
||||
|
||||
# --- LLM bridges (chat / responses / messages) ---
|
||||
# Shared expansion: responses/mcp/litellm_proxy_mcp_handler.py (litellm_proxy tool refs)
|
||||
- id: mcp.chat_completion.api_key.auto_executes_tools
|
||||
module: mcp
|
||||
tier: P0
|
||||
operation: chat_completion
|
||||
auth_family: api_key
|
||||
assertions: [auto_executes_tools]
|
||||
source: "responses/mcp/chat_completions_handler.py:acompletion_with_mcp"
|
||||
rationale: "/chat/completions with MCP tools lists, calls, and returns a final answer"
|
||||
- id: mcp.chat_completion.oauth.auto_executes_tools
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: chat_completion
|
||||
auth_family: oauth
|
||||
assertions: [auto_executes_tools]
|
||||
source: "test_mcp_chat_completion_oauth_e2e.py"
|
||||
rationale: "Gateway-managed OAuth: chat completion lists and executes tools with vaulted user token"
|
||||
- id: mcp.chat_completion.api_key.semantic_filter_narrows
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: chat_completion
|
||||
auth_family: api_key
|
||||
assertions: [semantic_filter_narrows]
|
||||
source: "semantic_tool_filter.py; hooks/mcp_semantic_filter"
|
||||
rationale: "Semantic filter reduces the tool set offered to the model without dropping needed tools"
|
||||
- id: mcp.responses.api_key.auto_executes_tools
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: responses
|
||||
auth_family: api_key
|
||||
assertions: [auto_executes_tools]
|
||||
source: "responses/main.py:aresponses_api_with_mcp; responses/mcp/mcp_streaming_iterator.py"
|
||||
rationale: "Responses API streams list_tools and call events and auto-executes MCP tools"
|
||||
- id: mcp.messages.api_key.auto_executes_tools
|
||||
module: mcp
|
||||
tier: P0
|
||||
operation: messages
|
||||
auth_family: api_key
|
||||
assertions: [auto_executes_tools]
|
||||
source: "llms/anthropic/experimental_pass_through/messages/mcp_handler.py:anthropic_messages_with_mcp"
|
||||
rationale: "/v1/messages expands litellm_proxy MCP tools, runs the tool_use loop, and returns a final Anthropic message (Claude Code path)"
|
||||
- id: mcp.messages.oauth.auto_executes_tools
|
||||
module: mcp
|
||||
tier: P0
|
||||
operation: messages
|
||||
auth_family: oauth
|
||||
assertions: [auto_executes_tools]
|
||||
source: "llms/anthropic/experimental_pass_through/messages/mcp_handler.py:anthropic_messages_with_mcp"
|
||||
rationale: "Messages bridge uses vaulted per-user OAuth token for list+call under Claude Code / Desktop OAuth clients"
|
||||
- id: mcp.messages.api_key.stream_auto_executes_tools
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: messages
|
||||
auth_family: api_key
|
||||
assertions: [stream_auto_executes_tools]
|
||||
source: "llms/anthropic/experimental_pass_through/messages/mcp_handler.py"
|
||||
rationale: "Streaming /v1/messages still expands MCP tools and completes the tool loop"
|
||||
|
||||
# --- Gateway reliability / multi-tenant edge cases (customer open issues) ---
|
||||
- id: mcp.auth.api_key.returns_401_not_500
|
||||
module: mcp
|
||||
tier: P0
|
||||
operation: auth
|
||||
auth_family: api_key
|
||||
assertions: [returns_401_not_500]
|
||||
source: "server.py:extract_mcp_auth_context; PR #31011"
|
||||
rationale: "Malformed/invalid/expired key on /mcp returns 401 with WWW-Authenticate, not a flattened 500"
|
||||
- id: mcp.auth.api_key.returns_429_on_budget
|
||||
module: mcp
|
||||
tier: P0
|
||||
operation: auth
|
||||
auth_family: api_key
|
||||
assertions: [returns_429_on_budget]
|
||||
source: "server.py:extract_mcp_auth_context"
|
||||
rationale: "Over-budget key on /mcp returns 429, matching REST chat semantics"
|
||||
- id: mcp.list_tools.api_key.empty_intersection_denies
|
||||
module: mcp
|
||||
tier: P0
|
||||
operation: list_tools
|
||||
auth_family: api_key
|
||||
assertions: [empty_intersection_denies]
|
||||
source: "auth/user_api_key_auth_mcp.py permission hierarchy"
|
||||
rationale: "Empty key∩team (or similar) MCP permission intersection is deny-all, never allow-all"
|
||||
- id: mcp.permission.api_key.dangling_grant_errors
|
||||
module: mcp
|
||||
tier: P0
|
||||
operation: permission
|
||||
auth_family: api_key
|
||||
assertions: [dangling_grant_errors]
|
||||
source: "mcp_server_manager.py:_generate_stable_server_id; object_permission.mcp_servers"
|
||||
rationale: "Grants pointing at missing server_ids surface a loud error (or migrate), not silent empty tool lists"
|
||||
- id: mcp.permission.api_key.stable_id_survives_repoint
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: permission
|
||||
auth_family: api_key
|
||||
assertions: [stable_id_survives_repoint]
|
||||
source: "mcp_server_manager.py:_generate_stable_server_id"
|
||||
rationale: "Config.yaml rename or URL repoint keeps grants valid (stable id or automatic migration)"
|
||||
- id: mcp.list_tools.api_key.per_server_route
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: list_tools
|
||||
auth_family: api_key
|
||||
assertions: [per_server_route]
|
||||
source: "server.py dynamic_mcp_route"
|
||||
rationale: "Per-server MCP URL (e.g. /mcp/{alias}/) lists that server's tools, not only aggregate /mcp/"
|
||||
- id: mcp.list_tools.api_key.sees_newly_added_server
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: list_tools
|
||||
auth_family: api_key
|
||||
assertions: [sees_newly_added_server]
|
||||
source: "mcp_server_manager registry reload / config sync"
|
||||
rationale: "After admin adds a server, aggregate list_tools includes its tools without process restart"
|
||||
- id: mcp.list_tools.api_key.trailing_slash_parity
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: list_tools
|
||||
auth_family: api_key
|
||||
assertions: [trailing_slash_parity]
|
||||
source: "server.py MCP route registration"
|
||||
rationale: "/mcp and /mcp/ accept the same authenticated list_tools behavior"
|
||||
- id: mcp.oauth.api_key.resource_metadata_public_https
|
||||
module: mcp
|
||||
tier: P0
|
||||
operation: oauth
|
||||
auth_family: api_key
|
||||
assertions: [resource_metadata_public_https]
|
||||
source: "oauth_utils.py X-Forwarded-Proto / public base URL"
|
||||
rationale: "Behind a TLS terminator, protected-resource metadata advertises https:// so Claude accepts OAuth"
|
||||
- id: mcp.toolset.api_key.prefix_separator_honored
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: toolset
|
||||
auth_family: api_key
|
||||
assertions: [prefix_separator_honored]
|
||||
source: "toolset_db.py; MCP_TOOL_PREFIX_SEPARATOR; PR #34559"
|
||||
rationale: "Tools stored on a toolset use server{separator}tool with the configured separator and resolve on call"
|
||||
- id: mcp.health.api_key.byok_not_false_unhealthy
|
||||
module: mcp
|
||||
tier: P2
|
||||
operation: health
|
||||
auth_family: api_key
|
||||
assertions: [byok_not_false_unhealthy]
|
||||
source: "mcp_management_endpoints.py:health_check_servers; is_byok"
|
||||
rationale: "BYOK/per-user auth servers are not reported as hard Unhealthy solely because health check has no user token"
|
||||
- id: mcp.call_tool.api_key.unregistered_server_blocked
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [unregistered_server_blocked]
|
||||
source: "gateway allowlist intent; user_api_key_auth_mcp"
|
||||
rationale: "Clients cannot invoke arbitrary upstream MCP URLs that are not registered on the gateway"
|
||||
- id: mcp.call_tool.api_key.acting_user_oauth_token
|
||||
module: mcp
|
||||
tier: P1
|
||||
operation: call_tool
|
||||
auth_family: api_key
|
||||
assertions: [acting_user_oauth_token]
|
||||
source: "management store_mcp_oauth_user_credential; per_user_oauth_store"
|
||||
rationale: "Agent key call resolves a documented acting-user identity to that user's vaulted upstream OAuth token"
|
||||
|
|
|
|||
|
|
@ -31,6 +31,9 @@
|
|||
- {id: mgmt.team.delete.persists, module: mgmt, tier: P1, surface: api, assertions: [persists], source: "team_endpoints.py:1750", rationale: "Deletion prevents key access"}
|
||||
- {id: mgmt.team.block.persists, module: mgmt, tier: P1, surface: api, assertions: [persists], source: "team_endpoints.py", rationale: "Block suspends all members"}
|
||||
- {id: mgmt.team.info.happy_path, module: mgmt, tier: P1, surface: api, assertions: [happy_path], source: "team_endpoints.py:2244", rationale: "Metadata+members+budgets"}
|
||||
- {id: mgmt.team.daily_activity.happy_path, module: mgmt, tier: P1, surface: api, assertions: [happy_path], source: "vendor testing strategy §9.20 / LIT-4778", rationale: "GET /team/daily/activity returns results+metadata for a valid date range"}
|
||||
- {id: mgmt.team.daily_activity.missing_start_date_rejected, module: mgmt, tier: P1, surface: api, assertions: [missing_start_date_rejected], source: "vendor testing strategy §9.20 / LIT-4778", rationale: "Missing start_date on /team/daily/activity is 400"}
|
||||
- {id: mgmt.team.daily_activity.missing_end_date_rejected, module: mgmt, tier: P1, surface: api, assertions: [missing_end_date_rejected], source: "vendor testing strategy §9.20 / LIT-4778", rationale: "Missing end_date on /team/daily/activity is 400"}
|
||||
- {id: mgmt.team.list.happy_path, module: mgmt, tier: P1, surface: api, assertions: [happy_path], source: "team_endpoints.py:3645", rationale: "Pagination/filtering"}
|
||||
- {id: mgmt.team.member_update.persists, module: mgmt, tier: P1, surface: api, assertions: [persists], source: "team_endpoints.py:2768", rationale: "Member budget/role updates persist"}
|
||||
- {id: mgmt.user.update.persists, module: mgmt, tier: P1, surface: api, assertions: [persists], source: "internal_user_endpoints.py:555", rationale: "Metadata/perm updates persist"}
|
||||
|
|
@ -52,8 +55,19 @@
|
|||
- {id: mgmt.model.block.persists, module: mgmt, tier: P1, surface: api, assertions: [persists], source: "model_management_endpoints.py", rationale: "Blocked model stays blocked"}
|
||||
- {id: mgmt.access_group.new.happy_path, module: mgmt, tier: P1, surface: api, assertions: [happy_path], source: "model_access_group_management_endpoints.py:450", rationale: "Model permissioning group"}
|
||||
- {id: mgmt.access_group.info.happy_path, module: mgmt, tier: P1, surface: api, assertions: [happy_path], source: "model_access_group_management_endpoints.py:600", rationale: "Access group membership query"}
|
||||
- {id: mgmt.mcp_server.register.happy_path, module: mgmt, tier: P1, surface: api, assertions: [happy_path], source: "mcp_management_endpoints.py:880", rationale: "MCP server registration"}
|
||||
- {id: mgmt.mcp_server.approve.persists, module: mgmt, tier: P1, surface: api, assertions: [persists], source: "mcp_management_endpoints.py:1200", rationale: "Admin approval persists"}
|
||||
- {id: mgmt.mcp_server.register.happy_path, module: mgmt, tier: P1, surface: api, assertions: [happy_path], source: "mcp_management_endpoints.py:register_mcp_server", rationale: "MCP server registration (non-admin submission or admin add)"}
|
||||
- {id: mgmt.mcp_server.approve.persists, module: mgmt, tier: P1, surface: api, assertions: [persists], source: "mcp_management_endpoints.py:approve_mcp_server_submission", rationale: "Admin approval persists and loads the server into the runtime registry"}
|
||||
- {id: mgmt.mcp_server.reject.persists, module: mgmt, tier: P1, surface: api, assertions: [persists], source: "mcp_management_endpoints.py:reject_mcp_server_submission", rationale: "Admin rejection leaves the server out of the runtime registry"}
|
||||
- {id: mgmt.mcp_server.add.persists, module: mgmt, tier: P0, surface: api, assertions: [persists], source: "mcp_management_endpoints.py:add_mcp_server", rationale: "Admin create persists and is visible on fetch/list"}
|
||||
- {id: mgmt.mcp_server.update.persists, module: mgmt, tier: P0, surface: api, assertions: [persists], source: "mcp_management_endpoints.py:edit_mcp_server", rationale: "Partial update of URL/auth/tools persists without clobbering secrets incorrectly"}
|
||||
- {id: mgmt.mcp_server.delete.persists, module: mgmt, tier: P0, surface: api, assertions: [persists], source: "mcp_management_endpoints.py:remove_mcp_server", rationale: "Delete removes the server from list_tools and management fetch"}
|
||||
- {id: mgmt.mcp_server.health.happy_path, module: mgmt, tier: P1, surface: api, assertions: [happy_path], source: "mcp_management_endpoints.py:health_check_servers", rationale: "Health check reports reachable servers"}
|
||||
- {id: mgmt.mcp_server.list.happy_path, module: mgmt, tier: P1, surface: api, assertions: [happy_path], source: "mcp_management_endpoints.py:fetch_all_mcp_servers", rationale: "Admin list returns registered servers the caller may see"}
|
||||
- {id: mgmt.mcp_server.make_public.persists, module: mgmt, tier: P2, surface: api, assertions: [persists], source: "mcp_management_endpoints.py:make_mcp_servers_public", rationale: "Marking servers public updates discovery/registry visibility"}
|
||||
- {id: mgmt.mcp_toolset.create.persists, module: mgmt, tier: P1, surface: api, assertions: [persists], source: "mcp_management_endpoints.py:add_mcp_toolset", rationale: "Toolset create persists tools and scopes keys that reference it"}
|
||||
- {id: mgmt.mcp_toolset.delete.persists, module: mgmt, tier: P1, surface: api, assertions: [persists], source: "mcp_management_endpoints.py:remove_mcp_toolset", rationale: "Toolset delete drops permission expansion for that toolset"}
|
||||
- {id: mgmt.mcp_user_oauth_credential.store.persists, module: mgmt, tier: P1, surface: api, assertions: [persists], source: "mcp_management_endpoints.py:store_mcp_oauth_user_credential", rationale: "Storing a per-user OAuth credential makes subsequent call_tool work for that user"}
|
||||
- {id: mgmt.mcp_user_env_vars.store.persists, module: mgmt, tier: P2, surface: api, assertions: [persists], source: "mcp_management_endpoints.py:store_mcp_user_env_vars", rationale: "Per-user env vars are stored and resolve on the next tool call"}
|
||||
- {id: mgmt.budget.update.persists, module: mgmt, tier: P1, surface: api, assertions: [persists], source: "budget_management_endpoints.py:155", rationale: "Limit changes apply"}
|
||||
- {id: mgmt.budget.delete.persists, module: mgmt, tier: P1, surface: api, assertions: [persists], source: "budget_management_endpoints.py:280", rationale: "Clears limits"}
|
||||
- {id: mgmt.budget.list.happy_path, module: mgmt, tier: P1, surface: api, assertions: [happy_path], source: "budget_management_endpoints.py:215", rationale: "Budget enumeration"}
|
||||
|
|
|
|||
|
|
@ -68,8 +68,11 @@ class TestMcpChatCompletionOauth:
|
|||
the user's stored upstream token, lists and executes Linear's tools during
|
||||
the completion, and returns the answer."""
|
||||
|
||||
@pytest.mark.covers("mcp.list_tools.oauth.succeeds")
|
||||
@pytest.mark.covers("mcp.call_tool.oauth.succeeds")
|
||||
@pytest.mark.covers(
|
||||
"mcp.list_tools.oauth.succeeds",
|
||||
"mcp.call_tool.oauth.succeeds",
|
||||
"mcp.chat_completion.oauth.auto_executes_tools",
|
||||
)
|
||||
def test_chat_completion_uses_linear_with_x_litellm_api_key_header(
|
||||
self, chat_client: ChatMcpClient, resources: ResourceManager
|
||||
) -> None:
|
||||
|
|
@ -132,8 +135,11 @@ class TestMcpChatCompletionOauth:
|
|||
f"Linear tool {alias}-{LINEAR_READONLY_TOOL} was not executed with a result: {meta.mcp_call_results}"
|
||||
)
|
||||
|
||||
@pytest.mark.covers("mcp.list_tools.oauth.succeeds")
|
||||
@pytest.mark.covers("mcp.call_tool.oauth.succeeds")
|
||||
@pytest.mark.covers(
|
||||
"mcp.list_tools.oauth.succeeds",
|
||||
"mcp.call_tool.oauth.succeeds",
|
||||
"mcp.chat_completion.oauth.auto_executes_tools",
|
||||
)
|
||||
def test_chat_completion_uses_linear_with_authorization_bearer_header(
|
||||
self, chat_client: ChatMcpClient, resources: ResourceManager
|
||||
) -> None:
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue