mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-14 23:21:35 +00:00
c792df64d2
* feat(mcp): support stateless and stateful clients via session-id routing
- Add session_manager_stateful (stateless=False) alongside stateless
- Route by mcp-session-id: has ID → stateful, initialize (no ID) → stateful, else → stateless
- Peek POST body to detect initialize for routing; replay via wrapped receive
- Handle stale session IDs for both managers
- Add test_mcp_routing_initialize_to_stateful_no_session_to_stateless
- Update test_valid_mcp_session_id_is_preserved, test_concurrent_initialize_session_managers
Made-with: Cursor
* fix(mcp): respect stateful routing and harden initialize detection
Ensure streamable MCP requests are dispatched via the computed target session manager, and guard initialize detection against non-object JSON bodies. Update stale-session test patches to target the stateful manager so routing assertions remain correct.
Made-with: Cursor
* test(mcp): patch stateless/stateful managers in concurrency init test
Update concurrent session-manager initialization test to patch session_manager_stateless and session_manager_stateful directly, matching initialize_session_managers() behavior and preventing NameError from undefined mocks.
Made-with: Cursor
* Fix tests
* Fix tests
* Fix MCP stateful routing edge cases
* Fix stateful MCP auth context refresh
* Fix MCP stateful session cleanup
* fix(mcp): bind stateful sessions to creator and reject hijacks
Stateful mcp-session-id was usable by any authenticated proxy caller. Track
the session creator's hashed API key (or user_id) when a new session is
issued and reject mismatched callers with 403 before _set_or_update_auth_context
overwrites the stored MCPAuthenticatedUser. Also formats nested with-statements
in test_mcp_stale_session.py and fixes a pre-existing AsyncMock mismatch in
test_stale_mcp_session_id_is_stripped.
* fix(mcp): serialize concurrent requests on same stateful session
Bugbot's 'Concurrent requests share context' finding: _update_auth_context
mutates the single MCPAuthenticatedUser stored per session in place on
every request, so two requests sharing one mcp-session-id can overwrite
each other's mcp_servers / auth headers / oauth state / client_ip while
in-flight callbacks are still reading the same object.
Owner-binding alone narrows this to same-principal racing, but the
in-place mutation race remains. Add a per-session asyncio.Lock around
handle_request so concurrent same-session requests run sequentially. The
lock is allocated on demand and torn down with the rest of the session
state on DELETE / idle expiry.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(mcp): include OAuth2 bearer in stateful session owner fingerprint
UserAPIKeyAuth() for OAuth2 passthrough has no api_key/user_id, so every
OAuth caller fingerprinted to "anonymous" and could hijack another OAuth
caller's mcp-session-id. Hash the upstream Authorization header into the
fingerprint as oauth:<sha256>.
* fix(mcp): don't hold stateful session lock for streaming GETs
The per-session lock wraps handle_request, so a long-lived GET (SSE
stream held open for the life of the session) would block every
subsequent POST on the same mcp-session-id. Only POST/DELETE mutate the
shared MCPAuthenticatedUser, so it's sufficient to serialize those —
GETs run lock-free and stream concurrently.
* fix(mcp): allow None user_api_key_auth in MCPAuthenticatedUser
The set_auth_context / _set_or_update_auth_context / _update_auth_context
helpers in server.py all accept Optional[UserAPIKeyAuth] and pass it
straight into MCPAuthenticatedUser, but the dataclass-style constructor
typed user_api_key_auth as required UserAPIKeyAuth. Mypy flagged this on
the stateful-routing branch:
server.py:3227: error: Incompatible types in assignment (expression
has type "UserAPIKeyAuth | None", variable has type "UserAPIKeyAuth")
server.py:3255: error: Argument "user_api_key_auth" to
"MCPAuthenticatedUser" has incompatible type "UserAPIKeyAuth | None";
expected "UserAPIKeyAuth"
Widen the parameter type to Optional[UserAPIKeyAuth] to match the call
sites. Runtime behavior is unchanged.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* style: replace with new alias
* fix(mcp): fall back to client_ip in stateful session owner fingerprint
Addresses Greptile review on PR #26857: when no API key, user_id, or
OAuth bearer is available (e.g. unauthenticated/passthrough callers),
the owner fingerprint collapsed to a single 'anonymous' value, allowing
two unrelated callers to drive each other's stateful MCP sessions.
Fold client IP into the fingerprint as a fallback identity signal so
distinct anonymous sources do not share an owner identity.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* Fix active stateful MCP session cleanup
* test(mcp): cancel leaked stateful auth-context cleanup task
initialize_session_managers() spawns a real asyncio.create_task running
_cleanup_expired_stateful_session_auth_contexts(). The
test_concurrent_initialize_session_managers test was saving and
restoring the session-manager context-manager globals but did not save,
cancel, or restore _stateful_auth_context_cleanup_task.
Because pyproject.toml sets asyncio_default_fixture_loop_scope=session,
the event loop is shared across tests in the same session, so the
leaked task kept running against module-level dicts for the rest of the
test run. Save and cancel the task in the finally block so the test
fully cleans up after itself.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* Fix stateful MCP session fingerprinting
* Hash MCP session user owner fingerprints
* Fix stale MCP session DELETE cleanup
* fix(mcp): harden owner fingerprint hashing for non-str api keys
_owner_fingerprint_for assumed api_key/user_id supported .encode();
MagicMock-based tests (and any non-str truthy values) crashed with
TypeError before routing. Only hash str/bytes secrets; fall through
otherwise so MCP routing and session tests behave correctly.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix MCP stateful cleanup loop resilience
* Fix stateful MCP initialize auth capture
* fix(mcp): drop orphan per-session lock when auth context absent
Defensive cleanup for _stateful_session_locks entries created on
sessions that never enter _stateful_session_auth_contexts. The
periodic cleanup loop only iterates auth_context_last_seen, so such
locks would otherwise live forever. Add a test that reproduces the
leak and verifies the request finalizer pops the lock.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* chore(mcp): trim verbose comment on lock cleanup
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* Fix stateful MCP delete failure tracking
* fix test
* fix(mcp): cap routing-peek body size to bound pre-dispatch memory
Authenticated clients that POST without an mcp-session-id forced the proxy
to buffer the entire request body before routing, since the peek loop
drained every body chunk to decide whether the JSON-RPC method was
'initialize'. Cap the peek at 4 KB (more than enough for an initialize
envelope) and let the remainder stream through wrapped_receive into the
downstream handler.
* test: replace dall-e-3 with gpt-image-1 in health check and router tests (#27813)
OpenAI returns 'The model dall-e-3 does not exist' for the test account,
breaking test_openai_img_gen_health_check and test_image_generation.
Switch to gpt-image-1, matching the existing TestOpenAIGPTImage1 pattern.
* fix(tests): drop dall-e-only test classes; route live image tests via gpt-image-1
Second wave of failures from the 2026-05-12 DALL-E shutdown:
- tests/image_gen_tests/test_image_edits.py::TestOpenAIImageEditDallE2
and tests/image_gen_tests/test_image_generation.py::TestOpenAIDalle3
are explicitly named for the deprecated models and can't pass; remove.
gpt-image-1 coverage already exists in sibling classes.
- tests/local_testing/test_router.py image gen tests use dall-e-3 only
as a routing example; swap to gpt-image-1.
- tests/local_testing/test_custom_callback_input.py image_generation
success/failure paths swapped to gpt-image-1.
* Fix MCP initialize session active tracking
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* Fix MCP reinitialize session tracking
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* Fix MCP reinitialize auth context aliasing
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* Apply black formatting after merge
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* Run owner-binding 403 before consuming POST body
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* Harden MCP routing peek bound and stateful purge race
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* Remove inadvertently committed Next.js build artifacts
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* Run owner check before stale MCP session cleanup
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(mcp): reverse cleanup ordering to terminate transport before clearing owner
Reverses _purge_expired_stateful_session_auth_contexts so the transport
is popped from server_instances and terminated BEFORE owner/auth tracking
is cleared. The previous order left a window where _stateful_session_owners
was already empty but server_instances still served the session, so a
concurrent request would observe expected_owner is None and bypass the
owner-binding check. Addresses Greptile review on PR #26857.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(mcp): fully reset stateful session tracking in auth-context refresh test
Use _remove_stateful_session_tracking in teardown so the test no longer
leaks _stateful_session_auth_context_last_seen and _stateful_session_locks
between tests, matching the cleanup used by the sibling stateful tests.
* fix(mcp): cap concurrent stateful sessions per caller to bound memory
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Co-authored-by: Sameerlite <sameerlite@users.noreply.github.com>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Claude Babysitter <claude@anthropic.com>
Co-authored-by: mateo-berri <mateo@berri.ai>
|
||
|---|---|---|
| .. | ||
| _experimental/mcp_server | ||
| agent_endpoints | ||
| anthropic_endpoints | ||
| auth | ||
| client | ||
| common_utils | ||
| db | ||
| discovery_endpoints | ||
| experimental/mcp_server | ||
| google_endpoints | ||
| guardrails | ||
| health_endpoints | ||
| hooks | ||
| image_endpoints | ||
| management_endpoints | ||
| management_helpers | ||
| memory | ||
| middleware | ||
| openai_files_endpoint | ||
| pass_through_endpoints | ||
| policy_engine | ||
| prompts | ||
| proxy_server | ||
| public_endpoints | ||
| rag_endpoints | ||
| realtime_endpoints | ||
| response_api_endpoints | ||
| spend_tracking | ||
| test_configs | ||
| types_utils | ||
| ui_crud_endpoints | ||
| vector_store_endpoints | ||
| __init__.py | ||
| conftest.py | ||
| test_aiohttp_cleanup_closed.py | ||
| test_aiohttp_session_recovery.py | ||
| test_api_key_masking_in_errors.py | ||
| test_audio_speech_prometheus_hooks.py | ||
| test_batch_expiry.py | ||
| test_batch_metadata_none_fix.py | ||
| test_batch_retrieve_bedrock.py | ||
| test_budget_reservation.py | ||
| test_caching_routes.py | ||
| test_chat_completion_metadata.py | ||
| test_common_request_processing.py | ||
| test_component_allowlists.py | ||
| test_cors_config.py | ||
| test_custom_proxy.py | ||
| test_dynamic_mcp_route.py | ||
| test_empty_model_list.py | ||
| test_enforce_user_param.py | ||
| test_fallback_management_endpoints.py | ||
| test_fastapi_offline_routes.py | ||
| test_filter_models_by_team_access_group.py | ||
| test_health_check_functions.py | ||
| test_health_check_max_tokens.py | ||
| test_langfuse_passthrough_security.py | ||
| test_lazy_openapi_snapshot.py | ||
| test_litellm_pre_call_utils.py | ||
| test_max_budget_env_var.py | ||
| test_mcp_asgi_response.py | ||
| test_model_dump_with_preserved_fields.py | ||
| test_model_id_header_propagation.py | ||
| test_model_info_default_limits.py | ||
| test_model_level_guardrails.py | ||
| test_openapi_schema_validation.py | ||
| test_pricing_field_strip.py | ||
| test_prometheus_cleanup.py | ||
| test_provider_url_destination_guard.py | ||
| test_proxy_cli.py | ||
| test_proxy_logging_hook_detection.py | ||
| test_proxy_server.py | ||
| test_proxy_types.py | ||
| test_proxy_utils.py | ||
| test_pyroscope.py | ||
| test_redis_auth_cache_flag.py | ||
| test_response_model_sanitization.py | ||
| test_route_a2a_models.py | ||
| test_route_llm_request.py | ||
| test_sensitive_route_auth.py | ||
| test_shared_health_check.py | ||
| test_spend_log_cleanup.py | ||
| test_swagger_chat_completions.py | ||
| test_team_member_update.py | ||
| test_team_org_move.py | ||
| test_tools_allowlist_enforcement.py | ||
| test_update_llm_router_resilience.py | ||
| test_utils.py | ||