mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-07 08:26:10 +00:00
1bb713bc7b
* feat(mcp): BYOK (Bring Your Own Key) for OpenAPI MCP servers with OAuth 2.1 flow
Adds per-user credential storage for BYOK MCP servers so external clients
can authenticate via standard OAuth 2.1 PKCE without needing a full identity
provider.
Backend:
- New DB table LiteLLM_MCPUserCredentials (user_id, server_id, credential_b64)
- is_byok, byok_description, byok_api_key_help_url fields on MCPServerTable
- OAuth 2.1 authorization server endpoints (/.well-known/oauth-authorization-server,
/.well-known/oauth-protected-resource, /v1/mcp/oauth/authorize, /v1/mcp/oauth/token)
- 401 challenge with WWW-Authenticate header when BYOK server has no credential
- CRUD endpoints: POST/DELETE /v1/mcp/server/{id}/user-credential
- has_user_credential annotated on GET /v1/mcp/server response
UI:
- ByokCredentialModal: 2-step Connect flow (access description + API key entry)
- BYOK toggle + description fields on admin MCP server create form
- Connect/Connected state in MCP server table
- BYOK Demo page (/tools/byok-demo) showing full OAuth 2.1 PKCE flow
* feat(mcp/byok): redesign OAuth authorize page to match 2-step Connect mockup
- Step 1: L→S logos, requested access checklist, How it works box, Continue button
- Step 2: API key input, Save toggle, Duration pills (1h/24h/7d/30d/until_revoked), security note
- Matches screenshots: white modal on dark bg, progress dots, dark CTA buttons
- Authorize handler now fetches byok_description and byok_api_key_help_url from server registry
- CLAUDE.md: replace SQL snippet with proper DB migration troubleshooting guidance
* fix: address greptile review feedback (greploop iteration 1)
- XSS: escape all user-supplied values in _build_authorize_html() with html.escape()
- Open redirect: validate redirect_uri scheme and URL-encode code/state in redirect
- N+1 query: batch BYOK credential lookup into single find_many() call
- Critical path DB: add 60s TTL in-memory cache to _check_byok_credential()
- Encrypt BYOK credentials at rest using encrypt_value_helper/decrypt_value_helper
* fix(byok): update OAuth popup with LiteLLM logo, MCP title suffix, remove emojis
* fix(byok-demo): fix token endpoint URL (/v1/mcp/oauth/token not /v1/mcp/token)
* feat(byok): inject stored BYOK credential as mcp_auth_header on tool execution
* feat(byok): use contextvars to inject per-user credential into OpenAPI tool closures; remove byok-demo from LiteLLM UI
OpenAPI tools have auth headers baked into their closures at registration time. BYOK servers have
no static auth token, so per-user credentials were never reaching the HTTP calls.
Fix: add _request_auth_header ContextVar in openapi_to_mcp_generator.py. create_tool_function now
reads this var at call time and overrides the Authorization header if set. execute_mcp_tool resolves
the MCP server and performs BYOK checks before the local-tool dispatch branch, then sets the
ContextVar around _handle_local_mcp_tool so the credential flows into the HTTP request.
Also remove the /tools/byok-demo page from the LiteLLM UI dashboard — the demo lives at
~/Downloads/litellm-byok-demo/index.html (served separately on port 8080).
* fix: address greptile review feedback (greploop iteration 2)
- Cache invalidation: add _invalidate_byok_cred_cache() and call it after
store_user_credential() in both token endpoint and management endpoint
- Unbounded cache: add _BYOK_CRED_CACHE_MAX_SIZE=4096 with clear-on-overflow
- Unbounded auth codes: add _AUTH_CODES_MAX_SIZE=1000 with 503 on overflow
- Double DB query: merge _check_byok_credential + _get_byok_credential into
single _get_byok_credential call; raise 401 inline if None returned
- Sidebar: remove byok-demo entry (page was deleted in prior commit)
- JWT comment: document why byok_session HS256 token can't be used as proxy auth
* fix: address greptile review feedback (greploop iteration 3)
- auth_type: pre-format Authorization header (Bearer/ApiKey/Basic) in server.py
before setting ContextVar so openapi_to_mcp_generator respects server auth_type
- cache invalidation on delete: call _invalidate_byok_cred_cache after
delete_user_credential so stale True entries don't persist for 60s
- ContextVar guard: only set _request_auth_header when mcp_auth_header is set,
avoiding unnecessary ContextVar overhead on non-BYOK tool calls
* fix: address greptile review feedback (greploop iteration 4)
- Unified credential cache: store actual credential value (Optional[str])
instead of just bool so _get_byok_credential also benefits from caching —
eliminates the DB hit on every BYOK tool call within the 60s TTL window
- Extracted _write_byok_cred_cache() helper for consistent cache writes
- Replaced has_user_credential with get_user_credential in _check_byok_credential
so one DB call satisfies both existence check and value retrieval
- Remove false 'encrypted at rest' claim from OAuth HTML and ByokCredentialModal
* Update tests/test_litellm/proxy/_experimental/mcp_server/test_byok_oauth_endpoints.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update tests/test_litellm/proxy/_experimental/mcp_server/test_byok_oauth_endpoints.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
|
||
|---|---|---|
| .. | ||
| _experimental | ||
| agent_endpoints | ||
| analytics_endpoints | ||
| anthropic_endpoints | ||
| auth | ||
| batches_endpoints | ||
| client | ||
| common_utils | ||
| config_management_endpoints | ||
| container_endpoints | ||
| credential_endpoints | ||
| custom_hooks | ||
| db | ||
| discovery_endpoints | ||
| example_config_yaml | ||
| fine_tuning_endpoints | ||
| google_endpoints | ||
| guardrails | ||
| health_check_utils | ||
| health_endpoints | ||
| hooks | ||
| image_endpoints | ||
| management_endpoints | ||
| management_helpers | ||
| middleware | ||
| ocr_endpoints | ||
| openai_evals_endpoints | ||
| openai_files_endpoints | ||
| pass_through_endpoints | ||
| policy_engine | ||
| prompts | ||
| public_endpoints | ||
| rag_endpoints | ||
| rerank_endpoints | ||
| response_api_endpoints | ||
| response_polling | ||
| search_endpoints | ||
| spend_tracking | ||
| swagger | ||
| test_prompts | ||
| types_utils | ||
| ui_crud_endpoints | ||
| vector_store_endpoints | ||
| vector_store_files_endpoints | ||
| vertex_ai_endpoints | ||
| video_endpoints | ||
| .gitignore | ||
| __init__.py | ||
| _logging.py | ||
| _new_new_secret_config.yaml | ||
| _new_secret_config.yaml | ||
| _super_secret_config.yaml | ||
| _types.py | ||
| cached_logo.jpg | ||
| caching_routes.py | ||
| common_request_processing.py | ||
| compliance_checks.py | ||
| custom_auth_auto.py | ||
| custom_prompt_management.py | ||
| custom_sso.py | ||
| custom_validate.py | ||
| dd_span_tagger.py | ||
| enterprise | ||
| health_check.py | ||
| lambda.py | ||
| litellm_pre_call_utils.py | ||
| llamaguard_prompt.txt | ||
| logo.jpg | ||
| mcp_registry.json | ||
| mcp_tools.py | ||
| model_config.yaml | ||
| openapi.json | ||
| post_call_rules.py | ||
| prisma_migration.py | ||
| prometheus_cleanup.py | ||
| proxy_cli.py | ||
| proxy_config.yaml | ||
| proxy_server.py | ||
| README.md | ||
| route_llm_request.py | ||
| schema.prisma | ||
| start.sh | ||
| utils.py | ||
litellm-proxy
A local, fast, and lightweight OpenAI-compatible server to call 100+ LLM APIs.
usage
$ pip install litellm
$ litellm --model ollama/codellama
#INFO: Ollama running on http://0.0.0.0:8000
replace openai base
import openai # openai v1.0.0+
client = openai.OpenAI(api_key="anything",base_url="http://0.0.0.0:8000") # set proxy to base_url
# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(model="gpt-3.5-turbo", messages = [
{
"role": "user",
"content": "this is a test request, write a short poem"
}
])
print(response)
See how to call Huggingface,Bedrock,TogetherAI,Anthropic, etc.
Folder Structure
Routes
proxy_server.py- all openai-compatible routes -/v1/chat/completion,/v1/embedding+ model info routes -/v1/models,/v1/model/info,/v1/model_group_inforoutes.health_endpoints/-/health,/health/liveliness,/health/readinessmanagement_endpoints/key_management_endpoints.py- all/key/*routesmanagement_endpoints/team_endpoints.py- all/team/*routesmanagement_endpoints/internal_user_endpoints.py- all/user/*routesmanagement_endpoints/ui_sso.py- all/sso/*routes