Backend:
- Extend _get_byok_credential to handle OAuth2 servers (auth_type=oauth2)
in addition to BYOK servers, retrieving stored user credentials
- Add _resolve_oauth_access_token() to extract access_token from JSON
credentials and check expiry
- Add _maybe_refresh_oauth_token() to automatically refresh expired
OAuth tokens using stored refresh_token before tool execution
- Extend store_mcp_user_credential endpoint to accept OAuth2 servers
(was BYOK-only)
- Annotate has_user_credential on OAuth2 servers in server list
(was BYOK-only)
- Extend tool execution to retrieve user credentials for OAuth2 servers
Frontend:
- After OAuth callback, store full token response (access_token +
refresh_token + expires_at) in backend via storeMCPUserCredential
- Add storeMCPUserCredential networking function
- Initialize OAuth token status from server's has_user_credential flag
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
getChatUrl() hardcoded /ui/chat which does not exist as a Next.js
route in dev mode. Use NEXT_PUBLIC_BASE_URL to determine the correct
prefix (/chat in dev, /ui/chat in production).
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Port MCP event handling from Playground to the standalone Chat UI:
- ChatPage: add mcpEvents state, handleMCPEvent callback, fetch
MCPServer[] objects, pass onMCPEvent + mcpServers to
makeOpenAIChatCompletionRequest and streamToModel
- ChatMessages: accept mcpEvents prop, render MCPEventsDisplay
(list tools, tool calls with request/response) inside the last
assistant bubble, reusing the existing Playground component
- Reset mcpEvents on each new send
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Fixes found by dogfooding the MCP OAuth 2.0 flow as both an admin and
an internal user:
1. Fix [object Object] error when creating MCP server
- Use parseErrorMessage() in create_mcp_server.tsx catch handler
- Fix deriveErrorMessage() in networking.tsx to always return a string
instead of potentially returning an object
2. Pre-configured OAuth servers auto-select auth type
- Add auth_type field to DiscoverableMCPServer type
- Add auth_type='oauth2' to Atlassian and Linear in mcp_registry.json
- Pre-fill auth_type in create form when selecting from catalog
- Use requestAnimationFrame to defer setting after transport dropdown mounts
3. Backend: drop tool_choice when MCP resolves no tools
- In chat_completions_handler.py, remove tool_choice from kwargs when
all_tools is empty after MCP resolution, preventing 400 errors from
providers that reject tool_choice without tools
4. Chat UI: OAuth-aware MCP server management
- MCPAppsPanel: show lock icon for OAuth servers in grid, show
'Authorization required' banner + 'Authorize' button in detail view,
and initiate PKCE OAuth flow via existing proxy endpoints
- MCPConnectPicker: show lock icon next to OAuth server names
- Store user OAuth tokens in sessionStorage for the session
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* feat: add LITELLM_WORKER_STARTUP_HOOKS for per-worker initialization (gflags support)
Add support for running user-defined startup hooks in each worker process
during proxy_startup_event. This enables re-initialization of in-process
state (like gflags.FLAGS) that doesn't survive uvicorn worker spawning.
Usage:
export LITELLM_WORKER_STARTUP_HOOKS=mymodule:init_fn,other:setup_fn
Hooks run early in proxy_startup_event (before config/DB loading).
Supports both sync and async callables. Errors propagate to prevent
broken workers from serving traffic. No-op when env var is unset.
Includes 5 tests covering sync/async hooks, multiple hooks, error
propagation, and no-hooks-set scenarios.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* docs: add Worker Startup Hooks page with gflags usage example
- New docs page: docs/proxy/worker_startup_hooks.md
- Explains the problem (per-process state lost in multi-worker deployments)
- Full gflags example with wrapper module and startup script
- Covers multiple hooks, async hooks, error behavior
- Architecture diagram showing master→worker flow
- Added LITELLM_WORKER_STARTUP_HOOKS to config_settings.md env var table
- Added to sidebar under Setup & Deployment
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Update litellm/proxy/proxy_server.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(mcp): resolve \$ref params and merge path-level params in OpenAPI tool registration
Real-world OpenAPI specs (e.g. GitHub's 11.8 MB official spec) use two
patterns that crashed tool registration:
1. \$ref parameters: params defined as {"$ref": "#/components/parameters/foo"}
instead of inline objects. Accessing param["name"] on a $ref raises KeyError.
Fix: resolve each param against components/parameters before processing.
2. Path-level parameters: params defined on the path object apply to all
HTTP methods on that path, but the operation object doesn't include them.
GitHub's spec uses this for owner/repo/etc. path params.
Fix: merge path-level params with operation-level params (op-level wins
when the same name+in combination appears in both).
With this fix the full GitHub REST API spec loads successfully:
720 paths → 1079 tools, all with correct parameter schemas.
* fix(mcp): resolve \$ref params in OpenAPI preview endpoint (test/tools/list)
The _preview_openapi_tools function (called by the UI add-server form to show
connection status and available tools) had the same bug as _register_openapi_tools:
it accessed param["name"] directly without resolving \$ref parameters or merging
path-level parameters from the path item.
This caused "Failed to load OpenAPI spec: 'name'" for any spec that uses
component-level parameter references (e.g. GitHub's official REST API spec).
Apply the same fix: resolve \$ref against components/parameters and merge
path-level params (with operation-level taking priority) before building schemas.
* refactor(openapi-mcp): extract resolve_operation_params, add tests
- Hoist _resolve_ref and _resolve_param_list to module level in
openapi_to_mcp_generator.py (were being redefined on every loop iteration)
- _resolve_ref now returns None for unresolvable $refs instead of
the stub dict, preventing (None, None) from poisoning deduplication
- Add resolve_operation_params() as a shared helper that handles both
$ref resolution and path-level param merging
- Replace duplicated inline logic in mcp_server_manager.py and
rest_endpoints.py with calls to resolve_operation_params()
- Add TestResolveRef, TestResolveParamList, TestResolveOperationParams
test classes covering $ref resolution, path-level merging, collision
semantics, unresolvable ref filtering, and a GitHub-style spec fixture
When Redis Cluster is configured via the REDIS_CLUSTER_NODES environment
variable, Cache.__init__() and Router._create_redis_cache() ignored the
env var and always created RedisCache instead of RedisClusterCache. This
caused the v3 rate limiter's cluster detection (_is_redis_cluster()) to
return False, skipping hash-slot key grouping. The resulting CROSSLOT
errors were silently caught, falling back to per-instance in-memory
counting — breaking RPM/TPM enforcement across multiple proxy instances.
Add REDIS_CLUSTER_NODES env var detection to both Cache.__init__() and
Router._create_redis_cache(), matching the existing pattern in
_redis.py:215-220. When the env var is set and no explicit startup_nodes
parameter is provided, parse it and create RedisClusterCache.
Fixes#22748
Related to #20836
Adds @ to the _KEY_ALIAS_PATTERN allowed character set so that
key aliases like user/user@example.com are accepted. Updates tests
to cover email-based alias formats.
Adds Vitest + RTL test files for policy_table, policy_templates,
guardrail_selection_modal, impact_popover, and add_attachment_form.
53 tests total covering rendering, user interactions, API calls,
and conditional UI behavior.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(vertex_ai): support explicit AWS credentials for WIF auth
The current Vertex AI AWS Workload Identity Federation implementation
exclusively uses google.auth.aws.Credentials.from_info(), which requires
EC2 instance metadata access to obtain AWS credentials. In environments
where the metadata service is blocked for security reasons, this makes
WIF unusable.
Add support for explicit AWS credentials by implementing a custom
AwsSecurityCredentialsSupplier (google-auth >= 2.29.0). When aws_* keys
(e.g. aws_role_name, aws_region_name) are present in the WIF credential
JSON, LiteLLM uses BaseAWSLLM.get_credentials() to obtain AWS creds via
STS AssumeRole (or any other supported AWS auth flow), wraps them in the
custom supplier, and passes them to aws.Credentials() — bypassing the
metadata service entirely.
When no aws_* keys are present, the existing from_info() flow is used
unchanged, preserving full backward compatibility.
* refactor(vertex_ai): extract AWS WIF auth to own class + add docs
Address PR review feedback:
- Move _AWS_CREDENTIAL_KEYS, _extract_aws_params(), and
_credentials_from_aws_with_explicit_auth() from VertexBase into
new VertexAIAwsWifAuth class in vertex_ai_aws_wif.py
- Add documentation for explicit AWS credentials WIF auth method
in vertex.md (supported params, JSON example, SDK/Proxy tabs)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(vertex_ai): use lazy credentials provider to prevent stale STS tokens
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Remove unused `completed_jobs` list (dead code after per-job update refactor)
- Wrap DB update in try/except to prevent one failed update from aborting remaining jobs
- Add test assertions verifying batch_processed, status, and file_object are written to DB
CheckBatchCost poller updated the status column but not the file_object
JSON column. The list_batches endpoint reads status from file_object,
so batches appeared stuck in "validating" even after Azure reported
them as completed. Now update file_object alongside status in the
per-job DB write.
Function calls not supported with reasoning_effort != 'none' on gpt-5.4.
Drop reasoning_effort when tools are in the request (small minority of volume).
Made-with: Cursor
The OpenAI chat completion API expects reasoning_effort as a string
('none', 'low', 'medium', 'high', 'xhigh'). Config/deployments may pass
the Responses API format: {'effort': 'high', 'summary': 'detailed'}.
Fix BadRequestError when model config uses dict format by extracting
the 'effort' value before passing to the API.
Made-with: Cursor
When forward_llm_provider_auth_headers=true, Authorization: Bearer <litellm-key> was
being forwarded to Anthropic if it looked like an OAuth key, causing auth failures.
Now checked against authenticated_with_header: if Authorization was used to authenticate
with the proxy, it is always stripped before forwarding to the LLM provider.
Made-with: Cursor
- Add forward_llm_provider_auth_headers support from litellm_settings
- When enabled, client x-api-key takes precedence over deployment keys
- Forward x-api-key when x-litellm-api-key or Authorization used for auth
- Fix duplicate patch lines in test_byok_oauth_endpoints.py
- Add Claude Code BYOK documentation with /login and ANTHROPIC_CUSTOM_HEADERS
- Add unit tests for clean_headers x-api-key forwarding logic
- Sync model_prices backup (pre-commit hook)
Made-with: Cursor
- Pass request_model to Azure AI cost calculator to detect router requests
- Add router flat cost ($0.14/M input tokens) even when Azure returns actual model in response
- Add test for router flat cost with response containing actual model
- Update docs with cost calculation flow and configuration requirements
Made-with: Cursor
- Shift from hardcoded model checks to dynamic lookup via _supports_factory
- Add supports_none_reasoning_effort for gpt-5.1/5.2/5.4 chat variants
- Add supports_xhigh_reasoning_effort for gpt-5.1-codex-max, gpt-5.2, gpt-5.4+
- Update model_prices_and_context_window.json and backup
- Add ProviderSpecificModelInfo types for new fields
- Fix Azure: use _supports_reasoning_effort_level instead of removed is_model_gpt_5_1_model
Made-with: Cursor
Support passing duration=null on /key/update to reset a key's expiry to never expires, alongside the existing "-1" magic string (kept for backward compat).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
OpenAI gpt-5.1, gpt-5.2, and gpt-5.3 chat models all support the
`web_search_options` parameter, but the model cost registry was missing
the `supports_web_search` flag. Only `gpt-5.2-pro` had it set.
Models updated:
- gpt-5.1, gpt-5.1-2025-11-13, gpt-5.1-chat-latest
- gpt-5.2, gpt-5.2-2025-12-11, gpt-5.2-chat-latest
- gpt-5.3-chat-latest
* fix(chat): fix router.push paths to use /ui/chat with serverRootPath support
* fix(chat): wrap chat page in Suspense boundary for Next.js static export
* fix(chat): fix clipboard writeText rejection handler - remove undefined message.error call
* feat(chat): rebuild UI with routing fixes
* fix(chat): use useTheme logoUrl + /get_image fallback for sidebar logo
* feat(chat): rebuild UI with logo fix
* fix(chat): use /get_image directly for logo (no ThemeProvider outside dashboard layout)
* feat(chat): add multi-model comparison and provider logos in chat UI
- Replace single model selector with multi-select (up to 3 models)
- Show provider logos next to model names in dropdown (openai, anthropic, gemini, mistral, groq, etc.)
- Selected models float to the top of the dropdown list
- Multi-model mode: responses stream in parallel side-by-side cards below each user message
- Multi-turn: each follow-up message carries full per-model history as context
- Surface API errors inline in response cards instead of silently swallowing them
- Rebuild UI
