Prevent future regressions by documenting that LLMClientCache must never
close HTTP/SDK clients on eviction — evicted clients may still be held
by in-flight requests.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Add two new tests using real AsyncOpenAI SDK clients (the exact production
path) that verify clients remain usable after cache eviction:
- test_evicted_openai_sdk_client_stays_usable: size-based eviction
- test_ttl_expired_openai_sdk_client_stays_usable: TTL-based eviction
All four e2e tests now sleep after eviction to drain any background
close tasks — without this sleep, a regression that schedules close via
create_task() would go undetected (the original tests passed by luck).
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
LLMClientCache._remove_key() was eagerly closing httpx/OpenAI clients on
cache eviction. If any in-flight request still held a reference to the
evicted client, the request failed with:
RuntimeError: Cannot send a request, as the client has been closed.
This manifested in production after 1-2 hours (matching the 1-hour
_DEFAULT_TTL_FOR_HTTPX_CLIENTS). The fix was previously applied in PR
#22247 but regressed when commit fb72979432 re-added the close logic to
address unawaited coroutine warnings.
The correct fix is to NOT close clients on eviction at all:
- Evicted clients may still be referenced by in-flight requests
- Without calling close(), no coroutines are created, so no warnings
- Python GC handles cleanup when all references are dropped
- close_litellm_async_clients() handles explicit shutdown cleanup
Fixes the 'Cannot send a request, as the client has been closed' error
for OpenAI, Azure, and Bedrock streaming after 1+ hours of operation.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(passthrough): raise_for_status in _async_streaming to propagate Azure 429s
* address greptile review feedback (greploop iteration 1)
Guard data/json args when content is provided to avoid httpx ValueError
* address greptile review feedback (greploop iteration 2)
Use bare raise to preserve original traceback in _async_streaming exception handler
* address greptile review feedback (greploop iteration 3)
Close httpx streaming response on error to prevent connection pool exhaustion
* address greptile review feedback (greploop iteration 4)
Guard aclose() call to prevent masking original exception; add explicit test for content param forwarding
* address greptile review feedback (greploop iteration 5)
Pass content to sign_request so AWS body-hash signing is correct when content is the sole body source
* revert sign_request content change - request_data expects dict, not bytes
Bedrock's sign_request calls json.dumps(request_data) — passing content bytes
would TypeError. sign_request should only receive data/json (dict), not raw bytes.
PR #22890 used cast(str, ...) / cast(Optional[str], ...) for the return
statements; this PR's approach uses str() for explicit runtime coercion
(addressing Greptile's concern). Keep the str() version and drop the
now-unused cast import.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Documents exactly how every request and response field gets translated
when LiteLLM routes an Anthropic /v1/messages call through the OpenAI
Responses API path (for OpenAI/Azure targets). Covers messages content
block mapping, tools, tool_choice, thinking→reasoning, context_management,
and the reverse response translation. Wired into the /v1/messages sidebar.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- CreateBatchRequest.output_expires_after: drop Optional since total=False
already makes the key absent-or-present; Optional[T] incorrectly allowed
the key to exist with value None, which is incompatible with the OpenAI
SDK's OutputExpiresAfter | NotGiven expectation on batches.create()
- cost_tracking_settings._resolve_model_for_cost_lookup: replace implicit
object-to-str returns with explicit str() calls so the function is safe
even if the surrounding truthiness guards are later weakened
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
PR #22850 (BYOK MCP servers) accidentally re-declared spec_path which was
already added by PR #22820, causing Prisma schema validation to fail with
error P1012 "Field is already defined".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- batches/main.py: import FileExpiresAfter, cast output_expires_after on assignment
- openai/openai.py, azure/batches/handler.py: add # type: ignore[arg-type] on
batches.create / batches.retrieve TypedDict unpacking calls
- searchapi/transformation.py: cast optional_params["country"] to str before .lower()
- openrouter/image_edit/transformation.py: cast iterated value to str for size/quality params
- spend_log_cleanup.py: narrow bool | None to bool with `or False`
- cost_tracking_settings.py: cast base_model/resolved_model to str and
custom_llm_provider to Optional[str] in return statements
- text_moderation.py: suppress misc TypedDict ** expansion error; use cast for response
- prompt_shield.py: use cast instead of TypedDict(**response_json) construction
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gemini/gemini-live-2.5-flash-preview-native-audio-09-2025 uses mode='realtime'
but the schema in test_aaamodel_prices_and_context_window_json_is_valid did
not include 'realtime' as a valid enum value, causing a ValidationError.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- a2a_protocol/main.py: replace bare assert with descriptive RuntimeError
in _execute_a2a_send_with_retry so retry exhaustion gives a clear message
- fine_tuning/main.py: fix _resolve_fine_tuning_timeout return type from
float to Union[float, httpx.Timeout] to accurately reflect the passthrough path
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Vertex AI does not support the output_config parameter in its API.
This parameter is being added by Anthropic/Gemini transformations but needs
to be removed before sending requests to Vertex AI endpoints.
This fix addresses the "Extra inputs are not permitted" error (issue #22312)
when using Claude models with structured outputs on Vertex AI.
Changes:
- Drop output_config in Gemini model transformation
- Drop output_config in Anthropic partner model transformation
- Drop output_config in Anthropic experimental pass-through transformation
- Add comprehensive tests to verify output_config is dropped
Fixes: #22312
Made-with: Cursor
Adds bedrock_mantle_models to the model_list union and models_by_provider
dict so models are discoverable via litellm.model_list and
litellm.models_by_provider["bedrock_mantle"].
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Azure AI Foundry's Anthropic endpoint does not support the scope field in
cache_control. Strip it from both system and messages before sending.
Made-with: Cursor
Bedrock does not support the scope field in cache_control (e.g. 'global' for
cross-request caching). Only type and ttl are supported per AWS docs.
- Remove scope from cache_control in both system and messages
- Extend _remove_ttl_from_cache_control to process system blocks
- Add test for scope removal
Made-with: Cursor
* feat(mcp): BYOK (Bring Your Own Key) for OpenAPI MCP servers with OAuth 2.1 flow
Adds per-user credential storage for BYOK MCP servers so external clients
can authenticate via standard OAuth 2.1 PKCE without needing a full identity
provider.
Backend:
- New DB table LiteLLM_MCPUserCredentials (user_id, server_id, credential_b64)
- is_byok, byok_description, byok_api_key_help_url fields on MCPServerTable
- OAuth 2.1 authorization server endpoints (/.well-known/oauth-authorization-server,
/.well-known/oauth-protected-resource, /v1/mcp/oauth/authorize, /v1/mcp/oauth/token)
- 401 challenge with WWW-Authenticate header when BYOK server has no credential
- CRUD endpoints: POST/DELETE /v1/mcp/server/{id}/user-credential
- has_user_credential annotated on GET /v1/mcp/server response
UI:
- ByokCredentialModal: 2-step Connect flow (access description + API key entry)
- BYOK toggle + description fields on admin MCP server create form
- Connect/Connected state in MCP server table
- BYOK Demo page (/tools/byok-demo) showing full OAuth 2.1 PKCE flow
* feat(mcp/byok): redesign OAuth authorize page to match 2-step Connect mockup
- Step 1: L→S logos, requested access checklist, How it works box, Continue button
- Step 2: API key input, Save toggle, Duration pills (1h/24h/7d/30d/until_revoked), security note
- Matches screenshots: white modal on dark bg, progress dots, dark CTA buttons
- Authorize handler now fetches byok_description and byok_api_key_help_url from server registry
- CLAUDE.md: replace SQL snippet with proper DB migration troubleshooting guidance
* fix: address greptile review feedback (greploop iteration 1)
- XSS: escape all user-supplied values in _build_authorize_html() with html.escape()
- Open redirect: validate redirect_uri scheme and URL-encode code/state in redirect
- N+1 query: batch BYOK credential lookup into single find_many() call
- Critical path DB: add 60s TTL in-memory cache to _check_byok_credential()
- Encrypt BYOK credentials at rest using encrypt_value_helper/decrypt_value_helper
* fix(byok): update OAuth popup with LiteLLM logo, MCP title suffix, remove emojis
* fix(byok-demo): fix token endpoint URL (/v1/mcp/oauth/token not /v1/mcp/token)
* feat(byok): inject stored BYOK credential as mcp_auth_header on tool execution
* feat(byok): use contextvars to inject per-user credential into OpenAPI tool closures; remove byok-demo from LiteLLM UI
OpenAPI tools have auth headers baked into their closures at registration time. BYOK servers have
no static auth token, so per-user credentials were never reaching the HTTP calls.
Fix: add _request_auth_header ContextVar in openapi_to_mcp_generator.py. create_tool_function now
reads this var at call time and overrides the Authorization header if set. execute_mcp_tool resolves
the MCP server and performs BYOK checks before the local-tool dispatch branch, then sets the
ContextVar around _handle_local_mcp_tool so the credential flows into the HTTP request.
Also remove the /tools/byok-demo page from the LiteLLM UI dashboard — the demo lives at
~/Downloads/litellm-byok-demo/index.html (served separately on port 8080).
* fix: address greptile review feedback (greploop iteration 2)
- Cache invalidation: add _invalidate_byok_cred_cache() and call it after
store_user_credential() in both token endpoint and management endpoint
- Unbounded cache: add _BYOK_CRED_CACHE_MAX_SIZE=4096 with clear-on-overflow
- Unbounded auth codes: add _AUTH_CODES_MAX_SIZE=1000 with 503 on overflow
- Double DB query: merge _check_byok_credential + _get_byok_credential into
single _get_byok_credential call; raise 401 inline if None returned
- Sidebar: remove byok-demo entry (page was deleted in prior commit)
- JWT comment: document why byok_session HS256 token can't be used as proxy auth
* fix: address greptile review feedback (greploop iteration 3)
- auth_type: pre-format Authorization header (Bearer/ApiKey/Basic) in server.py
before setting ContextVar so openapi_to_mcp_generator respects server auth_type
- cache invalidation on delete: call _invalidate_byok_cred_cache after
delete_user_credential so stale True entries don't persist for 60s
- ContextVar guard: only set _request_auth_header when mcp_auth_header is set,
avoiding unnecessary ContextVar overhead on non-BYOK tool calls
* fix: address greptile review feedback (greploop iteration 4)
- Unified credential cache: store actual credential value (Optional[str])
instead of just bool so _get_byok_credential also benefits from caching —
eliminates the DB hit on every BYOK tool call within the 60s TTL window
- Extracted _write_byok_cred_cache() helper for consistent cache writes
- Replaced has_user_credential with get_user_credential in _check_byok_credential
so one DB call satisfies both existence check and value retrieval
- Remove false 'encrypted at rest' claim from OAuth HTML and ByokCredentialModal
* Update tests/test_litellm/proxy/_experimental/mcp_server/test_byok_oauth_endpoints.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update tests/test_litellm/proxy/_experimental/mcp_server/test_byok_oauth_endpoints.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Adds provider documentation for bedrock_mantle including:
- API key and region configuration
- Supported models with pricing table
- SDK, streaming, and async usage examples
- LiteLLM Proxy config and usage
- Added to Bedrock category in sidebar
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
filter_helpers.ts was not populating the Organization ID filter dropdown
(always empty). TeamVirtualKeysTable was showing the team's org for all
keys instead of each key's own org.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The /key/list API returns `org_id` (the Pydantic field name), but the UI
was reading `organization_id`, causing the Organization field to always
show "Not Set" and the Organization ID filter to never match.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
