4 commits

Author	SHA1	Message	Date
Sameer Kankute	02bce7b393	fix(mcp): honor server_id for REST tool calls with shared upstream URLs (#30184) ... * fix(mcp): honor server_id for REST tool calls with shared upstream URLs When multiple MCP server entries point at the same backend URL and tool name, REST /mcp-rest/tools/call now routes and applies auth from the requested server_id instead of the global unprefixed tool-name mapping. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): classify prefixed REST tool names against full registry Use all registered MCP server prefixes for prefix detection so unauthorized prefixed names still trigger tool_server_mismatch, and reject ambiguous hyphenated REST tool names with server_id. Co-authored-by: Cursor <cursoragent@cursor.com> * test(mcp): cover server_id fallback for unresolved prefixed REST tool names execute_mcp_tool left the prefix-retry and requested-server fallback branches uncovered, dropping diff coverage below the project target. Add a regression test for a REST call that passes server_id with a prefixed tool name that resolves to no managed tool; it must still dispatch to the server identified by server_id rather than the server named by the prefix. * test(mcp): scope global tool-name mapping mutation with patch.dict * test(mcp): cover server_id guard on prefix-retry tool resolution The prefix-retry branch in execute_mcp_tool re-prefixes the tool name with the requested server's known prefixes when the bare lookup misses. The candidate-found path that assigns mcp_server from that lookup stayed uncovered, so codecov patch coverage remained below the diff target. Add a regression test where the re-prefixed lookup resolves a server whose server_id differs from the requested server_id; the tool_server_mismatch 403 guard must still fire instead of being silently bypassed. * test(mcp): assert requested server credentials injected on cross-server REST routing * perf(mcp): scan registry prefixes only when server_id is supplied * fix(mcp): allow hyphenated upstream tool names when REST server_id is authoritative * perf(mcp): skip registry prefix scan for separator-free REST tool names * test(http_handler): drop httpbin dependence from per-request timeout test The per-request timeout test posted to https://httpbin.org/delay/10 and asserted a Timeout was raised. httpbin's free /delay endpoint intermittently returns 503 even when the /get reachability guard succeeds, so local_testing_part1 flaked on that 503 instead of the expected timeout (failed identically across an initial run and a rerun-from-failed). Serve the slow response from a local ThreadingHTTPServer so the timeout fires deterministically with no third-party network dependence. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-12 07:25:53 -07:00
Sameer Kankute	7519e37d26	feat(proxy): enforce key/team guardrails on bedrock passthrough routes (#30194) ... * feat(proxy): enforce key/team guardrails on bedrock passthrough routes /bedrock/... passthrough routes silently skipped all guardrail hooks because CallTypes.allm_passthrough_route had no entry in guardrail_translation_mappings. Add a dispatcher (LlmPassthroughRouteHandler) registered for that call type that routes to BedrockPassthroughGuardrailHandler for Bedrock Converse endpoints; wire post_call_success_hook into both the JSON and AWS event-stream response paths in common_request_processing, including full de-anonymization for streaming responses with proportional text distribution across original event-stream delta frames. * refactor(proxy): address greptile feedback on bedrock passthrough guardrails Move botocore event-stream logic from proxy/ to BedrockPassthroughGuardrailHandler.de_anonymize_converse_stream; _handle_event_stream_allm_passthrough_route becomes a thin provider dispatcher. Pass custom_headers through _handle_non_streaming_allm_passthrough_route so early-return guardrail responses include x-litellm-call-id and related headers. * style: run black on handler and common_request_processing * refactor(proxy): dedupe non-streaming passthrough guardrail handling Replace the inline JSON/eventstream block in the streaming-request branch with a call to _handle_non_streaming_allm_passthrough_route so both paths share one implementation and cannot diverge. * fix(bedrock): preserve trailing bytes when re-encoding converse stream The event-stream re-encoder only emitted parsed frames, so any trailing bytes left after the parse loop (truncated/corrupt final frame, or fewer than 16 bytes after the last complete frame) were silently dropped from the de-anonymized output. Capture and re-append them so the transformer never truncates the stream. * fix(proxy): guard non-dict post-call hook return on bedrock passthrough JSON path * fix(proxy): guard malformed JSON body on bedrock passthrough guardrail path * fix(proxy): close guardrail bypass via tool result text and default-mode post-call guardrails on bedrock passthrough Pre-call extraction only read top-level Converse text blocks, so blocked content placed under toolResult.content[].text was forwarded to Bedrock without the key/team guardrail seeing it. Extraction now walks nested tool result text and write-back mutates the owning block in place. Post-call buffering for passthrough used _has_post_call_guardrails, which excludes event_hook=None guardrails. Those guardrails run at post_call, so their output processing was skipped and the raw upstream body was returned. Add a passthrough-specific predicate that counts them. * refactor(proxy): route bedrock event-stream de-anonymization through llm passthrough dispatcher Remove the hardcoded bedrock provider guard from common_request_processing by delegating event-stream de-anonymization to LlmPassthroughRouteHandler, which resolves the provider from the existing handler registry. Keeps proxy/ provider-agnostic and reuses the same dispatch path as the input and output guardrail handlers. Also log instead of silently dropping the result when post_call_success_hook returns a non-dict on the JSON and event-stream passthrough paths. * fix(proxy): close guardrail bypass on bedrock invoke passthrough routes Pre-call extraction and post-call output processing only handled Converse shapes, so /bedrock/model/{modelId}/invoke and invoke-with-response-stream returned unguarded. An authenticated caller could move blocked content into an InvokeModel payload and skip the key/team guardrail entirely. Non-Converse Bedrock routes now fall back to the generic passthrough handler, which scans the full request and response payloads so blocking guardrails still run, matching how other passthrough providers are guarded. * fix(proxy): keep non-bedrock passthrough streams streaming under post-call guardrails * fix(bedrock): scan non-text converse blocks for passthrough guardrails Key/team guardrails on bedrock converse passthrough only saw top-level text blocks, so a caller could hide prompt content in toolUse.input or toolResult.content[].json and have it forwarded to Bedrock without the configured guardrail inspecting it, bypassing blocking guardrails by default. Walk those arbitrary-JSON subtrees and write masked values back in place. Extend the non-streaming converse response path to the equivalent model-output fields (toolUse.input, reasoningContent text and citationsContent text) while leaving structural values such as reasoning signatures and citation sources untouched. * fix(bedrock): make passthrough guardrail string collection iterative and type-safe Rewrite _collect_strings with an explicit stack so it no longer recurses, satisfying the recursive-function CI guard, and widen the holder container type so mypy accepts indexing JSON nodes by str or int keys. * fix(proxy): scope passthrough post-call guardrail buffering to the request Buffering the Bedrock event stream into a single non-streaming response was gated on whether any post_call guardrail existed globally, so every converse-stream request lost streaming once any post_call guardrail was registered, even for keys that did not reference it. Mirror the gate used by post_call_success_hook (should_run_guardrail against the request's merged guardrails) so only requests whose key/team actually trigger a post_call guardrail are buffered. * fix(bedrock): guardrail non-text converse stream deltas on passthrough de_anonymize_event_stream only routed delta.text through the post-call guardrail, so model output streamed in reasoningContent.text, toolUse.input or citationsContent.content[].text was forwarded raw and skipped masking/blocking. Collect every user-visible text field per contentBlockDelta, concatenate per logical stream so split mask tokens still reassemble, run them through the hook, then redistribute the guardrailed text back into the matching delta fields. This brings streaming coverage in line with the non-streaming Converse output handler. * refactor(proxy): keep bedrock event-stream content-type detection in llms Move the vnd.amazon.eventstream content-type check out of the proxy passthrough path into BedrockPassthroughGuardrailHandler via the LlmPassthroughRouteHandler dispatcher, so proxy code stays provider-agnostic. Also patch the actually-called _has_post_call_guardrails_for_passthrough in the malformed-body regression test instead of the unused _has_post_call_guardrails. * fix(proxy): forward upstream headers on bedrock guardrail passthrough responses Mirror the non-guardrail passthrough path by merging the upstream response headers (via get_response_headers) into the guardrailed non-streaming and event-stream responses, so headers like x-amzn-requestid survive when a post-call guardrail rewrites the body. Drop the stray fastapi HTTPException import from the SDK-tree handler test in favor of a local sentinel exception. * fix(bedrock): scan tool definitions and additional request fields for passthrough guardrails Converse passthrough guardrails only scanned system and message content, so a key holder could route blocked or PII text through toolConfig tool names, descriptions and input schemas or through additionalModelRequestFields, all of which are still forwarded to Bedrock. Collect strings from those fields too so key/team guardrails inspect and rewrite them, matching how the chat-completions path forwards tool definitions to guardrails. * fix(bedrock): log instead of silently dropping passthrough guardrail edge cases * test(local): skip httpbin timeout probe when the service returns 5xx local_testing_part1 was failing on test_post_delay_exceeds_per_request_timeout_raises because httpbin.org/delay/10 intermittently answers 503 instead of delaying, so HTTPHandler.post raised MaskedHTTPStatusError rather than the expected Timeout. The test already means to skip when httpbin is unavailable, but its guard only probed GET /get and ignored a server error on the delay endpoint. Treat a 5xx from httpbin as 'service unavailable' and skip, which is outside this repo's control, while still asserting Timeout when httpbin genuinely delays. * fix(proxy): set content-type on buffered bedrock passthrough event-stream responses * fix(bedrock): skip passthrough output write-back when guardrail returns no texts * fix(proxy): apply response-headers hook on guardrailed bedrock passthrough responses * refactor(bedrock): import event-stream crc32 from binascii not botocore internals * fix(proxy): scope bedrock passthrough stream buffering to de-anonymizable endpoints Only buffer a passthrough event stream into a non-streaming response when the resolved provider and endpoint actually have an event-stream guardrail handler that can rewrite frames (Bedrock converse-stream). Other Bedrock event-stream endpoints such as invoke-with-response-stream keep streaming, since the Converse handler leaves their frames untouched and buffering would silently break the streaming contract for no content change. * test(ui-e2e): re-issue deep-link navigation when auth bootstrap drops the page param navigateToPage deep-links to /ui?page=<page> then proceeds once the network settles, but a fresh load can race the auth bootstrap: the app momentarily treats the session as anonymous, bounces through /ui/login, and returns to the default Virtual Keys page with the ?page= query param dropped. The helper never checked where it actually landed, so any single bounce left callers asserting against the wrong page and timing out (mcpServers, modelHub, addModel). Confirm the requested page is what rendered and re-issue the navigation when it was clobbered; auth is warm by the second load so the param sticks. Migrated path routes are left alone since they intentionally leave the legacy root. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-12 07:23:29 -07:00
Ishaan Jaffer	e8461b5b97	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
ishaan-berri	a588f76789	Litellm ishaan april15 2 (#25828) ... * [Test] Add Azure async chat completion timeout test. WIP * Capture TTFT for /v1/messages streaming responses The pass-through streaming path for /v1/messages (Anthropic, Bedrock, Vertex AI, Azure AI, Minimax) logged completion_start_time only after the entire stream finished. async_success_handler then fell back to end_time, making TTFT equal to total duration or null in the UI and Prometheus. Record the timestamp of the first chunk in async_sse_wrapper and propagate it to model_call_details before the logging handler runs, so gen_ai.response.time_to_first_token reflects the real first-chunk latency. Fixes #25598 * [Refactor] Implement timeout resolution logic in completion function add fetch ``request_timeout`` from litellm_settings * remove stale test case * remove extra print statement * default request timeout value in constants to 600s to match timeout defaults handled in the proxy * fix request timeout if using default value from constants.py * update code structure, test cases * only override if the global timeout sets timeout to 6000s * update code structure, move hard coded values to const and make the reslve function readable by moving fallback logic to a seperate function * modify default timeout values, replacing hard coded ones with default values defined --------- Co-authored-by: harish876 <harishgokul01@gmail.com> Co-authored-by: Joaquin Hui Gomez <joaquinhuigomez@users.noreply.github.com>	2026-04-15 18:42:23 -07:00