perf(plugin): offload runtime pip install to thread pool with asyncio.to_thread

install_frontmatter_requirements() calls subprocess.check_call() to
run pip install, which blocks the event loop for 500ms to 60+ seconds
depending on network speed and number of packages. Two runtime call
sites are inside async functions that execute when users load tools
or functions for the first time:

- load_tool_module_by_id() — first load of a tool with requirements
- load_function_module_by_id() — first load of a function with requirements

During a pip install at runtime, every concurrent user's request
stalls because the event loop is frozen waiting for the subprocess.

Wrap the 2 runtime call sites in asyncio.to_thread() so the
subprocess runs in the thread pool. The event loop remains free
to serve other requests during package installation.

The startup call site (install_tool_and_function_dependencies) is
intentionally left synchronous — the server is not yet accepting
requests during lifespan(), so there is no concurrency benefit.

Benchmark (pip install --dry-run requests, 3 trials averaged):
- BEFORE: max jitter 487ms, 1 blocked ping per trial
- AFTER:  max jitter 1ms, 0 blocked pings (487x improvement)
This commit is contained in:
silentoplayz 2026-06-08 08:16:14 -04:00
parent b1d40f3409
commit fbfe3c074c

View file

@ -1,5 +1,7 @@
from __future__ import annotations
import asyncio
import logging
import os
import re
@ -214,7 +216,7 @@ async def load_tool_module_by_id(tool_id, content=None):
else:
frontmatter = extract_frontmatter(content)
# Install required packages found within the frontmatter
install_frontmatter_requirements(frontmatter.get('requirements', ''))
await asyncio.to_thread(install_frontmatter_requirements, frontmatter.get('requirements', ''))
module_name = f'tool_{tool_id}'
module = types.ModuleType(module_name)
@ -258,7 +260,7 @@ async def load_function_module_by_id(function_id: str, content: str | None = Non
await Functions.update_function_by_id(function_id, {'content': content})
else:
frontmatter = extract_frontmatter(content)
install_frontmatter_requirements(frontmatter.get('requirements', ''))
await asyncio.to_thread(install_frontmatter_requirements, frontmatter.get('requirements', ''))
module_name = f'function_{function_id}'
module = types.ModuleType(module_name)