3 commits

Author	SHA1	Message	Date
Krrish Dholakia	6c21029cb7	feat(sandbox): reuse e2b container across requests when metadata.session_id is set (#31688) ... * feat(sandbox): reuse e2b container across requests when metadata.session_id is set When a client passes `metadata.session_id` in a /chat/completions request alongside a code_interpreter tool, the proxy now routes all requests sharing that session_id to the same sandbox container. State (variables, imports, installed packages) persists across requests within the session. Without a session_id the existing ephemeral behavior is unchanged: one container per agentic loop, deleted immediately after. The sandbox key is derived from session_id rather than a per-request UUID. The cleanup and post-loop hooks skip deletion for session-scoped containers. TTL-based pruning (15 min idle) still applies and refreshes on every use, so an active session never expires mid-use. The session_id-scoped key is registered in all_litellm_params and the proxy strip-list so it never leaks to the upstream LLM provider. * fix(sandbox): scope session sandbox key to API key identity; add per-identity LRU cap Two security issues addressed: 1. Cross-user sandbox isolation: the session_id supplied by the client is now combined with the server-minted user_api_key_hash to form the cache key (format: "{hash}:{session_id}" when authenticated, bare session_id for non-proxy use). Two tenants sharing the same session_id no longer share a sandbox. 2. Bounded session allocation: each API key identity is capped at _SESSION_SCOPED_PER_IDENTITY_CAP (10) live session-scoped containers. When a new session is opened beyond the cap, the least-recently-used entry for that identity is evicted and its sandbox deleted, preventing unbounded accumulation via rotating session IDs. The container cache tuple gains a fourth element (identity: str | None) so eviction can filter by identity without parsing key formats. Tests added for both properties.	2026-06-30 18:58:09 -07:00
Krrish Dholakia	c546b58c09	feat: add chat completions code interpreter loop (#31027) ... * feat: add chat code interpreter loop * fix: address code interpreter pr checks * fix: satisfy strict lint budget * test: cover chat no-op interception * fix: address code interpreter review * fix: clean up agentic loop helpers * fix: preserve agentic loop controls * fix: generalize agentic loop params * fix: carry agentic state via metadata * fix: restore litellm params helpers * refactor: move chat code-interpreter loop out of provider code Dispatch the chat-completions agentic loop from a provider-agnostic helper (litellm/litellm_core_utils/chat_completion_agentic_loop.py) called from main.acompletion, instead of from OpenAI provider files. Register the agentic loop control fields in all_litellm_params so they stay LiteLLM-level and never become provider payload, removing the need for the OpenAIGPTConfig scrubber. No litellm/llms/ files are modified for this feature. * docs: explain chat agentic loop dispatch and litellm-level param registration * style: drop Any annotations and use PEP585 generics to satisfy ruff strict budget * docs: replace module docstring with one-line patch note	2026-06-23 12:13:41 -07:00
Krrish Dholakia	84c1414aef	feat(sandbox): code interpreter interceptor on the Responses API (#30905) ... * feat(sandbox): code interpreter interceptor on the Responses API Route OpenAI's code interpreter to a configured sandbox (e2b) instead of OpenAI's container, with no client change. A client calls /v1/responses with a code_interpreter tool; the interceptor converts it to a function tool so the model emits the code, runs that code in the sandbox via the phase 1 primitive, feeds the result back, and lets the agentic loop continue. Reuses the existing agentic-loop hooks (no new hook methods). The anthropic agentic caller _call_agentic_completion_hooks gains an api_surface argument and a responses execute path (_execute_responses_agentic_plan re-calls aresponses); the responses handler invokes it after transforming the response. Web search and compression interceptors are untouched. Adds an api_base passthrough to the sandbox SDK and a sandbox_tools registry the proxy parses, so the interceptor resolves a named tool to provider/key/base. v0 limitation: no file upload or download yet; stdout and inline results flow back, attaching input files and downloading produced files do not. * feat(sandbox): re-inject code_interpreter_call so the response matches OpenAI The native OpenAI Responses code interpreter returns a code_interpreter_call output item (id, type, status, code, container_id, outputs) alongside the message. The interceptor now re-injects an equivalent item via async_post_agentic_loop_response_hook so a client gets the same response shape whether the code ran in OpenAI's container or the sandbox: build_plan records the executed code and the container id per call, and the post hook inserts the code_interpreter_call before the message in the final response output. * feat(sandbox): support streaming for the code interpreter interceptor A stream:true /v1/responses request with code_interpreter previously broke, because the agentic loop only runs on the non-streaming responses path. The interceptor now forces stream=False in the pre-call hook (so the loop runs in the sandbox) and the responses handler wraps the completed response back into a synthetic stream via MockResponsesAPIStreamingIterator, so the caller still gets SSE. The follow-up call and nested wrapping are guarded by stripping the converted-stream flag from the follow-up request and only wrapping at the outermost call (agentic loop depth 0). * fix(lint): use builtin generics in code interpreter interceptor to satisfy UP006 budget * fix(code-interpreter): gate sandbox execution, delete sandboxes, harden registry Gate the agentic loop on a server-set interception marker and re-check provider scope so an authenticated caller cannot trigger sandbox code execution by naming their own function tool litellm_code_execution; the marker is stripped from client requests at the proxy boundary and only set when the pre-call hook actually converts a native code_interpreter tool. Delete the sandbox once the final response is assembled instead of leaking it until its own timeout, and prune expired cache entries by deleting their containers too. Resolve sandbox params once at create time and reuse them for run and delete. Clear the sandbox-tool registry before re-registering so stale tools do not survive a config reload. * fix(lint): use PEP 604 X | None unions to satisfy UP045 budget * test(code-interpreter): cover execution-error and unparseable-argument tool-call paths * fix(code-interpreter): rewrite forced code_interpreter tool_choice to the function tool * test(sandbox): cover sandbox-tool registry resolution, reload clearing, and secret lookup * test(code-interpreter): cover dict-shaped responses and object-attribute tool-call detection * fix(proxy): strip client-supplied _code_interpreter_interception_converted_stream A client could inject the converted-stream marker to force the completed response to be re-wrapped as a synthetic SSE stream it never requested. Add it to the untrusted root control fields alongside the other agentic loop markers so the proxy strips it at the request boundary. * fix(code-interpreter): isolate sandboxes by server-minted key and clear registry on tool removal Key the per-request sandbox cache on a server-minted random token instead of the caller-controlled litellm_call_id (sourced from the x-litellm-call-id header). Two concurrent requests that send a colliding call id can no longer share a sandbox container and read each other's code or files. The token is minted in the pre-call hook when interception activates, stripped from client requests at the proxy boundary, and survives the server-driven followups so a single request still reuses one sandbox across the agentic loop. Register sandbox tools unconditionally with an empty-list fallback so a config reload that removes sandbox_tools clears the previously registered credentials instead of leaving them resolvable in the process. * refactor(sandbox): swap the tool registry atomically on reload Build the new registry and rebind it in one assignment instead of clearing then repopulating in place, so a concurrent resolve_sandbox_tool can never observe a transiently empty or half-populated registry during a config reload. clear_sandbox_tools now delegates to register_sandbox_tools([]). * fix(code-interpreter): cap caller loop limit and emit OpenAI-shaped outputs Strip max_agentic_loops at the proxy request boundary so an authenticated caller cannot raise the agentic-loop ceiling to drive many upstream model calls and sandbox executions from a single request; the loop stays bounded by the server default. Populate the re-injected code_interpreter_call.outputs with an OpenAI-shaped logs array ([{"type": "logs", "logs": stdout}], or [] when there is no stdout) instead of None, so clients that iterate over outputs or validate the response through the OpenAI SDK's Pydantic model do not break.	2026-06-20 21:12:13 -07:00