mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-11 03:38:38 +00:00
56009d32dd
Ports the shunt technique for Claude Code (https://engineering.atspotify.com/2026/9/portal-by-spotify-cut-my-claude-code-token-usage-by-90) into the auto router, server-side, so it works for any client with no plugin to install. shunt intercepts large file reads and boilerplate generation at the client with PreToolUse hooks and hands them to a cheap worker model. This makes the same decision on the proxy: a new always-on ShuntGuardrail injects bulk_read and code_write tool definitions pre-call, then post-call rewrites large Read/Bash/bulk_read/code_write tool_use blocks into a Bash command carrying shunt's own wc -l conditional, so small files still get read directly and only large ones are delegated. Two new endpoints, /v1/bulk_read and /v1/code_write, run the worker call through llm_router.acompletion with shunt's verbatim system prompts, so worker spend is tracked against the calling key and team like any other request. The guardrail arms per request off an auto-router marker's own litellm_params (auto_router_shunt_min_lines and the two worker-model fields), the same shape auto_router_compression already uses, so no guardrails: config entry is needed. It stays inert when the resolved model has no shunt config. In the UI, "Shunt" is one entry in the existing preset dropdown. Its tiers resolve through fallback chains against whatever models the proxy actually has, so it greys out only when there are no chat models at all, never merely because the named models are absent, and all three settings stay editable under Advanced. Streaming buffers the whole response before rewriting, matching tool_permission.py, because input_json_delta fragments split mid-token. An unparseable stream passes through untouched: shunt is an optimization, not a safety control, so a request should never fail because the rewrite could not run. The CLI's Bash allow rules only reduce prompts for the generated commands. Claude Code matches rule text before the first wildcard with no host-aware matching, so they cannot pin the destination, and the docs recommend a PreToolUse hook where a real boundary is needed. |
||
|---|---|---|
| .. | ||
| litellm-dashboard | ||
| Dockerfile | ||
| nginx.conf | ||