Commit graph

55766 commits

Author SHA1 Message Date
Yujong Lee
7e27432d61 test(proxy): cover each traffic shape through the Rust forwarding gateway
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 15:15:08 +09:00
Yujong Lee
f62c02a936 feat(deploy): set LITELLM_RUST from the charts' rustGateway toggle
Some checks are pending
LiteLLM Rust DB / db-queries (push) Waiting to run
LiteLLM Rust / rust-lint (push) Waiting to run
LiteLLM Rust / rust-test (push) Waiting to run
LiteLLM Rust / rust-wheel (push) Waiting to run
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 13:01:20 +09:00
Yujong Lee
270dd76891 feat(deploy): add a Rust forwarding toggle to the gateway component chart
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 13:01:20 +09:00
Yujong Lee
7a3612ed56 feat(deploy): add a Rust forwarding toggle to the single-deployment chart
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 13:01:20 +09:00
Yujong Lee
a948afa682 feat(proxy): opt into Rust forwarding with LITELLM_RUST
One switch turns on every optional Rust path, so forwarding reads the same
variable through the same parser as the rust bridge instead of its own
LITELLM_RUST_GATEWAY. An unrecognized value now leaves forwarding off, like
every other Rust path, instead of failing startup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 13:01:20 +09:00
Yujong Lee
1949baf3de fix(proxy): restore the client's peer behind Rust instead of trusting the Rust hop
Python now sees each request as if the client had connected to it directly.
The Rust hop sends the real TCP peer in x-litellm-rust-peer, an app middleware
puts it back into the ASGI scope, and uvicorn's ProxyHeadersMiddleware then
runs with the same trust settings Python alone would use. Server-level proxy
headers are off in supervised mode for uvicorn and gunicorn, since they would
judge the loopback hop. The trust-list merge that always trusted 127.0.0.1 is
gone, so operators keep the exact FORWARDED_ALLOW_IPS behavior they had.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 13:01:20 +09:00
Yujong Lee
a3e353c726 test(proxy): expect the listener fd argument in uvicorn init calls
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 13:01:20 +09:00
Yujong Lee
4cb3cdbb03 feat(proxy): supervise the Rust forwarding server from gateway.launch
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 13:01:20 +09:00
Yujong Lee
fc2fce7102 refactor(proxy): declare rust gateway unions as explicit type aliases
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 13:01:20 +09:00
Yujong Lee
8b7cdef990 refactor(proxy): pass the supervised children's environment into supervise
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 13:01:20 +09:00
Yujong Lee
077d903211 refactor(proxy): resolve the rust gateway mode once and make the supervisor policy a pure state machine
LITELLM_RUST_GATEWAY is now the only switch. rust_gateway_mode reads the
environment once and the entry points pass the result down. Shutdown
order and exit codes live in step(), child commands in python_spec and
rust_spec. The binary is looked up in the environment's scripts directory
only, and an operator's FORWARDED_ALLOW_IPS is kept with the Rust hop
added instead of being rejected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 13:01:20 +09:00
Devin AI
d0e1bebce0 fix(proxy): reject incompatible forwarding trust configuration 2026-11-03 13:01:20 +09:00
Devin AI
48997c340c fix(proxy): enforce specified loopback-only forwarding trust 2026-11-03 13:01:20 +09:00
Devin AI
21b2087981 fix(proxy): retain configured trusted proxies behind Rust 2026-11-03 13:01:20 +09:00
Devin AI
85e87e25e2 test(proxy): cover gateway shutdown and child failures 2026-11-03 13:01:20 +09:00
Devin AI
c0ec348583 fix(proxy): preserve gateway metrics and default-port fallback 2026-11-03 13:01:20 +09:00
Devin AI
608a5c3e57 feat(proxy): supervise the opt-in Rust forwarding server 2026-11-03 13:01:20 +09:00
Devin AI
cf48dc1769 fix: restore utils extra in gateway workflow 2026-11-03 13:01:20 +09:00
Devin AI
4ce59b6112 build: rename gateway distribution to litellm-proxy 2026-11-03 13:01:20 +09:00
Yujong Lee
fb3b66877d fix(packaging): keep gateway sdist includes in the manifest and fail when its rust path moves
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 13:01:20 +09:00
Devin AI
89f6ab22a1 fix(ci): budget for both native binaries before image scanning 2026-11-03 13:01:19 +09:00
Devin AI
60456676ca fix(ci): allow full Python gates and harness checks to finish 2026-11-03 13:01:19 +09:00
Devin AI
efb1ca1f6f fix(packaging): use dev builds for editable gateway installs 2026-11-03 13:01:19 +09:00
Devin AI
695284c4d5 fix(packaging): verify the first-party gateway license 2026-11-03 13:01:19 +09:00
Devin AI
f480b61f13 fix(packaging): share gateway wheels across builds and test shards 2026-11-03 13:01:19 +09:00
Devin AI
2bd211041a feat(packaging): ship the Rust gateway with the proxy extra 2026-11-03 13:01:19 +09:00
Yujong Lee
7fbdc9a38d test(gateway): read the many-header response with the raw client
The hyper test client has the same 100-header default the fix lifts, so it rejected the relayed response itself

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 13:01:19 +09:00
Yujong Lee
37c27147d8 fix(gateway): accept Python responses with more than 100 headers
hyper's client rejects a response past 100 headers, which turned a completed request into a 502. LiteLLM copies every provider header into its response and uvicorn sets no limit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 12:54:17 +09:00
Yujong Lee
ad7b8d9da5 fix(gateway): bound the forwarding shutdown drain by Python's graceful timeout
HTTP streams kept Rust draining with no limit and WebSockets were cut after a fixed 5s. Both now wait for Python to end them, up to GRACEFUL_SHUTDOWN_TIMEOUT plus the teardown headroom Python uses, so the closes Python sends reach clients

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 12:54:12 +09:00
Yujong Lee
c2833d5fd7 fix(gateway): disable Nagle on both forwarding hops
uvicorn sets TCP_NODELAY on every connection through asyncio, so token-by-token SSE and realtime audio deltas went out at once. Without it the forwarder can hold each small write for an ACK on real networks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 12:24:20 +09:00
Yujong Lee
d27528b10d fix(gateway): keep the public Host for HTTP/2 clients
HTTP/2 requests carry the host only in :authority, which the upstream URI rewrite replaced, so Python saw 127.0.0.1:<port> and built OAuth and redirect URLs from it. HTTP/2 serving is now declared by the crate instead of arriving through workspace feature unification

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 12:23:14 +09:00
Yujong Lee
bf3bab34ad fix(gateway): read Python's KEEPALIVE_TIMEOUT for the forwarding pool
gateway.launch and proxy_cli both hand KEEPALIVE_TIMEOUT to uvicorn, and the Rust child
inherits that environment, so reading the same variable keeps the two timeouts in step
with no supervisor change

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 11:58:00 +09:00
Yujong Lee
ba1f6164b7 refactor(gateway): move forwarding into a gateway-forward crate with protocol tests
The forwarder is a reverse proxy, not a mounted endpoint, so it gets its own crate
with a socket-level test harness and a property test that any generated request and
response cross it unchanged

The tests found two bugs, fixed here. Pooled connections to Python outlived uvicorn's
keep-alive, so a request could land on a connection Python was closing and get a 502.
They now retire at half of LITELLM_FORWARD_KEEPALIVE_TIMEOUT (default 5s, uvicorn's
default). A GET with a chunked body reached Python empty, because hyper's client drops
an unknown-length GET body unless transfer-encoding is set

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 11:23:47 +09:00
Yujong Lee
1c27bd33f0 feat(gateway): relay WebSocket upgrades to the Python proxy
Hop-by-hop stripping dropped Upgrade and Connection, so realtime and live
WebSocket routes got 404 from Python instead of 101. Ported from #48087
without its x-forwarded-for append or handshake timeouts, since Python
directly has neither. Open sockets get a five second drain on shutdown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:08:59 +09:00
Yujong Lee
b99ea3a0aa fix(gateway): pass forwarding headers through and carry the peer in x-litellm-rust-peer
Appending to X-Forwarded-For made Rust look like one more proxy, so Python code
that compares or counts X-Forwarded-For entries behaved differently once Rust
was enabled. Rust now leaves every forwarding header as the client sent it and
hands Python the TCP peer in a private header, replacing any client-sent copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 08:49:33 +09:00
Yujong Lee
7a581ae3f1 fix(gateway): log the upstream error when forwarding returns 502
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 06:16:04 +09:00
Devin AI
e361f7acae feat(gateway): forward HTTP requests to the Python proxy 2026-11-01 23:00:07 +00:00
yuneng-jiang
60fc121ce8
test(a2a): replace the third-party agent e2e test with an integration test (#49541)
Some checks failed
unit / auth-checks (push) Blocked by required conditions
unit / budgets (push) Blocked by required conditions
unit / custom-logging (push) Blocked by required conditions
unit / db-and-spend (push) Blocked by required conditions
unit / endpoints-and-responses (push) Blocked by required conditions
unit / enterprise-routing (push) Blocked by required conditions
unit / guardrails-hooks (push) Blocked by required conditions
unit / jwt-and-keys (push) Blocked by required conditions
unit / key-generation (push) Blocked by required conditions
unit / logging-misc (push) Blocked by required conditions
unit / mcp-oauth (push) Blocked by required conditions
unit / proxy-runtime (push) Blocked by required conditions
unit / proxy-server-core (push) Blocked by required conditions
unit / root (push) Blocked by required conditions
unit / proxy-utils (push) Blocked by required conditions
unit / ui-unit (push) Waiting to run
unit / docs (push) Waiting to run
unit / helm (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.0.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.25.0 compatibility (push) Blocked by required conditions
unit / redis-py 5.3.1 compatibility (push) Blocked by required conditions
unit / redis-py 6.4.0 compatibility (push) Blocked by required conditions
unit / redis-py 7.4.1 compatibility (push) Blocked by required conditions
unit / redis-py 8.0.1 compatibility (push) Blocked by required conditions
unit / coverage (push) Blocked by required conditions
unit / unit passed (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
LiteLLM Rust DB / db-queries (push) Has been cancelled
VS Code Extension / vscode-extension (push) Has been cancelled
Deletes the A2A e2e test that registered a third-party agent, which depended on a site the project does not control, along with its helpers and its coverage registry row. Adds an integration test where a scripted A2A agent receives a skill call as a data part and returns a data artifact, asserting the agent and the caller each get theirs unchanged
2026-10-31 18:59:04 -07:00
yuneng-jiang
1b3da63703
test: move Sail, Weave and Presidio e2e tests to integration tests (#49533)
* test: move Sail and Weave e2e tests to integration tests

The Sail service_tier and completion_window cases now run against the scripted upstream in tests/integration/pricing, asserting the outbound body and the billed rates for chat, responses and messages

The key-scoped Weave delivery cases now run against an OTLP collector in tests/integration/observability. The case where only the otel callback is configured is skipped with a BUG reason

Removes the e2e copies, the Weave read-back client, the Sail registry rows, the Weave docs paragraph and the unused devin MCP server entry from the stage mirror config

* test: move Presidio masking e2e tests to integration tests

Moving them to integration tests. The pre_call, post_call and default scope cases now run against a mock Presidio analyzer and anonymizer that follow the real wire contract, and a scripted model upstream, on chat completions, messages and responses, with and without streaming

The spend log cases check the stored response and the guardrail record with a proxy that stores prompts in spend logs. The e2e copy, its Presidio guardrail params body, its registry rows and the doc paragraphs naming the two PRESIDIO variables are removed
2026-10-31 18:31:01 -07:00
devin-ai-integration[bot]
187f936cb3
fix(proxy)!: support reload with multiple workers (#45909)
* fix(proxy): support reload with multiple workers

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): stop the reload worker pool on a broken edit instead of respawning it

Workers now load the app through a factory that exits with uvicorn's
STARTUP_FAILURE code on any import error, so the pool stops and the
reloader waits for the next save, the same as single-worker reload.
Multi-worker reload needs uvicorn>=0.50.0 for that, so older versions
keep the single-worker reload with a warning.

* test(integration): stop the reload test's proxy through the harness shutdown helper

Main's TQ010 rule rejects a raw .terminate() under tests/integration, so the multi-worker reload test now calls OwnedProxy.begin_shutdown(), which sends the same SIGTERM

* test(proxy): cover the reload worker pool end to end and edit the config the proxy loads

* docs(agents): scope the two-worker reload QA rule to uvicorn proxies that use Redis

* fix(proxy): keep the original traceback when a reloaded worker fails to import the app

* docs(agents): point both router and coordination Redis at the QA Redis and cover single-worker trees

* fix(proxy): stop the reload worker pool when the reloader exits and give reloads a longer worker healthcheck

Closing the terminal or pressing Ctrl-\ killed the reloader and left the pool parent holding the port while it respawned workers that could not start. The pool now stops once its parent reloader is gone. The reload pool also defaults the worker healthcheck to 30 seconds unless --timeout_worker_healthcheck is set, since uvicorn's 5 second default killed workers that were still booting on a reload.

* fix(proxy): watch the reloader through its process sentinel so the pool also stops on Windows

* fix(proxy): carry the graceful shutdown timeout into the reload worker pool

* fix(proxy): stop the reload worker pool on Ctrl-\ instead of waiting out a dying worker's healthcheck

* test(proxy): skip the startup-failure reload tests on uvicorn without a startup failure code

* docs(agents): keep the QA proxy rule to two workers, Redis, and reload

* fix(proxy): keep Windows on single-worker reload and pass unlisted uvicorn args to the reload pool

* fix(proxy): stop reload workers when their pool parent dies

* test(proxy): move the reload pool process tests to the integration suite

The unit test guide keeps tests/unit in-process, so the subprocess pool cases now live in tests/integration/configuration and use the harness port allocator and process helpers

* docs(agents): keep main's proof rules and inline the multi-worker QA command

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): exit reload workers as startup failures when the app factory raises

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): check the reload pool serves the overridden worker app

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-01 00:57:32 +00:00
devin-ai-integration[bot]
a1c5829f6b
fix(bedrock): raise GovCloud West Llama 3 max output to the model card 8K (#49543)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-01 00:30:38 +00:00
devin-ai-integration[bot]
f4f5e2c80f
fix(cost-map): add 2026-11-04 deprecation_date to together_ai Qwen3.8-2.4T-A95B (#49542)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-01 00:28:59 +00:00
yuneng-jiang
68dbf54b09
chore(e2e): remove the unused arize_phoenix callback from the stage mirror config (#49538)
Phoenix is unused by the e2e suite and has no account behind it. Drops the callback from the stage mirror gateway config, and updates the OTEL trace test's failure message and the e2e contributing notes that still pointed at the arize_phoenix preset.
2026-10-31 17:21:06 -07:00
yucheng-berri
0f4d5c6ad0
feat(ovalix)!: scan files, tool calls and tool results (#49309)
* feat(ovalix): file and tool scanning, app discovery from the key alias

Mirror of BerriAI/litellm#34132 at 5ffc268585, rebuilt on main.

* fix(ovalix): keep inspecting requests whose key has no user

The tracker allows a checkpoint call with an empty actor without inspecting
it, so a key with no user (or a /v1/messages or /v1/responses request that
carries its own metadata object) got no Ovalix enforcement. Fall back to
"unknown" as the guardrail did before, so every request is inspected.

* refactor(ovalix): check texts without mutation and test discovery through tracker requests

- _check_texts builds its result from per-text verdicts checked newest first by _check_text
- discovery and routing cache unit tests answer the tracker routes instead of replacing private methods
- integration test adds streaming cells and a tracker outage during a concurrent burst

* fix(ovalix): resolve tool result names in one pass

Each tool result rescanned every earlier message for its call id, so a request with thousands of
tool results spent seconds in the event loop. Declarations are now indexed once per request and
each result takes the nearest earlier one by bisection

* test(ovalix): annotate new test bindings and build the burst plan with product

* fix(ovalix): drop app discovery from the key alias

A config with no application_id no longer boots in discovery mode, where
the [Name] prefix of the key alias picked the Ovalix application and so its
policy. Anyone who could rename a key could move it to a weaker app, or to
an unknown name the tracker created with its default policy. The guardrail
now needs application_id at startup again, as on main, and every checkpoint
call goes by the configured application and checkpoint ids.

Removes enable_routing_cache and fail_if_no_application, the
app_name_regex and resolve_application tracker calls, and the per-alias
routing cache. Restores main's startup check that a requested pre_call or
post_call hook has its checkpoint id.

---------

Co-authored-by: Shalom Jamil <shalom@ovalix.io>
2026-10-31 17:01:12 -07:00
Mateo Wang
901b7f762f
test(mcp): cancel disconnected HTTP tool handlers (#49436)
* test(mcp): cancel disconnected HTTP tool handlers

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(mcp): verify owned peer disconnect cleanup

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 16:41:49 -07:00
moe-berri
28679bb9c3
fix(chat): describe general chat and deep research on the chat home (#49531)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 16:39:33 -07:00
Mateo Wang
8c46b2c0d6
test(router): trace the deployment index lock in the file that owns it (#49520)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 16:38:53 -07:00
Kerry Lu
137fd7cd66
fix(docs): replace stale docs/proxy/enterprise links with the current docs pages (#49491)
* fix(ui): point the Spend Per entity docs link at the cost tracking page

* fix(docs): replace the remaining docs/proxy/enterprise links with current pages

* revert(terraform): leave the team tags docs link for a code-owned follow-up

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-31 16:38:37 -07:00
ishaan-j-berri
6105aaa63e
Configure full Bedrock guardrail assessments (#49217) (#49470) 2026-10-31 23:37:45 +00:00
ishaan-j-berri
2751cfcad3
Add guardrails.apply SDK helper (#49217) (#49469) 2026-10-31 23:37:45 +00:00