One switch turns on every optional Rust path, so forwarding reads the same
variable through the same parser as the rust bridge instead of its own
LITELLM_RUST_GATEWAY. An unrecognized value now leaves forwarding off, like
every other Rust path, instead of failing startup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Python now sees each request as if the client had connected to it directly.
The Rust hop sends the real TCP peer in x-litellm-rust-peer, an app middleware
puts it back into the ASGI scope, and uvicorn's ProxyHeadersMiddleware then
runs with the same trust settings Python alone would use. Server-level proxy
headers are off in supervised mode for uvicorn and gunicorn, since they would
judge the loopback hop. The trust-list merge that always trusted 127.0.0.1 is
gone, so operators keep the exact FORWARDED_ALLOW_IPS behavior they had.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LITELLM_RUST_GATEWAY is now the only switch. rust_gateway_mode reads the
environment once and the entry points pass the result down. Shutdown
order and exit codes live in step(), child commands in python_spec and
rust_spec. The binary is looked up in the environment's scripts directory
only, and an operator's FORWARDED_ALLOW_IPS is kept with the Rust hop
added instead of being rejected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The hyper test client has the same 100-header default the fix lifts, so it rejected the relayed response itself
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hyper's client rejects a response past 100 headers, which turned a completed request into a 502. LiteLLM copies every provider header into its response and uvicorn sets no limit
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HTTP streams kept Rust draining with no limit and WebSockets were cut after a fixed 5s. Both now wait for Python to end them, up to GRACEFUL_SHUTDOWN_TIMEOUT plus the teardown headroom Python uses, so the closes Python sends reach clients
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
uvicorn sets TCP_NODELAY on every connection through asyncio, so token-by-token SSE and realtime audio deltas went out at once. Without it the forwarder can hold each small write for an ACK on real networks
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HTTP/2 requests carry the host only in :authority, which the upstream URI rewrite replaced, so Python saw 127.0.0.1:<port> and built OAuth and redirect URLs from it. HTTP/2 serving is now declared by the crate instead of arriving through workspace feature unification
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gateway.launch and proxy_cli both hand KEEPALIVE_TIMEOUT to uvicorn, and the Rust child
inherits that environment, so reading the same variable keeps the two timeouts in step
with no supervisor change
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The forwarder is a reverse proxy, not a mounted endpoint, so it gets its own crate
with a socket-level test harness and a property test that any generated request and
response cross it unchanged
The tests found two bugs, fixed here. Pooled connections to Python outlived uvicorn's
keep-alive, so a request could land on a connection Python was closing and get a 502.
They now retire at half of LITELLM_FORWARD_KEEPALIVE_TIMEOUT (default 5s, uvicorn's
default). A GET with a chunked body reached Python empty, because hyper's client drops
an unknown-length GET body unless transfer-encoding is set
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hop-by-hop stripping dropped Upgrade and Connection, so realtime and live
WebSocket routes got 404 from Python instead of 101. Ported from #48087
without its x-forwarded-for append or handshake timeouts, since Python
directly has neither. Open sockets get a five second drain on shutdown.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Appending to X-Forwarded-For made Rust look like one more proxy, so Python code
that compares or counts X-Forwarded-For entries behaved differently once Rust
was enabled. Rust now leaves every forwarding header as the client sent it and
hands Python the TCP peer in a private header, replacing any client-sent copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Deletes the A2A e2e test that registered a third-party agent, which depended on a site the project does not control, along with its helpers and its coverage registry row. Adds an integration test where a scripted A2A agent receives a skill call as a data part and returns a data artifact, asserting the agent and the caller each get theirs unchanged
* test: move Sail and Weave e2e tests to integration tests
The Sail service_tier and completion_window cases now run against the scripted upstream in tests/integration/pricing, asserting the outbound body and the billed rates for chat, responses and messages
The key-scoped Weave delivery cases now run against an OTLP collector in tests/integration/observability. The case where only the otel callback is configured is skipped with a BUG reason
Removes the e2e copies, the Weave read-back client, the Sail registry rows, the Weave docs paragraph and the unused devin MCP server entry from the stage mirror config
* test: move Presidio masking e2e tests to integration tests
Moving them to integration tests. The pre_call, post_call and default scope cases now run against a mock Presidio analyzer and anonymizer that follow the real wire contract, and a scripted model upstream, on chat completions, messages and responses, with and without streaming
The spend log cases check the stored response and the guardrail record with a proxy that stores prompts in spend logs. The e2e copy, its Presidio guardrail params body, its registry rows and the doc paragraphs naming the two PRESIDIO variables are removed
* fix(proxy): support reload with multiple workers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): stop the reload worker pool on a broken edit instead of respawning it
Workers now load the app through a factory that exits with uvicorn's
STARTUP_FAILURE code on any import error, so the pool stops and the
reloader waits for the next save, the same as single-worker reload.
Multi-worker reload needs uvicorn>=0.50.0 for that, so older versions
keep the single-worker reload with a warning.
* test(integration): stop the reload test's proxy through the harness shutdown helper
Main's TQ010 rule rejects a raw .terminate() under tests/integration, so the multi-worker reload test now calls OwnedProxy.begin_shutdown(), which sends the same SIGTERM
* test(proxy): cover the reload worker pool end to end and edit the config the proxy loads
* docs(agents): scope the two-worker reload QA rule to uvicorn proxies that use Redis
* fix(proxy): keep the original traceback when a reloaded worker fails to import the app
* docs(agents): point both router and coordination Redis at the QA Redis and cover single-worker trees
* fix(proxy): stop the reload worker pool when the reloader exits and give reloads a longer worker healthcheck
Closing the terminal or pressing Ctrl-\ killed the reloader and left the pool parent holding the port while it respawned workers that could not start. The pool now stops once its parent reloader is gone. The reload pool also defaults the worker healthcheck to 30 seconds unless --timeout_worker_healthcheck is set, since uvicorn's 5 second default killed workers that were still booting on a reload.
* fix(proxy): watch the reloader through its process sentinel so the pool also stops on Windows
* fix(proxy): carry the graceful shutdown timeout into the reload worker pool
* fix(proxy): stop the reload worker pool on Ctrl-\ instead of waiting out a dying worker's healthcheck
* test(proxy): skip the startup-failure reload tests on uvicorn without a startup failure code
* docs(agents): keep the QA proxy rule to two workers, Redis, and reload
* fix(proxy): keep Windows on single-worker reload and pass unlisted uvicorn args to the reload pool
* fix(proxy): stop reload workers when their pool parent dies
* test(proxy): move the reload pool process tests to the integration suite
The unit test guide keeps tests/unit in-process, so the subprocess pool cases now live in tests/integration/configuration and use the harness port allocator and process helpers
* docs(agents): keep main's proof rules and inline the multi-worker QA command
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): exit reload workers as startup failures when the app factory raises
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): check the reload pool serves the overridden worker app
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Phoenix is unused by the e2e suite and has no account behind it. Drops the callback from the stage mirror gateway config, and updates the OTEL trace test's failure message and the e2e contributing notes that still pointed at the arize_phoenix preset.
* feat(ovalix): file and tool scanning, app discovery from the key alias
Mirror of BerriAI/litellm#34132 at 5ffc268585, rebuilt on main.
* fix(ovalix): keep inspecting requests whose key has no user
The tracker allows a checkpoint call with an empty actor without inspecting
it, so a key with no user (or a /v1/messages or /v1/responses request that
carries its own metadata object) got no Ovalix enforcement. Fall back to
"unknown" as the guardrail did before, so every request is inspected.
* refactor(ovalix): check texts without mutation and test discovery through tracker requests
- _check_texts builds its result from per-text verdicts checked newest first by _check_text
- discovery and routing cache unit tests answer the tracker routes instead of replacing private methods
- integration test adds streaming cells and a tracker outage during a concurrent burst
* fix(ovalix): resolve tool result names in one pass
Each tool result rescanned every earlier message for its call id, so a request with thousands of
tool results spent seconds in the event loop. Declarations are now indexed once per request and
each result takes the nearest earlier one by bisection
* test(ovalix): annotate new test bindings and build the burst plan with product
* fix(ovalix): drop app discovery from the key alias
A config with no application_id no longer boots in discovery mode, where
the [Name] prefix of the key alias picked the Ovalix application and so its
policy. Anyone who could rename a key could move it to a weaker app, or to
an unknown name the tracker created with its default policy. The guardrail
now needs application_id at startup again, as on main, and every checkpoint
call goes by the configured application and checkpoint ids.
Removes enable_routing_cache and fail_if_no_application, the
app_name_regex and resolve_application tracker calls, and the per-alias
routing cache. Restores main's startup check that a requested pre_call or
post_call hook has its checkpoint id.
---------
Co-authored-by: Shalom Jamil <shalom@ovalix.io>
* fix(ui): point the Spend Per entity docs link at the cost tracking page
* fix(docs): replace the remaining docs/proxy/enterprise links with current pages
* revert(terraform): leave the team tags docs link for a code-owned follow-up
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>