scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-14 09:03:16 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 214
litellm/tests/integration/_support
History
Exact
yujonglee 95ecfbbac6
feat(proxy): supervise the opt-in Rust forwarding server (#49606) 
* feat(proxy): supervise the opt-in Rust forwarding server

With LITELLM_RUST on, `litellm` and `python -m gateway.launch` start a
supervisor that runs the Rust gateway on the public port and Python on a
private loopback listener. Python still serves every route and sees each
request as if the client had connected to it directly.

Rust sends the client's TCP peer in `x-litellm-rust-peer`. The supervised
server turns server-level proxy headers off, and a middleware restores
the peer and then applies the operator's own trust, so client IPs,
trusted proxies and every IP-based setting behave the same with Rust on
or off.

The supervisor is transparent to Python. Signals pass through, Python's
exit status is the result, and both children shut down when the
supervisor's end of their stdin closes, so nothing outlives it. Rust
starts only once Python listens. When forwarding cannot start (no
gateway binary, TLS, a Unix socket, Hypercorn or Granian), the entry
point warns and serves from Python alone.

Every forwarding decision lives in litellm/proxy/rust_forward, paired
with the gateway-forward crate. OS differences sit behind a Platform
protocol, with Windows supported for local CLI runs. AGENTS.md files in
the touched folders state the boundary. The `--rust_gateway` flag is
gone in favor of the environment variable.

* refactor(proxy): switch Rust forwarding on with LITELLM_RUST_FIRST

Forwarding gets its own switch, independent of LITELLM_RUST, which keeps
selecting only the Rust paths inside the SDK. Neither turns the other on.

---------

Co-authored-by: narmstrong-berri <narmstrong@berri.ai>
2026-11-11 19:40:00 +00:00
..
__init__.py test: add CircleCI integration contract foundation 2026-09-14 03:30:52 -07:00
a2a_denial_probe.py fix(a2a): log authenticated pre-dispatch agent rejections (#49492) 2026-10-31 15:53:01 -07:00
actors.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
agentic_probe.py fix(responses): honor caller stream flag when provider forces SSE (internal copy of #34095) (#41235) 2026-10-06 03:30:47 +00:00
anthropic_refusal.py fix(caching): stop counting an unrun server tool call mark, and follow #49192 and #35602 in their integration tests (#49468) 2026-10-31 12:27:17 -07:00
anthropic_sse.py fix(router): retry a /v1/messages stream the provider drops before the first content chunk (#44276) 2026-10-05 22:53:39 +00:00
anthropic_thinking.py test: expect the list-position index on streamed thinking blocks in the integration suite (#46618) 2026-10-15 05:52:56 +00:00
asgi.py test(integration): keep owned peers off ephemeral ports (#51019) 2026-11-10 21:19:57 +00:00
async_server.py test(integration): keep owned peers off ephemeral ports (#51019) 2026-11-10 21:19:57 +00:00
aws_control_plane.py test(integration): keep owned peers off ephemeral ports (#51019) 2026-11-10 21:19:57 +00:00
bedrock_discovery.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
bedrock_realtime_peer.py fix(bedrock)!: block realtime replay after an input-first start and reject malformed Nova Sonic session settings (#46936) 2026-10-17 00:29:50 +00:00
bedrock_runtime_peer.py test(ci): fix six CircleCI test regressions on main (#44429) 2026-10-03 20:55:01 -07:00
browser_state.py test: add extension and browser integration contracts 2026-09-16 13:15:53 -07:00
budget.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
claude_code.py test(anthropic): native /v1/messages reasoning integration tests built on a captured Claude Code request (#43361) 2026-10-01 09:12:53 -07:00
client.py fix(caching): read the cache key's model group from the router's metadata for the call (#48917) 2026-10-29 05:35:23 -07:00
codex_vendor.py fix(chatgpt): forward text and max_output_tokens to the Codex backend (#38086) 2026-11-06 06:14:32 -08:00
copilot_session_prometheus_worker.py fix(prometheus): count provider-side 401s in LLM failure and deployment metrics (#46668) 2026-10-26 17:57:06 -07:00
custom_video_provider.py feat(videos): serve custom_provider_map BaseVideoConfig providers on /v1/videos (#49785) 2026-11-02 21:28:46 -08:00
daily_activity.py test(usage): assert per-user entity spend and metadata (#50251) 2026-11-05 20:16:40 +00:00
daily_spend_rows.py fix(ui): group cost optimization cache leakage by model group (#43008) 2026-09-24 23:51:51 -07:00
database.py test(integration): isolate every multi-worker integration suite by default (#46027) 2026-10-12 03:46:06 -07:00
database_relay.py test(integration): recognize cached statements in database relay barriers (#50651) 2026-11-09 06:58:46 -08:00
deepseek_tokens.py fix(token_counter): count DeepSeek V4 text with DeepSeek's own tokenizer (#47353) 2026-10-20 03:43:35 +00:00
device_login.py test(integration): keep owned peers off ephemeral ports (#51019) 2026-11-10 21:19:57 +00:00
forward_proxy.py test(integration): keep owned peers off ephemeral ports (#51019) 2026-11-10 21:19:57 +00:00
generation.py fix(tests): resolve the integration support package without run.py's PYTHONPATH 2026-09-15 21:10:08 -07:00
hook_count_probe.py fix(mcp): bill both calls of a streamed chat completions MCP auto-execute (#48899) 2026-10-30 07:34:10 -07:00
hook_counts.py fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124) 2026-10-20 15:19:14 -07:00
idle_gateway.py test(integration): keep owned peers off ephemeral ports (#51019) 2026-11-10 21:19:57 +00:00
interception_vendor.py fix(spend): bill web search interception follow-ups once under the outer request (#49762) 2026-11-02 20:08:45 -08:00
mail.py test(integration): keep owned peers off ephemeral ports (#51019) 2026-11-10 21:19:57 +00:00
manifest.py fix(caching): never store or serve a chat completion with no choices (#44709) 2026-10-06 20:07:30 +00:00
mcp.py test(mcp): cancel disconnected HTTP tool handlers (#49436) 2026-10-31 16:41:49 -07:00
mcp_e2e_upstream_server.py test: migrate proxy security, schema migration and MCP legacy tests (#45937) 2026-10-11 16:17:24 -07:00
mcp_grants.py fix(mcp): resolve team-granted toolsets for non-admin keys and dashboard sessions (#43908) 2026-10-01 14:15:06 -07:00
mcp_stdio_peer.py test(integration): add MCP gateway coverage wave 1 with a dedicated mcp shard and proxy coverage artifact (#42711) 2026-09-23 07:48:46 -07:00
oauth_server.py test(integration): credential canary slots for MCP and pass-through credentials (#43308) 2026-09-28 17:10:51 -07:00
openai_wire.py test: fix shared-provider discovery, Codex catalog size and generated master key mismatches in CircleCI suites (#44905) 2026-10-07 00:11:25 +00:00
ordering.py fix(ci): isolate integration workers and repair test fixtures (#45863) 2026-10-11 01:56:02 -07:00
otlp_sink.py test(integration): hand every harness server a port from one per-worker block (#46761) 2026-10-15 16:56:40 -07:00
pdf_document.py fix(token_counter): price a base64 PDF document per page instead of as one image (#45301) 2026-10-08 21:20:46 -07:00
ports.py test(integration): hand every harness server a port from one per-worker block (#46761) 2026-10-15 16:56:40 -07:00
prisma_clients.py test: migrate tests/proxy_behavior into integration and unit tiers (#45939) 2026-10-11 08:28:42 -07:00
process.py feat(proxy): supervise the opt-in Rust forwarding server (#49606) 2026-11-11 19:40:00 +00:00
prometheus_series.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
prompt_cache_breakpoint.py fix(responses): map input_audio blocks in the chat-to-Responses bridge (#45224) 2026-10-07 22:25:35 -07:00
provider.py fix(ci): isolate integration workers and repair test fixtures (#45863) 2026-10-11 01:56:02 -07:00
proxy.py perf(ci): isolate and shard integration suites (#44146) 2026-10-10 11:15:19 -07:00
redaction_probe.py fix(logging): deliver completed batch success callbacks consistently (#47192) 2026-10-18 09:19:04 -07:00
redis_breaker_worker.py fix(caching): report Redis circuit-breaker metrics through the Prometheus services logger (#46484) 2026-10-14 14:29:54 -07:00
redis_process.py test(integration): hand every harness server a port from one per-worker block (#46761) 2026-10-15 16:56:40 -07:00
responses_replay.py fix(responses): replay the original Responses item order, phase, and boundaries on the next chat turn (#48912) 2026-10-29 00:56:02 -07:00
responses_stream.py fix(responses): keep the provider code and message on top-level stream error events (#46438) 2026-10-14 16:16:46 -07:00
responses_vendor.py fix(responses): drop bridge-minted reasoning items from OpenAI replays (#44132) 2026-10-02 19:39:48 -07:00
responses_ws_peer.py test(integration): keep owned peers off ephemeral ports (#51019) 2026-11-10 21:19:57 +00:00
riva.py test(integration): keep owned peers off ephemeral ports (#51019) 2026-11-10 21:19:57 +00:00
routing.py test(integration): isolate every multi-worker integration suite by default (#46027) 2026-10-12 03:46:06 -07:00
runtime.py test(integration): wait for every worker to report the websocket session cap instead of sleeping (#47412) 2026-10-19 15:35:29 -07:00
rust_forward.py feat(proxy): supervise the opt-in Rust forwarding server (#49606) 2026-11-11 19:40:00 +00:00
safeguards_rig.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
sagemaker_embedding.py feat(sagemaker)!: inference components and OpenAI-compatible embeddings (#44985) 2026-10-12 22:25:52 +00:00
scim.py fix(scim)!: delete a group through the team delete's bulk path instead of one member at a time (#47544) 2026-10-21 00:59:23 +00:00
sdk_models.py test(integration): build OpenAI and Anthropic SDK models before tests start threads (#46797) 2026-10-15 18:28:32 -07:00
sharding.py perf(ci): use native timing splits with bounded integration concurrency (#45837) 2026-10-10 13:35:11 -07:00
sigv4.py fix(bedrock): list the account's invocable models behind bedrock/* when check_provider_endpoint is on (#43981) 2026-10-09 14:14:17 -07:00
timing.py test(integration): give chaos and timing tests a harness API and gate raw signals (#46031) 2026-10-12 05:27:33 -07:00
tls.py fix(azure_storage): name Data Lake objects without base64 padding or slashes (#43914) 2026-09-30 18:17:13 -07:00
tool_rows.py feat(tool-policies): show the user who owns the key that discovered a tool (#43892) 2026-10-01 20:34:57 +00:00
tool_search.py fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns (#46444) 2026-10-14 21:40:14 -07:00
upstream.py test(integration): stop GET /__observations from draining the upstream log (#47256) 2026-10-18 09:58:05 +00:00
vertex.py fix(vertex_ai): forward system and tools to partner model count_tokens (#43900) 2026-10-03 16:36:14 -07:00
wire.py test(integration): keep owned peers off ephemeral ports (#51019) 2026-11-10 21:19:57 +00:00
worker_readiness.py test(integration): cover responses endpoint contracts (#44733) 2026-10-14 06:59:15 +00:00
workers.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00