mirror of https://github.com/BerriAI/litellm.git synced 2026-09-08 22:21:35 +00:00

History

Ishaan Jaff a42132f329 fix(passthrough): propagate Azure 429/5xx errors in async streaming instead of silent HTTP 200 (#22913 ) * fix(passthrough): raise_for_status in _async_streaming to propagate Azure 429s * address greptile review feedback (greploop iteration 1) Guard data/json args when content is provided to avoid httpx ValueError * address greptile review feedback (greploop iteration 2) Use bare raise to preserve original traceback in _async_streaming exception handler * address greptile review feedback (greploop iteration 3) Close httpx streaming response on error to prevent connection pool exhaustion * address greptile review feedback (greploop iteration 4) Guard aclose() call to prevent masking original exception; add explicit test for content param forwarding * address greptile review feedback (greploop iteration 5) Pass content to sign_request so AWS body-hash signing is correct when content is the sole body source * revert sign_request content change - request_data expects dict, not bytes Bedrock's sign_request calls json.dumps(request_data) — passing content bytes would TypeError. sign_request should only receive data/json (dict), not raw bytes.		2026-03-05 10:12:43 -08:00
..
__init__.py	build(VLLM-Passthrough-with-loadbalancing-support-(enables-using-model-list-for-VLLM-/classify-endpoint)): Closes #11205	2025-05-31 09:00:04 -07:00
main.py	fix(passthrough): propagate Azure 429/5xx errors in async streaming instead of silent HTTP 200 (#22913 )	2026-03-05 10:12:43 -08:00
README.md	build(VLLM-Passthrough-with-loadbalancing-support-(enables-using-model-list-for-VLLM-/classify-endpoint)): Closes #11205	2025-05-31 09:00:04 -07:00
utils.py	fix(types): fix mypy errors in pass-through endpoint query param types	2026-02-19 12:24:14 -03:00

README.md

This makes it easier to pass through requests to the LLM APIs.

E.g. Route to VLLM's /classify endpoint:

SDK (Basic)

import litellm


response = litellm.llm_passthrough_route(
    model="hosted_vllm/papluca/xlm-roberta-base-language-detection",
    method="POST",
    endpoint="classify",
    api_base="http://localhost:8090",
    api_key=None,
    json={
        "model": "swapped-for-litellm-model",
        "input": "Hello, world!",
    }
)

print(response)

SDK (Router)

import asyncio
from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "roberta-base-language-detection",
            "litellm_params": {
                "model": "hosted_vllm/papluca/xlm-roberta-base-language-detection",
                "api_base": "http://localhost:8090", 
            }
        }
    ]
)

request_data = {
    "model": "roberta-base-language-detection",
    "method": "POST",
    "endpoint": "classify",
    "api_base": "http://localhost:8090",
    "api_key": None,
    "json": {
        "model": "roberta-base-language-detection",
        "input": "Hello, world!",
    }
}

async def main():
    response = await router.allm_passthrough_route(**request_data)
    print(response)

if __name__ == "__main__":
    asyncio.run(main())

PROXY

Setup config.yaml

model_list:
  - model_name: roberta-base-language-detection
    litellm_params:
      model: hosted_vllm/papluca/xlm-roberta-base-language-detection
      api_base: http://localhost:8090

Run the proxy

litellm proxy --config config.yaml

# RUNNING on http://localhost:4000

Use the proxy

curl -X POST http://localhost:4000/vllm/classify \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <your-api-key>" \
-d '{"model": "roberta-base-language-detection", "input": "Hello, world!"}' \

How to add a provider for passthrough

See VLLMModelInfo for an example.

Inherit from BaseModelInfo

from litellm.llms.base_llm.base_utils import BaseLLMModelInfo

class VLLMModelInfo(BaseLLMModelInfo):
    pass

from litellm.utils import ProviderConfigManager
from litellm.types.utils import LlmProviders

provider_config = ProviderConfigManager.get_provider_model_info(
    model="my-test-model", provider=LlmProviders.VLLM
)

print(provider_config)