mirror of https://github.com/BerriAI/litellm.git synced 2026-09-05 08:07:05 +00:00

History

Julio Quinteros Pro f3003a9840 fix(types): fix mypy errors in pass-through endpoint query param types - passthrough/utils.py: change `request_query_params` from `Dict` to `Mapping` so callers passing `dict[str, str]` satisfy the type checker (dict is invariant; Mapping is covariant in its value type) - pass_through_endpoints.py: cast `param_default_query_params` to `Optional[dict]` consistent with the existing cast pattern for other params extracted from the untyped `target_params` dict Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>		2026-02-19 12:24:14 -03:00
..
__init__.py	build(VLLM-Passthrough-with-loadbalancing-support-(enables-using-model-list-for-VLLM-/classify-endpoint)): Closes #11205	2025-05-31 09:00:04 -07:00
main.py	Add support for model id in bedrock passthrough	2026-01-08 13:03:58 +05:30
README.md	build(VLLM-Passthrough-with-loadbalancing-support-(enables-using-model-list-for-VLLM-/classify-endpoint)): Closes #11205	2025-05-31 09:00:04 -07:00
utils.py	fix(types): fix mypy errors in pass-through endpoint query param types	2026-02-19 12:24:14 -03:00

README.md

This makes it easier to pass through requests to the LLM APIs.

E.g. Route to VLLM's /classify endpoint:

SDK (Basic)

import litellm


response = litellm.llm_passthrough_route(
    model="hosted_vllm/papluca/xlm-roberta-base-language-detection",
    method="POST",
    endpoint="classify",
    api_base="http://localhost:8090",
    api_key=None,
    json={
        "model": "swapped-for-litellm-model",
        "input": "Hello, world!",
    }
)

print(response)

SDK (Router)

import asyncio
from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "roberta-base-language-detection",
            "litellm_params": {
                "model": "hosted_vllm/papluca/xlm-roberta-base-language-detection",
                "api_base": "http://localhost:8090", 
            }
        }
    ]
)

request_data = {
    "model": "roberta-base-language-detection",
    "method": "POST",
    "endpoint": "classify",
    "api_base": "http://localhost:8090",
    "api_key": None,
    "json": {
        "model": "roberta-base-language-detection",
        "input": "Hello, world!",
    }
}

async def main():
    response = await router.allm_passthrough_route(**request_data)
    print(response)

if __name__ == "__main__":
    asyncio.run(main())

PROXY

Setup config.yaml

model_list:
  - model_name: roberta-base-language-detection
    litellm_params:
      model: hosted_vllm/papluca/xlm-roberta-base-language-detection
      api_base: http://localhost:8090

Run the proxy

litellm proxy --config config.yaml

# RUNNING on http://localhost:4000

Use the proxy

curl -X POST http://localhost:4000/vllm/classify \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <your-api-key>" \
-d '{"model": "roberta-base-language-detection", "input": "Hello, world!"}' \

How to add a provider for passthrough

See VLLMModelInfo for an example.

Inherit from BaseModelInfo

from litellm.llms.base_llm.base_utils import BaseLLMModelInfo

class VLLMModelInfo(BaseLLMModelInfo):
    pass

from litellm.utils import ProviderConfigManager
from litellm.types.utils import LlmProviders

provider_config = ProviderConfigManager.get_provider_model_info(
    model="my-test-model", provider=LlmProviders.VLLM
)

print(provider_config)