mirror of https://github.com/BerriAI/litellm.git synced 2026-09-16 23:41:43 +00:00

History

Cursor Agent 3791abf4bb fix(passthrough): log when streaming spend-tracking flush fails to schedule Address Greptile feedback: the bare `except Exception: pass` in the finally blocks of _sync_streaming / _async_streaming silently dropped errors from executor.submit() / asyncio.create_task() (e.g. saturated thread pool, closed event loop). Since the entire point of the fix is that spend tracking should not silently lose data, mirror the peer streaming_handler.py logging pattern so any scheduling failure is diagnosable in production. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>		2026-04-30 02:39:29 +00:00
..
__init__.py	build(VLLM-Passthrough-with-loadbalancing-support-(enables-using-model-list-for-VLLM-/classify-endpoint)): Closes #11205	2025-05-31 09:00:04 -07:00
main.py	fix(passthrough): log when streaming spend-tracking flush fails to schedule	2026-04-30 02:39:29 +00:00
README.md	build(VLLM-Passthrough-with-loadbalancing-support-(enables-using-model-list-for-VLLM-/classify-endpoint)): Closes #11205	2025-05-31 09:00:04 -07:00
utils.py	[Fix] Apply black formatting to fix CI lint failures	2026-04-20 15:53:12 -07:00

README.md

This makes it easier to pass through requests to the LLM APIs.

E.g. Route to VLLM's /classify endpoint:

SDK (Basic)

import litellm


response = litellm.llm_passthrough_route(
    model="hosted_vllm/papluca/xlm-roberta-base-language-detection",
    method="POST",
    endpoint="classify",
    api_base="http://localhost:8090",
    api_key=None,
    json={
        "model": "swapped-for-litellm-model",
        "input": "Hello, world!",
    }
)

print(response)

SDK (Router)

import asyncio
from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "roberta-base-language-detection",
            "litellm_params": {
                "model": "hosted_vllm/papluca/xlm-roberta-base-language-detection",
                "api_base": "http://localhost:8090", 
            }
        }
    ]
)

request_data = {
    "model": "roberta-base-language-detection",
    "method": "POST",
    "endpoint": "classify",
    "api_base": "http://localhost:8090",
    "api_key": None,
    "json": {
        "model": "roberta-base-language-detection",
        "input": "Hello, world!",
    }
}

async def main():
    response = await router.allm_passthrough_route(**request_data)
    print(response)

if __name__ == "__main__":
    asyncio.run(main())

PROXY

Setup config.yaml

model_list:
  - model_name: roberta-base-language-detection
    litellm_params:
      model: hosted_vllm/papluca/xlm-roberta-base-language-detection
      api_base: http://localhost:8090

Run the proxy

litellm proxy --config config.yaml

# RUNNING on http://localhost:4000

Use the proxy

curl -X POST http://localhost:4000/vllm/classify \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <your-api-key>" \
-d '{"model": "roberta-base-language-detection", "input": "Hello, world!"}' \

How to add a provider for passthrough

See VLLMModelInfo for an example.

Inherit from BaseModelInfo

from litellm.llms.base_llm.base_utils import BaseLLMModelInfo

class VLLMModelInfo(BaseLLMModelInfo):
    pass

from litellm.utils import ProviderConfigManager
from litellm.types.utils import LlmProviders

provider_config = ProviderConfigManager.get_provider_model_info(
    model="my-test-model", provider=LlmProviders.VLLM
)

print(provider_config)