mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-14 23:21:35 +00:00
101f246fc5
* fix(logging): classify allm_passthrough_route as async to prevent duplicate success callbacks Async passthrough requests set kwargs["allm_passthrough_route"]=True but that flag is never propagated into litellm_params, and _is_sync_litellm_request only checks acompletion/aresponses/aembedding/aimage_generation/atranscription. Every async passthrough is misclassified as sync, which trips the CustomLogger sync branch in success_handler and fires log_success_event in addition to the async worker's async_log_success_event, causing 2-3 duplicate LangSmith runs per Bedrock passthrough request Propagate allm_passthrough_route through get_litellm_params and teach the classifier about it. /chat/completions and other non-passthrough paths are untouched * test(passthrough): assert allm_passthrough_route flag propagates end-to-end Integration-level guard on top of the unit tests in test_litellm_logging.py: verifies that when kwargs["allm_passthrough_route"]=True enters llm_passthrough_route, the flag survives get_litellm_params(**kwargs), lands in the logging object's litellm_params, and _is_sync_litellm_request reads the request as async --------- Co-authored-by: yucheng <yucheng@yuchengs-MBP.localdomain> |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| main.py | ||
| README.md | ||
| timeout_utils.py | ||
| utils.py | ||
This makes it easier to pass through requests to the LLM APIs.
E.g. Route to VLLM's /classify endpoint:
SDK (Basic)
import litellm
response = litellm.llm_passthrough_route(
model="hosted_vllm/papluca/xlm-roberta-base-language-detection",
method="POST",
endpoint="classify",
api_base="http://localhost:8090",
api_key=None,
json={
"model": "swapped-for-litellm-model",
"input": "Hello, world!",
}
)
print(response)
SDK (Router)
import asyncio
from litellm import Router
router = Router(
model_list=[
{
"model_name": "roberta-base-language-detection",
"litellm_params": {
"model": "hosted_vllm/papluca/xlm-roberta-base-language-detection",
"api_base": "http://localhost:8090",
}
}
]
)
request_data = {
"model": "roberta-base-language-detection",
"method": "POST",
"endpoint": "classify",
"api_base": "http://localhost:8090",
"api_key": None,
"json": {
"model": "roberta-base-language-detection",
"input": "Hello, world!",
}
}
async def main():
response = await router.allm_passthrough_route(**request_data)
print(response)
if __name__ == "__main__":
asyncio.run(main())
PROXY
- Setup config.yaml
model_list:
- model_name: roberta-base-language-detection
litellm_params:
model: hosted_vllm/papluca/xlm-roberta-base-language-detection
api_base: http://localhost:8090
- Run the proxy
litellm proxy --config config.yaml
# RUNNING on http://localhost:4000
- Use the proxy
curl -X POST http://localhost:4000/vllm/classify \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <your-api-key>" \
-d '{"model": "roberta-base-language-detection", "input": "Hello, world!"}' \
How to add a provider for passthrough
See VLLMModelInfo for an example.
- Inherit from BaseModelInfo
from litellm.llms.base_llm.base_utils import BaseLLMModelInfo
class VLLMModelInfo(BaseLLMModelInfo):
pass
- Register the provider in the ProviderConfigManager.get_provider_model_info
from litellm.utils import ProviderConfigManager
from litellm.types.utils import LlmProviders
provider_config = ProviderConfigManager.get_provider_model_info(
model="my-test-model", provider=LlmProviders.VLLM
)
print(provider_config)