From a305c4a54cee383920793108dc17d6936727ab2f Mon Sep 17 00:00:00 2001 From: Ishaan Jaff Date: Sat, 19 Jul 2025 12:11:19 -0700 Subject: [PATCH] docs vLLM Rerank --- docs/my-website/docs/providers/vllm.md | 106 ++++++++++++++++++++++++- docs/my-website/docs/rerank.md | 3 +- 2 files changed, 107 insertions(+), 2 deletions(-) diff --git a/docs/my-website/docs/providers/vllm.md b/docs/my-website/docs/providers/vllm.md index 5c8233b0564..d8b201956e2 100644 --- a/docs/my-website/docs/providers/vllm.md +++ b/docs/my-website/docs/providers/vllm.md @@ -10,7 +10,7 @@ LiteLLM supports all models on VLLM. | Description | vLLM is a fast and easy-to-use library for LLM inference and serving. [Docs](https://docs.vllm.ai/en/latest/index.html) | | Provider Route on LiteLLM | `hosted_vllm/` (for OpenAI compatible server), `vllm/` (for vLLM sdk usage) | | Provider Doc | [vLLM ↗](https://docs.vllm.ai/en/latest/index.html) | -| Supported Endpoints | `/chat/completions`, `/embeddings`, `/completions` | +| Supported Endpoints | `/chat/completions`, `/embeddings`, `/completions`, `/rerank` | # Quick Start @@ -157,6 +157,110 @@ curl -L -X POST 'http://0.0.0.0:4000/embeddings' \ +## Rerank + + + + +```python +from litellm import rerank +import os + +os.environ["HOSTED_VLLM_API_BASE"] = "http://localhost:8000" +os.environ["HOSTED_VLLM_API_KEY"] = "" # [optional], if your VLLM server requires an API key + +query = "What is the capital of the United States?" +documents = [ + "Carson City is the capital city of the American state of Nevada.", + "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.", + "Washington, D.C. is the capital of the United States.", + "Capital punishment has existed in the United States since before it was a country.", +] + +response = rerank( + model="hosted_vllm/your-rerank-model", + query=query, + documents=documents, + top_n=3, +) +print(response) +``` + +### Async Usage + +```python +from litellm import arerank +import os, asyncio + +os.environ["HOSTED_VLLM_API_BASE"] = "http://localhost:8000" +os.environ["HOSTED_VLLM_API_KEY"] = "" # [optional], if your VLLM server requires an API key + +async def test_async_rerank(): + query = "What is the capital of the United States?" + documents = [ + "Carson City is the capital city of the American state of Nevada.", + "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.", + "Washington, D.C. is the capital of the United States.", + "Capital punishment has existed in the United States since before it was a country.", + ] + + response = await arerank( + model="hosted_vllm/your-rerank-model", + query=query, + documents=documents, + top_n=3, + ) + print(response) + +asyncio.run(test_async_rerank()) +``` + + + + +1. Setup config.yaml + +```yaml +model_list: + - model_name: my-rerank-model + litellm_params: + model: hosted_vllm/your-rerank-model # add hosted_vllm/ prefix to route as VLLM provider + api_base: http://localhost:8000 # add api base for your VLLM server + # api_key: your-api-key # [optional] if your VLLM server requires authentication +``` + +2. Start the proxy + +```bash +$ litellm --config /path/to/config.yaml + +# RUNNING on http://0.0.0.0:4000 +``` + +3. Test it! + +```bash +curl -L -X POST 'http://0.0.0.0:4000/rerank' \ +-H 'Authorization: Bearer sk-1234' \ +-H 'Content-Type: application/json' \ +-d '{ + "model": "my-rerank-model", + "query": "What is the capital of the United States?", + "documents": [ + "Carson City is the capital city of the American state of Nevada.", + "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.", + "Washington, D.C. is the capital of the United States.", + "Capital punishment has existed in the United States since before it was a country." + ], + "top_n": 3 +}' +``` + +[See OpenAI SDK/Langchain/etc. examples](../rerank.md#litellm-proxy-usage) + + + + ## Send Video URL to VLLM Example Implementation from VLLM [here](https://github.com/vllm-project/vllm/pull/10020) diff --git a/docs/my-website/docs/rerank.md b/docs/my-website/docs/rerank.md index c7c0b37a936..11dcae777e4 100644 --- a/docs/my-website/docs/rerank.md +++ b/docs/my-website/docs/rerank.md @@ -117,4 +117,5 @@ curl http://0.0.0.0:4000/rerank \ | Jina AI| [Usage](../docs/providers/jina_ai) | | AWS Bedrock| [Usage](../docs/providers/bedrock#rerank-api) | | HuggingFace| [Usage](../docs/providers/huggingface_rerank) | -| Infinity| [Usage](../docs/providers/infinity) | \ No newline at end of file +| Infinity| [Usage](../docs/providers/infinity) | +| vLLM| [Usage](../docs/providers/vllm#rerank-endpoint) | \ No newline at end of file