docs(custom_llm_server.md): document anthropic custom llm translation

This commit is contained in:
Krrish Dholakia 2025-06-24 14:10:06 -07:00
parent ded1734d34
commit 47be18a17c

View file

@ -13,11 +13,12 @@ Call your custom torch-serve / internal LLM APIs via LiteLLM
:::
Supported Routes:
- `/v1/chat/completions` -> `litellm.completion`
- `/v1/completions` -> `litellm.text_completion`
- `/v1/embeddings` -> `litellm.embedding`
- `/v1/images/generations` -> `litellm.image_generation`
- `/v1/chat/completions` -> `litellm.acompletion`
- `/v1/completions` -> `litellm.atext_completion`
- `/v1/embeddings` -> `litellm.aembedding`
- `/v1/images/generations` -> `litellm.aimage_generation`
- `/v1/messages` -> `litellm.acompletion`
## Quick Start
@ -262,6 +263,102 @@ Expected Response
}
```
## Anthropic `/v1/messages`
- Write the integration for .acompletion
- litellm will transform it to /v1/messages
1. Setup your `custom_handler.py` file
```python
import litellm
from litellm import CustomLLM, completion, get_llm_provider
class MyCustomLLM(CustomLLM):
async def acompletion(self, *args, **kwargs) -> litellm.ModelResponse:
return litellm.completion(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Hello world"}],
mock_response="Hi!",
) # type: ignore
my_custom_llm = MyCustomLLM()
```
2. Add to `config.yaml`
In the config below, we pass
python_filename: `custom_handler.py`
custom_handler_instance_name: `my_custom_llm`. This is defined in Step 1
custom_handler: `custom_handler.my_custom_llm`
```yaml
model_list:
- model_name: "test-model"
litellm_params:
model: "openai/text-embedding-ada-002"
- model_name: "my-custom-model"
litellm_params:
model: "my-custom-llm/my-model"
litellm_settings:
custom_provider_map:
- {"provider": "my-custom-llm", "custom_handler": custom_handler.my_custom_llm}
```
```bash
litellm --config /path/to/config.yaml
```
3. Test it!
```bash
curl -L -X POST 'http://0.0.0.0:4000/v1/messages' \
-H 'anthropic-version: 2023-06-01' \
-H 'content-type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{
"model": "my-custom-model",
"max_tokens": 1024,
"messages": [{
"role": "user",
"content": [
{
"type": "text",
"text": "What are the key findings in this document 12?"
}]
}]
}'
```
Expected Response
```json
{
"id": "chatcmpl-Bm4qEp4h4vCe7Zi4Gud1MAxTWgibO",
"type": "message",
"role": "assistant",
"model": "gpt-3.5-turbo-0125",
"stop_sequence": null,
"usage": {
"input_tokens": 18,
"output_tokens": 44
},
"content": [
{
"type": "text",
"text": "Without the specific document being provided, it is not possible to determine the key findings within it. If you can provide the content or a summary of document 12, I would be happy to help identify the key findings."
}
],
"stop_reason": "end_turn"
}
```
## Additional Parameters
Additional parameters are passed inside `optional_params` key in the `completion` or `image_generation` function.