mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-09 03:18:44 +00:00
docs(custom_llm_server.md): document anthropic custom llm translation
This commit is contained in:
parent
ded1734d34
commit
47be18a17c
1 changed files with 101 additions and 4 deletions
|
|
@ -13,11 +13,12 @@ Call your custom torch-serve / internal LLM APIs via LiteLLM
|
|||
:::
|
||||
|
||||
Supported Routes:
|
||||
- `/v1/chat/completions` -> `litellm.completion`
|
||||
- `/v1/completions` -> `litellm.text_completion`
|
||||
- `/v1/embeddings` -> `litellm.embedding`
|
||||
- `/v1/images/generations` -> `litellm.image_generation`
|
||||
- `/v1/chat/completions` -> `litellm.acompletion`
|
||||
- `/v1/completions` -> `litellm.atext_completion`
|
||||
- `/v1/embeddings` -> `litellm.aembedding`
|
||||
- `/v1/images/generations` -> `litellm.aimage_generation`
|
||||
|
||||
- `/v1/messages` -> `litellm.acompletion`
|
||||
|
||||
## Quick Start
|
||||
|
||||
|
|
@ -262,6 +263,102 @@ Expected Response
|
|||
}
|
||||
```
|
||||
|
||||
## Anthropic `/v1/messages`
|
||||
|
||||
- Write the integration for .acompletion
|
||||
- litellm will transform it to /v1/messages
|
||||
|
||||
1. Setup your `custom_handler.py` file
|
||||
|
||||
```python
|
||||
import litellm
|
||||
from litellm import CustomLLM, completion, get_llm_provider
|
||||
|
||||
|
||||
class MyCustomLLM(CustomLLM):
|
||||
async def acompletion(self, *args, **kwargs) -> litellm.ModelResponse:
|
||||
return litellm.completion(
|
||||
model="gpt-3.5-turbo",
|
||||
messages=[{"role": "user", "content": "Hello world"}],
|
||||
mock_response="Hi!",
|
||||
) # type: ignore
|
||||
|
||||
|
||||
my_custom_llm = MyCustomLLM()
|
||||
```
|
||||
|
||||
2. Add to `config.yaml`
|
||||
|
||||
In the config below, we pass
|
||||
|
||||
python_filename: `custom_handler.py`
|
||||
custom_handler_instance_name: `my_custom_llm`. This is defined in Step 1
|
||||
|
||||
custom_handler: `custom_handler.my_custom_llm`
|
||||
|
||||
```yaml
|
||||
model_list:
|
||||
- model_name: "test-model"
|
||||
litellm_params:
|
||||
model: "openai/text-embedding-ada-002"
|
||||
- model_name: "my-custom-model"
|
||||
litellm_params:
|
||||
model: "my-custom-llm/my-model"
|
||||
|
||||
litellm_settings:
|
||||
custom_provider_map:
|
||||
- {"provider": "my-custom-llm", "custom_handler": custom_handler.my_custom_llm}
|
||||
```
|
||||
|
||||
```bash
|
||||
litellm --config /path/to/config.yaml
|
||||
```
|
||||
|
||||
3. Test it!
|
||||
|
||||
```bash
|
||||
curl -L -X POST 'http://0.0.0.0:4000/v1/messages' \
|
||||
-H 'anthropic-version: 2023-06-01' \
|
||||
-H 'content-type: application/json' \
|
||||
-H 'Authorization: Bearer sk-1234' \
|
||||
-d '{
|
||||
"model": "my-custom-model",
|
||||
"max_tokens": 1024,
|
||||
"messages": [{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "What are the key findings in this document 12?"
|
||||
}]
|
||||
}]
|
||||
}'
|
||||
```
|
||||
|
||||
Expected Response
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "chatcmpl-Bm4qEp4h4vCe7Zi4Gud1MAxTWgibO",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"model": "gpt-3.5-turbo-0125",
|
||||
"stop_sequence": null,
|
||||
"usage": {
|
||||
"input_tokens": 18,
|
||||
"output_tokens": 44
|
||||
},
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Without the specific document being provided, it is not possible to determine the key findings within it. If you can provide the content or a summary of document 12, I would be happy to help identify the key findings."
|
||||
}
|
||||
],
|
||||
"stop_reason": "end_turn"
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## Additional Parameters
|
||||
|
||||
Additional parameters are passed inside `optional_params` key in the `completion` or `image_generation` function.
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue