scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-10 22:41:41 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 129

docs(quick_start.md): add docs on calling openai-compatible endpoint on proxy

Browse source
This commit is contained in:
Krrish Dholakia 2023-12-06 11:06:01 -08:00
parent cc48b35a8d
commit f6546076b0
2 changed files with 25 additions and 7 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

28
docs/my-website/docs/proxy/quick_start.md
View file

@ -144,7 +144,7 @@ $ litellm --model azure/my-deployment-name
```
</TabItem>
<TabItem value="openai-proxy" label="OpenAI">
<TabItem value="openai" label="OpenAI">
```shell
$ export OPENAI_API_KEY=my-api-key
@ -154,13 +154,23 @@ $ export OPENAI_API_KEY=my-api-key
$ litellm --model gpt-3.5-turbo
```
</TabItem>
<TabItem value="openai-proxy" label="OpenAI Compatible Endpoint">
```shell
$ export OPENAI_API_KEY=my-api-key
```
```shell
$ litellm --model openai/<your model name> --api_base <your-api-base> # e.g. http://0.0.0.0:3000
```
</TabItem>
<TabItem value="huggingface" label="Huggingface (TGI) Deployed">
```shell
$ export HUGGINGFACE_API_KEY=my-api-key #[OPTIONAL]
```
```shell
$ litellm --model huggingface/<your model name> --api_base https://k58ory32yinf1ly0.us-east-1.aws.endpoints.huggingface.cloud
$ litellm --model huggingface/<your model name> --api_base <your-api-base> # e.g. http://0.0.0.0:3000
```
</TabItem>
@ -270,6 +280,8 @@ $ litellm --model command-nightly
</Tabs>
## Quick Start - LiteLLM Proxy + Config.yaml
The config allows you to create a model list and set `api_base`, `max_tokens` (all litellm params). See more details about the config [here](https://docs.litellm.ai/docs/proxy/configs)
@ -277,9 +289,9 @@ The config allows you to create a model list and set `api_base`, `max_tokens` (a
Example config
```yaml
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model_list:
- model_name: gpt-3.5-turbo # user-facing model alias
litellm_params: # all params accepted by litellm.completion() - https://docs.litellm.ai/docs/completion/input
model: azure/<your-deployment-name>
api_base: <your-azure-api-endpoint>
api_key: <your-azure-api-key>
@ -288,6 +300,10 @@ model_list:
model: azure/gpt-turbo-small-ca
api_base: https://my-endpoint-canada-berri992.openai.azure.com/
api_key: <your-azure-api-key>
- model_name: vllm-model
litellm_params:
model: openai/<your-model-name>
api_base: <your-api-base> # e.g. http://0.0.0.0:3000
```
### Run proxy with config
@ -296,6 +312,8 @@ model_list:
litellm --config your_config.yaml
```
[**More Info**](./configs.md)
## Quick Start Docker Image: Github Container Registry
### Pull the litellm ghcr docker image

4
litellm/tests/test_provider_specific_config.py
View file

@ -395,7 +395,7 @@ def sagemaker_test_completion():
try:
# OVERRIDE WITH DYNAMIC MAX TOKENS
response_1 = litellm.completion(
model="sagemaker/jumpstart-dft-meta-textgeneration-llama-2-7b",
model="sagemaker/berri-benchmarking-Llama-2-70b-chat-hf-4",
messages=[{ "content": "Hello, how are you? Be as verbose as possible","role": "user"}],
max_tokens=100
)
@ -404,7 +404,7 @@ def sagemaker_test_completion():
# USE CONFIG TOKENS
response_2 = litellm.completion(
model="sagemaker/jumpstart-dft-meta-textgeneration-llama-2-7b",
model="sagemaker/berri-benchmarking-Llama-2-70b-chat-hf-4",
messages=[{ "content": "Hello, how are you? Be as verbose as possible","role": "user"}],
)
response_2_text = response_2.choices[0].message.content