mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-07 08:26:10 +00:00
(docs) proxy server
This commit is contained in:
ishaan-jaff
parent
7c37d24b46
commit
adf7539be2
1 changed files with 34 additions and 135 deletions
|
|
@ -183,36 +183,47 @@ $ litellm --model command-nightly
|
|||
- POST `/embeddings` - embedding endpoint for Azure, OpenAI, Huggingface endpoints
|
||||
- GET `/models` - available models on server
|
||||
|
||||
## Advanced
|
||||
## Usage
|
||||
|
||||
### Tutorial: Use with Multiple LLMs + Aider/AutoGen/Langroid/etc.
|
||||
<Tabs>
|
||||
<TabItem value="multiple-LLMs" label="Multiple LLMs">
|
||||
<TabItem value="lm-harness" label="LM-Harness Evals">
|
||||
This tutorial assumes you're using the `big-refactor` branch of LM Harness https://github.com/EleutherAI/lm-evaluation-harness/tree/big-refactor
|
||||
|
||||
```shell
|
||||
$ litellm
|
||||
|
||||
#INFO: litellm proxy running on http://0.0.0.0:8000
|
||||
**Step 1: Start the local proxy**
|
||||
```shell
|
||||
$ litellm --model huggingface/bigcode/starcoder
|
||||
```
|
||||
|
||||
#### Send a request to your proxy
|
||||
```python
|
||||
import openai
|
||||
Using a custom api base
|
||||
|
||||
openai.api_key = "any-string-here"
|
||||
openai.api_base = "http://0.0.0.0:8080" # your proxy url
|
||||
|
||||
# call gpt-3.5-turbo
|
||||
response = openai.ChatCompletion.create(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey"}])
|
||||
|
||||
print(response)
|
||||
|
||||
# call ollama/llama2
|
||||
response = openai.ChatCompletion.create(model="ollama/llama2", messages=[{"role": "user", "content": "Hey"}])
|
||||
|
||||
print(response)
|
||||
```shell
|
||||
$ export HUGGINGFACE_API_KEY=my-api-key #[OPTIONAL]
|
||||
$ litellm --model huggingface/tinyllama --api_base https://k58ory32yinf1ly0.us-east-1.aws.endpoints.huggingface.cloud
|
||||
```
|
||||
|
||||
OpenAI Compatible Endpoint at http://0.0.0.0:8000
|
||||
|
||||
**Step 2: Set OpenAI API Base & Key**
|
||||
```shell
|
||||
$ export OPENAI_API_BASE=http://0.0.0.0:8000
|
||||
```
|
||||
|
||||
LM Harness requires you to set an OpenAI API key `OPENAI_API_SECRET_KEY` for running benchmarks
|
||||
```shell
|
||||
export OPENAI_API_SECRET_KEY=anything
|
||||
```
|
||||
|
||||
**Step 3: Run LM-Eval-Harness**
|
||||
|
||||
```shell
|
||||
python3 -m lm_eval \
|
||||
--model openai-completions \
|
||||
--model_args engine=davinci \
|
||||
--task crows_pairs_english_age
|
||||
|
||||
```
|
||||
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="continue-dev" label="ContinueDev">
|
||||
|
||||
|
|
@ -269,98 +280,7 @@ user_proxy.initiate_chat(assistant, message="Plot a chart of META and TESLA stoc
|
|||
|
||||
Credits [@victordibia](https://github.com/microsoft/autogen/issues/45#issuecomment-1749921972) for this tutorial.
|
||||
</TabItem>
|
||||
<TabItem value="multi-LLM AutoGen" label="AutoGen Multi-LLM">
|
||||
|
||||
|
||||
```python
|
||||
from autogen import AssistantAgent, GroupChatManager, UserProxyAgent
|
||||
from autogen.agentchat import GroupChat
|
||||
config_list = [
|
||||
{
|
||||
"model": "ollama/mistralorca",
|
||||
"api_base": "http://localhost:8000", # litellm compatible endpoint
|
||||
"api_type": "open_ai",
|
||||
"api_key": "NULL", # just a placeholder
|
||||
}
|
||||
]
|
||||
llm_config = {"config_list": config_list, "seed": 42}
|
||||
|
||||
code_config_list = [
|
||||
{
|
||||
"model": "ollama/phind-code",
|
||||
"api_base": "http://localhost:8000", # litellm compatible endpoint
|
||||
"api_type": "open_ai",
|
||||
"api_key": "NULL", # just a placeholder
|
||||
}
|
||||
]
|
||||
|
||||
code_config = {"config_list": code_config_list, "seed": 42}
|
||||
|
||||
admin = UserProxyAgent(
|
||||
name="Admin",
|
||||
system_message="A human admin. Interact with the planner to discuss the plan. Plan execution needs to be approved by this admin.",
|
||||
llm_config=llm_config,
|
||||
code_execution_config=False,
|
||||
)
|
||||
|
||||
|
||||
engineer = AssistantAgent(
|
||||
name="Engineer",
|
||||
llm_config=code_config,
|
||||
system_message="""Engineer. You follow an approved plan. You write python/shell code to solve tasks. Wrap the code in a code block that specifies the script type. The user can't modify your code. So do not suggest incomplete code which requires others to modify. Don't use a code block if it's not intended to be executed by the executor.
|
||||
Don't include multiple code blocks in one response. Do not ask others to copy and paste the result. Check the execution result returned by the executor.
|
||||
If the result indicates there is an error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.
|
||||
""",
|
||||
)
|
||||
planner = AssistantAgent(
|
||||
name="Planner",
|
||||
system_message="""Planner. Suggest a plan. Revise the plan based on feedback from admin and critic, until admin approval.
|
||||
The plan may involve an engineer who can write code and a scientist who doesn't write code.
|
||||
Explain the plan first. Be clear which step is performed by an engineer, and which step is performed by a scientist.
|
||||
""",
|
||||
llm_config=llm_config,
|
||||
)
|
||||
executor = UserProxyAgent(
|
||||
name="Executor",
|
||||
system_message="Executor. Execute the code written by the engineer and report the result.",
|
||||
human_input_mode="NEVER",
|
||||
llm_config=llm_config,
|
||||
code_execution_config={"last_n_messages": 3, "work_dir": "paper"},
|
||||
)
|
||||
critic = AssistantAgent(
|
||||
name="Critic",
|
||||
system_message="Critic. Double check plan, claims, code from other agents and provide feedback. Check whether the plan includes adding verifiable info such as source URL.",
|
||||
llm_config=llm_config,
|
||||
)
|
||||
groupchat = GroupChat(
|
||||
agents=[admin, engineer, planner, executor, critic],
|
||||
messages=[],
|
||||
max_round=50,
|
||||
)
|
||||
manager = GroupChatManager(groupchat=groupchat, llm_config=llm_config)
|
||||
|
||||
|
||||
admin.initiate_chat(
|
||||
manager,
|
||||
message="""
|
||||
""",
|
||||
)
|
||||
```
|
||||
|
||||
Credits [@Nathan](https://gist.github.com/CUexter) for this tutorial.
|
||||
</TabItem>
|
||||
|
||||
|
||||
<TabItem value="gpt-pilot" label="GPT-Pilot">
|
||||
GPT-Pilot helps you build apps with AI Agents. [For more](https://github.com/Pythagora-io/gpt-pilot)
|
||||
|
||||
In your .env set the openai endpoint to your local server.
|
||||
|
||||
```
|
||||
OPENAI_ENDPOINT=http://0.0.0.0:8000
|
||||
OPENAI_API_KEY=my-fake-key
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="guidance" label="guidance">
|
||||
A guidance language for controlling large language models.
|
||||
https://github.com/guidance-ai/guidance
|
||||
|
|
@ -406,30 +326,9 @@ print(result)
|
|||
|
||||
### [TUTORIAL] LM-Evaluation Harness with TGI
|
||||
|
||||
Evaluate LLMs 20x faster with TGI via litellm proxy's `/completions` endpoint.
|
||||
|
||||
This tutorial assumes you're using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)
|
||||
## Advanced
|
||||
|
||||
**Step 1: Start the local proxy**
|
||||
```shell
|
||||
$ litellm --model huggingface/bigcode/starcoder
|
||||
```
|
||||
|
||||
OpenAI Compatible Endpoint at http://0.0.0.0:8000
|
||||
|
||||
**Step 2: Set OpenAI API Base**
|
||||
```shell
|
||||
$ export OPENAI_API_BASE="http://0.0.0.0:8000"
|
||||
```
|
||||
|
||||
**Step 3: Run LM-Eval-Harness**
|
||||
|
||||
```shell
|
||||
$ python3 main.py \
|
||||
--model gpt3 \
|
||||
--model_args engine=huggingface/bigcode/starcoder \
|
||||
--tasks hellaswag
|
||||
```
|
||||
|
||||
|
||||
### Caching
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue