diff --git a/.dockerignore b/.dockerignore
index 8459d0cd482..f0fdcf9d479 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -1,5 +1,5 @@
docs
-cookbook
+litellm/proxy/discovery_endpoints/skill_scanner_adapter
.circleci
.github
tests
diff --git a/.github/workflows/test-unit.yml b/.github/workflows/test-unit.yml
index e82d81207e2..90ac7a431e4 100644
--- a/.github/workflows/test-unit.yml
+++ b/.github/workflows/test-unit.yml
@@ -459,7 +459,6 @@ jobs:
tests/unit/ocr
tests/unit/passthrough
tests/unit/rag
- tests/unit/cookbook
tests/unit/realtime_api
tests/unit/rerank_api
tests/unit/search
diff --git a/cookbook/Benchmarking_LLMs_by_use_case.ipynb b/cookbook/Benchmarking_LLMs_by_use_case.ipynb
deleted file mode 100644
index 6ea6211bfb6..00000000000
--- a/cookbook/Benchmarking_LLMs_by_use_case.ipynb
+++ /dev/null
@@ -1,753 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "4Cq-_Y-TKf0r"
- },
- "source": [
- "# LiteLLM - Benchmark Llama2, Claude1.2 and GPT3.5 for a use case\n",
- "In this notebook for a given use case we run the same question and view:\n",
- "* LLM Response\n",
- "* Response Time\n",
- "* Response Cost\n",
- "\n",
- "## Sample output for a question\n",
- ""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "O3ENsWYB27Mb"
- },
- "outputs": [],
- "source": [
- "!pip install litellm"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "Pk55Mjq_3DiR"
- },
- "source": [
- "## Example Use Case 1 - Code Generator\n",
- "### For this use case enter your system prompt and questions\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 21,
- "metadata": {
- "id": "_1SZYJFB3HmQ"
- },
- "outputs": [],
- "source": [
- "# enter your system prompt if you have one\n",
- "system_prompt = \"\"\"\n",
- "You are a coding assistant helping users using litellm.\n",
- "litellm is a light package to simplify calling OpenAI, Azure, Cohere, Anthropic, Huggingface API Endpoints\n",
- "--\n",
- "Sample Usage:\n",
- "```\n",
- "pip install litellm\n",
- "from litellm import completion\n",
- "## set ENV variables\n",
- "os.environ[\"OPENAI_API_KEY\"] = \"openai key\"\n",
- "os.environ[\"COHERE_API_KEY\"] = \"cohere key\"\n",
- "messages = [{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}]\n",
- "# openai call\n",
- "response = completion(model=\"gpt-3.5-turbo\", messages=messages)\n",
- "# cohere call\n",
- "response = completion(\"command-nightly\", messages)\n",
- "```\n",
- "\n",
- "\"\"\"\n",
- "\n",
- "\n",
- "# qustions/logs you want to run the LLM on\n",
- "questions = [\n",
- " \"what is litellm?\",\n",
- " \"why should I use LiteLLM\",\n",
- " \"does litellm support Anthropic LLMs\",\n",
- " \"write code to make a litellm completion call\",\n",
- "]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "AHH3cqeU3_ZT"
- },
- "source": [
- "## Running questions\n",
- "### Select from 100+ LLMs here: https://docs.litellm.ai/docs/providers"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "BpQD4A5339L3"
- },
- "outputs": [],
- "source": [
- "from litellm import completion, completion_cost\n",
- "import os\n",
- "import time\n",
- "\n",
- "# optional use litellm dashboard to view logs\n",
- "# litellm.use_client = True\n",
- "# litellm.token = \"ishaan_2@berri.ai\" # set your email\n",
- "\n",
- "\n",
- "# set API keys\n",
- "os.environ['TOGETHERAI_API_KEY'] = \"\"\n",
- "os.environ['OPENAI_API_KEY'] = \"\"\n",
- "os.environ['ANTHROPIC_API_KEY'] = \"\"\n",
- "\n",
- "\n",
- "# select LLMs to benchmark\n",
- "# using https://api.together.xyz/playground for llama2\n",
- "# try any supported LLM here: https://docs.litellm.ai/docs/providers\n",
- "\n",
- "models = ['togethercomputer/llama-2-70b-chat', 'gpt-3.5-turbo', 'claude-instant-1.2']\n",
- "data = []\n",
- "\n",
- "for question in questions: # group by question\n",
- " for model in models:\n",
- " print(f\"running question: {question} for model: {model}\")\n",
- " start_time = time.time()\n",
- " # show response, response time, cost for each question\n",
- " response = completion(\n",
- " model=model,\n",
- " max_tokens=500,\n",
- " messages = [\n",
- " {\n",
- " \"role\": \"system\", \"content\": system_prompt\n",
- " },\n",
- " {\n",
- " \"role\": \"user\", \"content\": question\n",
- " }\n",
- " ],\n",
- " )\n",
- " end = time.time()\n",
- " total_time = end-start_time # response time\n",
- " # print(response)\n",
- " cost = completion_cost(response) # cost for completion\n",
- " raw_response = response['choices'][0]['message']['content'] # response string\n",
- "\n",
- "\n",
- " # add log to pandas df\n",
- " data.append(\n",
- " {\n",
- " 'Model': model,\n",
- " 'Question': question,\n",
- " 'Response': raw_response,\n",
- " 'ResponseTime': total_time,\n",
- " 'Cost': cost\n",
- " })"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "apOSV3PBLa5Y"
- },
- "source": [
- "## View Benchmarks for LLMs"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 22,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 1000
- },
- "id": "CJqBlqUh_8Ws",
- "outputId": "e02c3427-d8c6-4614-ff07-6aab64247ff6"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Question: does litellm support Anthropic LLMs\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "
\n",
- " \n",
- "
\n",
- "
\n",
- "
Model
\n",
- "
Question
\n",
- "
Response
\n",
- "
ResponseTime
\n",
- "
Cost
\n",
- "
\n",
- " \n",
- " \n",
- "
\n",
- "
6
\n",
- "
togethercomputer/llama-2-70b-chat
\n",
- "
does litellm support Anthropic LLMs
\n",
- "
Yes, litellm supports Anthropic LLMs.\\n\\nIn the example usage you provided, the `completion` function is called with the `model` parameter set to `\"gpt-3.5-turbo\"` for OpenAI and `\"command-nightly\"` for Cohere.\\n\\nTo use an Anthropic LLM with litellm, you would set the `model` parameter to the name of the Anthropic model you want to use, followed by the version number, if applicable. For example:\\n```\\nresponse = completion(model=\"anthropic-gpt-2\", messages=messages)\\n```\\nThis would call the Anthropic GPT-2 model to generate a completion for the given input messages.\\n\\nNote that you will need to set the `ANTHROPIC_API_KEY` environment variable to your Anthropic API key before making the call. You can do this by running the following command in your terminal:\\n```\\nos.environ[\"ANTHROPIC_API_KEY\"] = \"your-anthropic-api-key\"\\n```\\nReplace `\"your-anthropic-api-key\"` with your actual Anthropic API key.\\n\\nOnce you've set the environment variable, you can use the `completion` function with the `model` parameter set to an Anthropic model name to call the Anthropic API and generate a completion.
\n",
- "
21.513009
\n",
- "
0.001347
\n",
- "
\n",
- "
\n",
- "
7
\n",
- "
gpt-3.5-turbo
\n",
- "
does litellm support Anthropic LLMs
\n",
- "
No, currently litellm does not support Anthropic LLMs. It mainly focuses on simplifying the usage of OpenAI, Azure, Cohere, and Huggingface API endpoints.
\n",
- "
8.656510
\n",
- "
0.000342
\n",
- "
\n",
- "
\n",
- "
8
\n",
- "
claude-instant-1.2
\n",
- "
does litellm support Anthropic LLMs
\n",
- "
Yes, litellm supports calling Anthropic LLMs through the completion function.\\n\\nTo use an Anthropic model with litellm:\\n\\n1. Set the ANTHROPIC_API_KEY environment variable with your Anthropic API key\\n\\n2. Pass the model name as the 'model' argument to completion(). Anthropic model names follow the format 'anthropic/<model_name>'\\n\\nFor example:\\n\\n```python \\nimport os\\nfrom litellm import completion\\n\\nos.environ[\"ANTHROPIC_API_KEY\"] = \"your_anthropic_api_key\"\\n\\nmessages = [{\"content\": \"Hello\", \"role\": \"user\"}]\\n\\nresponse = completion(model=\"anthropic/constitutional\", messages=messages)\\n```\\n\\nThis would call the Constitutional AI model from Anthropic.\\n\\nSo in summary, litellm provides a simple interface to call any Anthropic models as long as you specify the model name correctly and set the ANTHROPIC_API_KEY env variable.
Litellm is a lightweight Python package that simplifies calling various AI API endpoints, including OpenAI, Azure, Cohere, Anthropic, and Hugging Face. It provides a convenient interface for making requests to these APIs, allowing developers to easily integrate them into their applications. With Litellm, developers can quickly and easily interact with multiple AI models and services, without having to handle the details of authentication, API calls, and response parsing. This makes it easier to build and deploy AI-powered applications, and can help developers save time and effort.
\n",
- "
13.479644
\n",
- "
0.000870
\n",
- "
\n",
- "
\n",
- "
1
\n",
- "
gpt-3.5-turbo
\n",
- "
what is litellm?
\n",
- "
litellm is a light package that provides a simplified interface for making API calls to various language models and APIs. It abstracts away the complexities of handling network requests, authentication, and response parsing, making it easier for developers to integrate powerful language models into their applications.\\n\\nWith litellm, you can quickly make API calls to models like OpenAI's GPT-3.5 Turbo, Azure's Text Analytics, Cohere's Command API, Anthropic's API, and Huggingface's models. It also supports additional functionality like conversational AI, summarization, translation, and more.\\n\\nBy using litellm, you can focus on your application logic without getting tangled in the details of API integration, allowing you to quickly build intelligent and conversational applications.
\n",
- "
8.324332
\n",
- "
0.000566
\n",
- "
\n",
- "
\n",
- "
2
\n",
- "
claude-instant-1.2
\n",
- "
what is litellm?
\n",
- "
litellm is a Python library that simplifies calling various AI API endpoints like OpenAI, Azure, Cohere, Anthropic, and Huggingface. \\n\\nSome key things to know about litellm:\\n\\n- It provides a consistent interface for completing prompts and generating responses from different AI models through a single method called completion().\\n\\n- You specify the API (e.g. OpenAI, Cohere etc.) and model either by name or by setting environment variables before making the completion call.\\n\\n- This avoids having to use different SDKs or APIs for each provider and standardizes the call structure. \\n\\n- It handles things like setting headers, encoding inputs, parsing responses so the user doesn't have to deal with those details.\\n\\n- The goal is to make it easy to try different AI APIs and models without having to change code or learn different interfaces.\\n\\n- It's lightweight with no other dependencies required besides what's needed for each API (e.g. openai, azure SDKs etc.).\\n\\nSo in summary, litellm is a small library that provides a common way to interact with multiple conversational AI APIs through a single Python method, avoiding the need to directly use each provider's specific SDK.
\\nThere are several reasons why you might want to use LiteLLM:\\n\\n1. Simplified API calls: LiteLLM provides a simple and consistent API for calling various language models, making it easier to use multiple models and switch between them.\\n2. Environment variable configuration: LiteLLM allows you to set environment variables for API keys and model names, making it easier to manage and switch between different models and APIs.\\n3. Support for multiple models and APIs: LiteLLM supports a wide range of language models and APIs, including OpenAI, Azure, Cohere, Anthropic, and Hugging Face.\\n4. Easy integration with popular frameworks: LiteLLM can be easily integrated with popular frameworks such as PyTorch and TensorFlow, making it easy to use with your existing codebase.\\n5. Lightweight: LiteLLM is a lightweight package, making it easy to install and use, even on resource-constrained devices.\\n6. Flexible: LiteLLM allows you to define your own models and APIs, making it easy to use with custom models and APIs.\\n7. Extensive documentation: LiteLLM has extensive documentation, making it easy to get started and learn how to use the package.\\n8. Active community: LiteLLM has an active community of developers and users, making it easy to get help and feedback on your projects.\\n\\nOverall, LiteLLM can help you to simplify your workflow, improve your productivity, and make it easier to work with multiple language models and APIs.
\n",
- "
23.777885
\n",
- "
0.001443
\n",
- "
\n",
- "
\n",
- "
4
\n",
- "
gpt-3.5-turbo
\n",
- "
why should I use LiteLLM
\n",
- "
LiteLLM is a lightweight Python package that simplifies the process of making API calls to various language models. Here are some reasons why you should use LiteLLM:\\n\\n1. **Simplified API Calls**: LiteLLM abstracts away the complexity of making API calls to different language models. It provides a unified interface for invoking models from OpenAI, Azure, Cohere, Anthropic, Huggingface, and more.\\n\\n2. **Easy Integration**: LiteLLM seamlessly integrates with your existing codebase. You can import the package and start making API calls with just a few lines of code.\\n\\n3. **Flexibility**: LiteLLM supports a variety of language models, including GPT-3, GPT-Neo, chatGPT, and more. You can choose the model that suits your requirements and easily switch between them.\\n\\n4. **Convenience**: LiteLLM handles the authentication and connection details for you. You just need to set the relevant environment variables, and the package takes care of the rest.\\n\\n5. **Quick Prototyping**: LiteLLM is ideal for rapid prototyping and experimentation. With its simple API, you can quickly generate text, chat with models, and build interactive applications.\\n\\n6. **Community Support**: LiteLLM is actively maintained and supported by a community of developers. You can find help, share ideas, and collaborate with others to enhance your projects.\\n\\nOverall, LiteLLM simplifies the process of making API calls to language models, saving you time and effort while providing flexibility and convenience.
\n",
- "
12.109881
\n",
- "
0.000881
\n",
- "
\n",
- "
\n",
- "
5
\n",
- "
claude-instant-1.2
\n",
- "
why should I use LiteLLM
\n",
- "
Here are some key reasons why you may want to consider using LiteLLM:\\n\\n- Simplifies calling multiple large language models - LiteLLM provides a unified API to call models from different providers like OpenAI, Azure, HuggingFace, Anthropic etc. This avoids having to deal with different APIs from each provider.\\n\\n- Easy to get started - LiteLLM is very lightweight and simple to install with just one pip install. It has minimal configuration needed with just API keys. This lowers the barrier to start experimenting with LLMs.\\n\\n- Standard interface - The completion API provides a standard way to send messages and get responses back regardless of underlying model. This keeps the code interface consistent. \\n\\n- Model agnostic - Your code doesn't need to change to switch between different models. You can easily compare performance of models from different providers.\\n\\n- Open source - As it is open source, the library can be expanded to support more models and providers over time via community contributions.\\n\\n- Cost effective - LiteLLM helps avoid vendor lock-in. You can optimize costs by testing multiple models and choosing the best suitable and cost-effective option for your needs.\\n\\nSo in summary, L
\\nTo make a completion call using litellm, you can use the `completion` function from the litellm package. Here's an example of how to use it:\\n```\\nfrom litellm import completion\\n\\n# Set ENV variables\\nos.environ[\"OPENAI_API_KEY\"] = \"your_openai_api_key\"\\nos.environ[\"COHERE_API_KEY\"] = \"your_cohere_api_key\"\\n\\n# Define the messages to be completed\\nmessages = [\\n {\\n \"content\": \"Hello, how are you?\",\\n \"role\": \"user\"\\n }\\n]\\n\\n# Make a completion call using OpenAI\\nresponse = completion(model=\"gpt-3.5-turbo\", messages=messages)\\n\\n# Make a completion call using Cohere\\nresponse = completion(\"command-nightly\", messages)\\n```\\nIn this example, we first set the ENV variables for the OpenAI and Cohere API keys. Then, we define a list of messages to be completed, which in this case contains a single message with the content \"Hello, how are you?\" and the role \"user\".\\n\\nNext, we make two completion calls using the `completion` function from litellm. The first call uses the OpenAI model `gpt-3.5-turbo` and passes in the list of messages. The second call uses the Cohere model `command-nightly` and passes in the same list of messages.\\n\\nThe `completion` function returns a response object that contains the completed messages. You can then use the `response.messages` attribute to access the completed messages.\\n\\nHere's an example of how to access the completed messages:\\n```\\n# Print the completed messages\\nprint(response.messages)\\n```\\nThis will print the completed messages, which should look something like this:\\n```\\n[{'content': 'Hello, how are you?', 'role': 'user', 'completion': 'I am doing well, thanks for asking.'}]\\n```\\nNote that the `completion` attribute contains the completed message. You can use this attribute to retrieve the completed message and use it in your application.
\n",
- "
32.192453
\n",
- "
0.001821
\n",
- "
\n",
- "
\n",
- "
10
\n",
- "
gpt-3.5-turbo
\n",
- "
write code to make a litellm completion call
\n",
- "
Sure! Here's an example code to make a completion call using the litellm package:\\n\\n```python\\nfrom litellm import completion\\n\\n# Set the model and messages\\nmodel = \"gpt-3.5-turbo\"\\nmessages = [\\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\\n {\"role\": \"user\", \"content\": \"Who won the world series in 2020?\"},\\n {\"role\": \"assistant\", \"content\": \"The Los Angeles Dodgers won the World Series in 2020.\"},\\n {\"role\": \"user\", \"content\": \"Where was it played?\"}\\n]\\n\\n# Make the completion call\\nresponse = completion(model=model, messages=messages)\\n\\n# Print the assistant's reply\\nassistant_reply = response[\"choices\"][0][\"message\"][\"content\"]\\nprint(\"Assistant: \", assistant_reply)\\n```\\n\\nMake sure you have the litellm package installed (`pip install litellm`) and set the necessary environment variables for the API keys before running this code.
\n",
- "
9.377155
\n",
- "
0.000686
\n",
- "
\n",
- "
\n",
- "
11
\n",
- "
claude-instant-1.2
\n",
- "
write code to make a litellm completion call
\n",
- "
Here is an example of making a completion call using litellm:\\n\\n```python\\nimport os\\nfrom litellm import completion\\n\\n# Set API keys as environment variables\\nos.environ[\"OPENAI_API_KEY\"] = \"your openai api key\" \\n\\n# Conversation context \\nmessages = [{\\n \"content\": \"Hello, how can I help you today?\",\\n \"role\": \"assistant\"\\n}]\\n\\n# Make completion call with GPT-3 model\\nresponse = completion(\\n model=\"gpt-3.5-turbo\", \\n messages=messages\\n)\\n\\nprint(response)\\n```\\n\\nTo break it down:\\n\\n- Import completion from litellm\\n- Set the OPENAI_API_KEY env var \\n- Define a messages list with the conversation context\\n- Call completion(), specifying the model (\"gpt-3.5-turbo\") and messages\\n- It will return the response from the API\\n- Print the response\\n\\nThis makes a simple completion call to OpenAI GPT-3 using litellm to handle the API details. You can also call other models like Cohere or Anthropic by specifying their name instead of the OpenAI
\n",
- "
9.839988
\n",
- "
0.001578
\n",
- "
\n",
- " \n",
- "
"
- ],
- "text/plain": [
- ""
- ]
- },
- "execution_count": 22,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "from IPython.core.interactiveshell import InteractiveShell\n",
- "InteractiveShell.ast_node_interactivity = \"all\"\n",
- "from IPython.display import HTML\n",
- "import pandas as pd\n",
- "\n",
- "df = pd.DataFrame(data)\n",
- "grouped_by_question = df.groupby('Question')\n",
- "\n",
- "for question, group_data in grouped_by_question:\n",
- " print(f\"Question: {question}\")\n",
- " HTML(group_data.to_html())\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "bmtAbC1rGVAm"
- },
- "source": [
- "## Use Case 2 - Rewrite user input concisely"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 23,
- "metadata": {
- "id": "boiHO1PhGXSL"
- },
- "outputs": [],
- "source": [
- "# enter your system prompt if you have one\n",
- "system_prompt = \"\"\"\n",
- "For a given user input, rewrite the input to make be more concise.\n",
- "\"\"\"\n",
- "\n",
- "# user input for re-writing questions\n",
- "questions = [\n",
- " \"LiteLLM is a lightweight Python package that simplifies the process of making API calls to various language models. Here are some reasons why you should use LiteLLM:\\n\\n1. **Simplified API Calls**: LiteLLM abstracts away the complexity of making API calls to different language models. It provides a unified interface for invoking models from OpenAI, Azure, Cohere, Anthropic, Huggingface, and more.\\n\\n2. **Easy Integration**: LiteLLM seamlessly integrates with your existing codebase. You can import the package and start making API calls with just a few lines of code.\\n\\n3. **Flexibility**: LiteLLM supports a variety of language models, including GPT-3, GPT-Neo, chatGPT, and more. You can choose the model that suits your requirements and easily switch between them.\\n\\n4. **Convenience**: LiteLLM handles the authentication and connection details for you. You just need to set the relevant environment variables, and the package takes care of the rest.\\n\\n5. **Quick Prototyping**: LiteLLM is ideal for rapid prototyping and experimentation. With its simple API, you can quickly generate text, chat with models, and build interactive applications.\\n\\n6. **Community Support**: LiteLLM is actively maintained and supported by a community of developers. You can find help, share ideas, and collaborate with others to enhance your projects.\\n\\nOverall, LiteLLM simplifies the process of making API calls to language models, saving you time and effort while providing flexibility and convenience\",\n",
- " \"Hi everyone! I'm [your name] and I'm currently working on [your project/role involving LLMs]. I came across LiteLLM and was really excited by how it simplifies working with different LLM providers. I'm hoping to use LiteLLM to [build an app/simplify my code/test different models etc]. Before finding LiteLLM, I was struggling with [describe any issues you faced working with multiple LLMs]. With LiteLLM's unified API and automatic translation between providers, I think it will really help me to [goals you have for using LiteLLM]. Looking forward to being part of this community and learning more about how I can build impactful applications powered by LLMs!Let me know if you would like me to modify or expand on any part of this suggested intro. I'm happy to provide any clarification or additional details you need!\",\n",
- " \"Traceloop is a platform for monitoring and debugging the quality of your LLM outputs. It provides you with a way to track the performance of your LLM application; rollout changes with confidence; and debug issues in production. It is based on OpenTelemetry, so it can provide full visibility to your LLM requests, as well vector DB usage, and other infra in your stack.\"\n",
- "]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "fwNcC_obICUc"
- },
- "source": [
- "## Run Questions"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "KtBjZ1mUIBiJ"
- },
- "outputs": [],
- "source": [
- "from litellm import completion, completion_cost\n",
- "import os\n",
- "import time\n",
- "\n",
- "# optional use litellm dashboard to view logs\n",
- "# litellm.use_client = True\n",
- "# litellm.token = \"ishaan_2@berri.ai\" # set your email\n",
- "\n",
- "os.environ['TOGETHERAI_API_KEY'] = \"\"\n",
- "os.environ['OPENAI_API_KEY'] = \"\"\n",
- "os.environ['ANTHROPIC_API_KEY'] = \"\"\n",
- "\n",
- "models = ['togethercomputer/llama-2-70b-chat', 'gpt-3.5-turbo', 'claude-instant-1.2'] # enter llms to benchmark\n",
- "data_2 = []\n",
- "\n",
- "for question in questions: # group by question\n",
- " for model in models:\n",
- " print(f\"running question: {question} for model: {model}\")\n",
- " start_time = time.time()\n",
- " # show response, response time, cost for each question\n",
- " response = completion(\n",
- " model=model,\n",
- " max_tokens=500,\n",
- " messages = [\n",
- " {\n",
- " \"role\": \"system\", \"content\": system_prompt\n",
- " },\n",
- " {\n",
- " \"role\": \"user\", \"content\": \"User input:\" + question\n",
- " }\n",
- " ],\n",
- " )\n",
- " end = time.time()\n",
- " total_time = end-start_time # response time\n",
- " # print(response)\n",
- " cost = completion_cost(response) # cost for completion\n",
- " raw_response = response['choices'][0]['message']['content'] # response string\n",
- " #print(raw_response, total_time, cost)\n",
- "\n",
- " # add to pandas df\n",
- " data_2.append(\n",
- " {\n",
- " 'Model': model,\n",
- " 'Question': question,\n",
- " 'Response': raw_response,\n",
- " 'ResponseTime': total_time,\n",
- " 'Cost': cost\n",
- " })\n",
- "\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "-PCYIzG5M0II"
- },
- "source": [
- "## View Logs - Group by Question"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 20,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 1000
- },
- "id": "-3R5-2q8IiL2",
- "outputId": "c4a0d9e5-bb21-4de0-fc4c-9f5e71d0f177"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Question: Hi everyone! I'm [your name] and I'm currently working on [your project/role involving LLMs]. I came across LiteLLM and was really excited by how it simplifies working with different LLM providers. I'm hoping to use LiteLLM to [build an app/simplify my code/test different models etc]. Before finding LiteLLM, I was struggling with [describe any issues you faced working with multiple LLMs]. With LiteLLM's unified API and automatic translation between providers, I think it will really help me to [goals you have for using LiteLLM]. Looking forward to being part of this community and learning more about how I can build impactful applications powered by LLMs!Let me know if you would like me to modify or expand on any part of this suggested intro. I'm happy to provide any clarification or additional details you need!\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "
\n",
- " \n",
- "
\n",
- "
\n",
- "
Model
\n",
- "
Question
\n",
- "
Response
\n",
- "
ResponseTime
\n",
- "
Cost
\n",
- "
\n",
- " \n",
- " \n",
- "
\n",
- "
3
\n",
- "
togethercomputer/llama-2-70b-chat
\n",
- "
Hi everyone! I'm [your name] and I'm currently working on [your project/role involving LLMs]. I came across LiteLLM and was really excited by how it simplifies working with different LLM providers. I'm hoping to use LiteLLM to [build an app/simplify my code/test different models etc]. Before finding LiteLLM, I was struggling with [describe any issues you faced working with multiple LLMs]. With LiteLLM's unified API and automatic translation between providers, I think it will really help me to [goals you have for using LiteLLM]. Looking forward to being part of this community and learning more about how I can build impactful applications powered by LLMs!Let me know if you would like me to modify or expand on any part of this suggested intro. I'm happy to provide any clarification or additional details you need!
\n",
- "
\\nHere's a more concise version of the user input:\\n\\n\"Hi everyone! I'm [your name] and I'm working on [your project/role involving LLMs]. I recently discovered LiteLLM and I'm excited to use it to [build an app/simplify my code/test different models etc]. Before LiteLLM, I struggled with [describe any issues you faced working with multiple LLMs]. I'm looking forward to using LiteLLM's unified API and automatic translation to achieve my goals. I'm eager to learn more about building impactful applications powered by LLMs and to be part of this community. Let me know if you have any questions or need further clarification.\"\\n\\nIn this revised version, we've kept the essential information and removed some of the extraneous language. We've also rephrased some of the sentences to make them more concise and easier to read.
\n",
- "
18.300620
\n",
- "
0.001200
\n",
- "
\n",
- "
\n",
- "
4
\n",
- "
gpt-3.5-turbo
\n",
- "
Hi everyone! I'm [your name] and I'm currently working on [your project/role involving LLMs]. I came across LiteLLM and was really excited by how it simplifies working with different LLM providers. I'm hoping to use LiteLLM to [build an app/simplify my code/test different models etc]. Before finding LiteLLM, I was struggling with [describe any issues you faced working with multiple LLMs]. With LiteLLM's unified API and automatic translation between providers, I think it will really help me to [goals you have for using LiteLLM]. Looking forward to being part of this community and learning more about how I can build impactful applications powered by LLMs!Let me know if you would like me to modify or expand on any part of this suggested intro. I'm happy to provide any clarification or additional details you need!
\n",
- "
User input: Hi, I'm [your name] and I'm excited about using LiteLLM to simplify working with different LLM providers. Before finding LiteLLM, I faced challenges working with multiple LLMs. With LiteLLM's unified API and automatic translation, I believe it will help me achieve my goals of [state your goals]. I look forward to being part of this community and learning how to build impactful applications with LLMs. Let me know if you need any further clarification or details.
\n",
- "
7.385472
\n",
- "
0.000525
\n",
- "
\n",
- "
\n",
- "
5
\n",
- "
claude-instant-1.2
\n",
- "
Hi everyone! I'm [your name] and I'm currently working on [your project/role involving LLMs]. I came across LiteLLM and was really excited by how it simplifies working with different LLM providers. I'm hoping to use LiteLLM to [build an app/simplify my code/test different models etc]. Before finding LiteLLM, I was struggling with [describe any issues you faced working with multiple LLMs]. With LiteLLM's unified API and automatic translation between providers, I think it will really help me to [goals you have for using LiteLLM]. Looking forward to being part of this community and learning more about how I can build impactful applications powered by LLMs!Let me know if you would like me to modify or expand on any part of this suggested intro. I'm happy to provide any clarification or additional details you need!
\n",
- "
Here is a more concise rewrite of the user input:\\n\\nHi everyone, I'm [your name]. I'm currently [your project/role] and came across LiteLLM, which simplifies working with different LLMs through its unified API. I hope to [build an app/simplify code/test models] with LiteLLM since I previously struggled with [issues]. LiteLLM's automatic translation between providers will help me [goals] and build impactful LLM applications. Looking forward to learning more as part of this community. Let me know if you need any clarification on my plans to use LiteLLM.
\n",
- "
8.628217
\n",
- "
0.001022
\n",
- "
\n",
- " \n",
- "
"
- ],
- "text/plain": [
- ""
- ]
- },
- "execution_count": 20,
- "metadata": {},
- "output_type": "execute_result"
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Question: LiteLLM is a lightweight Python package that simplifies the process of making API calls to various language models. Here are some reasons why you should use LiteLLM:\n",
- "\n",
- "1. **Simplified API Calls**: LiteLLM abstracts away the complexity of making API calls to different language models. It provides a unified interface for invoking models from OpenAI, Azure, Cohere, Anthropic, Huggingface, and more.\n",
- "\n",
- "2. **Easy Integration**: LiteLLM seamlessly integrates with your existing codebase. You can import the package and start making API calls with just a few lines of code.\n",
- "\n",
- "3. **Flexibility**: LiteLLM supports a variety of language models, including GPT-3, GPT-Neo, chatGPT, and more. You can choose the model that suits your requirements and easily switch between them.\n",
- "\n",
- "4. **Convenience**: LiteLLM handles the authentication and connection details for you. You just need to set the relevant environment variables, and the package takes care of the rest.\n",
- "\n",
- "5. **Quick Prototyping**: LiteLLM is ideal for rapid prototyping and experimentation. With its simple API, you can quickly generate text, chat with models, and build interactive applications.\n",
- "\n",
- "6. **Community Support**: LiteLLM is actively maintained and supported by a community of developers. You can find help, share ideas, and collaborate with others to enhance your projects.\n",
- "\n",
- "Overall, LiteLLM simplifies the process of making API calls to language models, saving you time and effort while providing flexibility and convenience\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "
\n",
- " \n",
- "
\n",
- "
\n",
- "
Model
\n",
- "
Question
\n",
- "
Response
\n",
- "
ResponseTime
\n",
- "
Cost
\n",
- "
\n",
- " \n",
- " \n",
- "
\n",
- "
0
\n",
- "
togethercomputer/llama-2-70b-chat
\n",
- "
LiteLLM is a lightweight Python package that simplifies the process of making API calls to various language models. Here are some reasons why you should use LiteLLM:\\n\\n1. **Simplified API Calls**: LiteLLM abstracts away the complexity of making API calls to different language models. It provides a unified interface for invoking models from OpenAI, Azure, Cohere, Anthropic, Huggingface, and more.\\n\\n2. **Easy Integration**: LiteLLM seamlessly integrates with your existing codebase. You can import the package and start making API calls with just a few lines of code.\\n\\n3. **Flexibility**: LiteLLM supports a variety of language models, including GPT-3, GPT-Neo, chatGPT, and more. You can choose the model that suits your requirements and easily switch between them.\\n\\n4. **Convenience**: LiteLLM handles the authentication and connection details for you. You just need to set the relevant environment variables, and the package takes care of the rest.\\n\\n5. **Quick Prototyping**: LiteLLM is ideal for rapid prototyping and experimentation. With its simple API, you can quickly generate text, chat with models, and build interactive applications.\\n\\n6. **Community Support**: LiteLLM is actively maintained and supported by a community of developers. You can find help, share ideas, and collaborate with others to enhance your projects.\\n\\nOverall, LiteLLM simplifies the process of making API calls to language models, saving you time and effort while providing flexibility and convenience
\n",
- "
Here's a more concise version of the user input:\\n\\nLiteLLM is a lightweight Python package that simplifies API calls to various language models. It abstracts away complexity, integrates seamlessly, supports multiple models, and handles authentication. It's ideal for rapid prototyping and has community support. It saves time and effort while providing flexibility and convenience.
\n",
- "
11.294250
\n",
- "
0.001251
\n",
- "
\n",
- "
\n",
- "
1
\n",
- "
gpt-3.5-turbo
\n",
- "
LiteLLM is a lightweight Python package that simplifies the process of making API calls to various language models. Here are some reasons why you should use LiteLLM:\\n\\n1. **Simplified API Calls**: LiteLLM abstracts away the complexity of making API calls to different language models. It provides a unified interface for invoking models from OpenAI, Azure, Cohere, Anthropic, Huggingface, and more.\\n\\n2. **Easy Integration**: LiteLLM seamlessly integrates with your existing codebase. You can import the package and start making API calls with just a few lines of code.\\n\\n3. **Flexibility**: LiteLLM supports a variety of language models, including GPT-3, GPT-Neo, chatGPT, and more. You can choose the model that suits your requirements and easily switch between them.\\n\\n4. **Convenience**: LiteLLM handles the authentication and connection details for you. You just need to set the relevant environment variables, and the package takes care of the rest.\\n\\n5. **Quick Prototyping**: LiteLLM is ideal for rapid prototyping and experimentation. With its simple API, you can quickly generate text, chat with models, and build interactive applications.\\n\\n6. **Community Support**: LiteLLM is actively maintained and supported by a community of developers. You can find help, share ideas, and collaborate with others to enhance your projects.\\n\\nOverall, LiteLLM simplifies the process of making API calls to language models, saving you time and effort while providing flexibility and convenience
\n",
- "
LiteLLM is a lightweight Python package that simplifies API calls to various language models. Here's why you should use it:\\n1. Simplified API Calls: Works with multiple models (OpenAI, Azure, Cohere, Anthropic, Huggingface).\\n2. Easy Integration: Import and start using it quickly in your codebase.\\n3. Flexibility: Supports GPT-3, GPT-Neo, chatGPT, etc. easily switch between models.\\n4. Convenience: Handles authentication and connection details, just set environment variables.\\n5. Quick Prototyping: Great for rapid prototyping and building interactive applications.\\n6. Community Support: Actively maintained and supported by a developer community.
\n",
- "
9.778315
\n",
- "
0.000795
\n",
- "
\n",
- "
\n",
- "
2
\n",
- "
claude-instant-1.2
\n",
- "
LiteLLM is a lightweight Python package that simplifies the process of making API calls to various language models. Here are some reasons why you should use LiteLLM:\\n\\n1. **Simplified API Calls**: LiteLLM abstracts away the complexity of making API calls to different language models. It provides a unified interface for invoking models from OpenAI, Azure, Cohere, Anthropic, Huggingface, and more.\\n\\n2. **Easy Integration**: LiteLLM seamlessly integrates with your existing codebase. You can import the package and start making API calls with just a few lines of code.\\n\\n3. **Flexibility**: LiteLLM supports a variety of language models, including GPT-3, GPT-Neo, chatGPT, and more. You can choose the model that suits your requirements and easily switch between them.\\n\\n4. **Convenience**: LiteLLM handles the authentication and connection details for you. You just need to set the relevant environment variables, and the package takes care of the rest.\\n\\n5. **Quick Prototyping**: LiteLLM is ideal for rapid prototyping and experimentation. With its simple API, you can quickly generate text, chat with models, and build interactive applications.\\n\\n6. **Community Support**: LiteLLM is actively maintained and supported by a community of developers. You can find help, share ideas, and collaborate with others to enhance your projects.\\n\\nOverall, LiteLLM simplifies the process of making API calls to language models, saving you time and effort while providing flexibility and convenience
\n",
- "
Here is a concise rewrite of the user input:\\n\\nLiteLLM is a lightweight Python package that simplifies accessing various language models. It provides a unified interface for models from OpenAI, Azure, Cohere, Anthropic, Huggingface, and more. Key benefits include simplified API calls, easy integration, flexibility to use different models, automated handling of authentication, and support for quick prototyping. The actively maintained package saves time by abstracting away complexity while offering convenience and a collaborative community.
\n",
- "
7.697520
\n",
- "
0.001098
\n",
- "
\n",
- " \n",
- "
"
- ],
- "text/plain": [
- ""
- ]
- },
- "execution_count": 20,
- "metadata": {},
- "output_type": "execute_result"
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Question: Traceloop is a platform for monitoring and debugging the quality of your LLM outputs. It provides you with a way to track the performance of your LLM application; rollout changes with confidence; and debug issues in production. It is based on OpenTelemetry, so it can provide full visibility to your LLM requests, as well vector DB usage, and other infra in your stack.\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "
\n",
- " \n",
- "
\n",
- "
\n",
- "
Model
\n",
- "
Question
\n",
- "
Response
\n",
- "
ResponseTime
\n",
- "
Cost
\n",
- "
\n",
- " \n",
- " \n",
- "
\n",
- "
6
\n",
- "
togethercomputer/llama-2-70b-chat
\n",
- "
Traceloop is a platform for monitoring and debugging the quality of your LLM outputs. It provides you with a way to track the performance of your LLM application; rollout changes with confidence; and debug issues in production. It is based on OpenTelemetry, so it can provide full visibility to your LLM requests, as well vector DB usage, and other infra in your stack.
\n",
- "
\\nRewritten input: Traceloop is a platform for monitoring and debugging LLM outputs. It allows users to track performance, rollout changes confidently, and debug issues in production. It uses OpenTelemetry for full visibility into LLM requests, vector DB usage, and other infrastructure.
\n",
- "
9.060444
\n",
- "
0.000525
\n",
- "
\n",
- "
\n",
- "
7
\n",
- "
gpt-3.5-turbo
\n",
- "
Traceloop is a platform for monitoring and debugging the quality of your LLM outputs. It provides you with a way to track the performance of your LLM application; rollout changes with confidence; and debug issues in production. It is based on OpenTelemetry, so it can provide full visibility to your LLM requests, as well vector DB usage, and other infra in your stack.
\n",
- "
Traceloop is a platform for monitoring and debugging the quality of your LLM outputs. It helps track performance, rollout changes, and debug issues in production. It is based on OpenTelemetry, providing visibility to LLM requests, vector DB usage, and other infrastructure in your stack.
\n",
- "
7.304661
\n",
- "
0.000283
\n",
- "
\n",
- "
\n",
- "
8
\n",
- "
claude-instant-1.2
\n",
- "
Traceloop is a platform for monitoring and debugging the quality of your LLM outputs. It provides you with a way to track the performance of your LLM application; rollout changes with confidence; and debug issues in production. It is based on OpenTelemetry, so it can provide full visibility to your LLM requests, as well vector DB usage, and other infra in your stack.
\n",
- "
Here is a more concise rewrite of the user input:\\n\\nTraceloop monitors and debugs LLM quality. It tracks LLM performance, enables confident changes, and debugs production issues. Based on OpenTelemetry, Traceloop provides full visibility into LLM requests, vector DB usage, and other stack infrastructure.
\n"
- ],
- "text/plain": [
- "Model Name claude-instant-1 \\\n",
- "Prompt \n",
- "\\nIs paul graham a writer? Yes, Paul Graham is considered a writer in ad... \n",
- "\\nWhat has Paul Graham done? Paul Graham has made significant contribution... \n",
- "\\nWhat is Paul Graham known for? Paul Graham is known for several things:\\n\\n-... \n",
- "\\nWhere does Paul Graham live? Based on the information provided:\\n\\n- Paul ... \n",
- "\\nWho is Paul Graham? Paul Graham is an influential computer scient... \n",
- "\n",
- "Model Name gpt-3.5-turbo-0613 \\\n",
- "Prompt \n",
- "\\nIs paul graham a writer? Yes, Paul Graham is a writer. He has written s... \n",
- "\\nWhat has Paul Graham done? Paul Graham has achieved several notable accom... \n",
- "\\nWhat is Paul Graham known for? Paul Graham is known for his work on the progr... \n",
- "\\nWhere does Paul Graham live? According to the given information, Paul Graha... \n",
- "\\nWho is Paul Graham? Paul Graham is an English computer scientist, ... \n",
- "\n",
- "Model Name gpt-3.5-turbo-16k-0613 \\\n",
- "Prompt \n",
- "\\nIs paul graham a writer? Yes, Paul Graham is a writer. He has authored ... \n",
- "\\nWhat has Paul Graham done? Paul Graham has made significant contributions... \n",
- "\\nWhat is Paul Graham known for? Paul Graham is known for his work on the progr... \n",
- "\\nWhere does Paul Graham live? Paul Graham currently lives in England, where ... \n",
- "\\nWho is Paul Graham? Paul Graham is an English computer scientist, ... \n",
- "\n",
- "Model Name gpt-4-0613 \\\n",
- "Prompt \n",
- "\\nIs paul graham a writer? Yes, Paul Graham is a writer. He is an essayis... \n",
- "\\nWhat has Paul Graham done? Paul Graham is known for his work on the progr... \n",
- "\\nWhat is Paul Graham known for? Paul Graham is known for his work on the progr... \n",
- "\\nWhere does Paul Graham live? The text does not provide a current place of r... \n",
- "\\nWho is Paul Graham? Paul Graham is an English computer scientist, ... \n",
- "\n",
- "Model Name replicate/llama-2-70b-chat:58d078176e02c219e11eb4da5a02a7830a283b14cf8f94537af893ccff5ee781 \n",
- "Prompt \n",
- "\\nIs paul graham a writer? Yes, Paul Graham is an author. According to t... \n",
- "\\nWhat has Paul Graham done? Paul Graham has had a diverse career in compu... \n",
- "\\nWhat is Paul Graham known for? Paul Graham is known for many things, includi... \n",
- "\\nWhere does Paul Graham live? Based on the information provided, Paul Graha... \n",
- "\\nWho is Paul Graham? Paul Graham is an English computer scientist,... "
- ]
- },
- "execution_count": 17,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "import pandas as pd\n",
- "\n",
- "# Create an empty list to store the row data\n",
- "table_data = []\n",
- "\n",
- "# Iterate through the list and extract the required data\n",
- "for item in result:\n",
- " prompt = item['prompt'][0]['content'].replace(context, \"\") # clean the prompt for easy comparison\n",
- " model = item['response']['model']\n",
- " response = item['response']['choices'][0]['message']['content']\n",
- " table_data.append([prompt, model, response])\n",
- "\n",
- "# Create a DataFrame from the table data\n",
- "df = pd.DataFrame(table_data, columns=['Prompt', 'Model Name', 'Response'])\n",
- "\n",
- "# Pivot the DataFrame to get the desired table format\n",
- "table = df.pivot(index='Prompt', columns='Model Name', values='Response')\n",
- "table"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "zOxUM40PINDC"
- },
- "source": [
- "# Load Test endpoint\n",
- "\n",
- "Run 100+ simultaneous queries across multiple providers to see when they fail + impact on latency"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "ZkQf_wbcIRQ9"
- },
- "outputs": [],
- "source": [
- "models=[\"gpt-3.5-turbo\", \"replicate/llama-2-70b-chat:58d078176e02c219e11eb4da5a02a7830a283b14cf8f94537af893ccff5ee781\", \"claude-instant-1\"]\n",
- "context = \"\"\"Paul Graham (/ɔrƦm/; born 1964)[3] is an English computer scientist, essayist, entrepreneur, venture capitalist, and author. He is best known for his work on the programming language Lisp, his former startup Viaweb (later renamed Yahoo! Store), cofounding the influential startup accelerator and seed capital firm Y Combinator, his essays, and Hacker News. He is the author of several computer programming books, including: On Lisp,[4] ANSI Common Lisp,[5] and Hackers & Painters.[6] Technology journalist Steven Levy has described Graham as a \"hacker philosopher\".[7] Graham was born in England, where he and his family maintain permanent residence. However he is also a citizen of the United States, where he was educated, lived, and worked until 2016.\"\"\"\n",
- "prompt = \"Where does Paul Graham live?\"\n",
- "final_prompt = context + prompt\n",
- "result = load_test_model(models=models, prompt=final_prompt, num_calls=5)"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "8vSNBFC06aXY"
- },
- "source": [
- "## Visualize the data"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 552
- },
- "id": "SZfiKjLV3-n8",
- "outputId": "00f7f589-b3da-43ed-e982-f9420f074b8d"
- },
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAIXCAYAAACy1HXAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABn5UlEQVR4nO3dd1QT2d8G8Cf0ojQBEUFRsSv2FXvvvSx2saNi7733ihXELotd7KuIir33sjZUsIuKVGmS+/7hy/yM6K7RYEZ4PufkaO5Mkm/IJHly594ZhRBCgIiIiEiGdLRdABEREdG3MKgQERGRbDGoEBERkWwxqBAREZFsMagQERGRbDGoEBERkWwxqBAREZFsMagQERGRbDGoEBERkWwxqBCR7Dk5OaFLly7aLkNtc+fORd68eaGrq4uSJUtquxyNO3bsGBQKBbZv367tUtSmUCgwadIktW8XGhoKhUKBdevWabwm+joGFVKxfPlyKBQKlC9fXtulyI6TkxMUCoV0MTU1xR9//IENGzZou7TfTuoX3PdcfleHDh3CiBEjUKlSJaxduxYzZszQdkmys27dOul1PnXqVJrlQgg4OjpCoVCgcePGWqiQ5EBP2wWQvPj7+8PJyQkXLlxASEgInJ2dtV2SrJQsWRJDhw4FALx8+RKrVq2Cu7s7EhMT0bNnTy1X9/soXLgw/Pz8VNpGjx6NLFmyYOzYsWnWv3fvHnR0fq/fVUePHoWOjg5Wr14NAwMDbZcja0ZGRti4cSMqV66s0n78+HE8e/YMhoaGWqqM5IBBhSSPHz/GmTNnEBAQAA8PD/j7+2PixIm/tAalUomkpCQYGRn90sf9Xjlz5kTHjh2l6126dEHevHmxcOFCBhU1ZM+eXeXvCACzZs2CtbV1mnYAv+UXVXh4OIyNjTUWUoQQSEhIgLGxsUbuT04aNmyIbdu2YfHixdDT+9/X0saNG1GmTBm8fftWi9WRtv1eP1EoXfn7+8PS0hKNGjVC69at4e/vLy1LTk6GlZUVunbtmuZ20dHRMDIywrBhw6S2xMRETJw4Ec7OzjA0NISjoyNGjBiBxMREldsqFAr069cP/v7+KFq0KAwNDXHw4EEAwLx581CxYkVky5YNxsbGKFOmzFf3hcfHx2PAgAGwtrZG1qxZ0bRpUzx//vyr+6CfP3+Obt26IXv27DA0NETRokWxZs2aH/6b2djYoFChQnj48KFKu1KphJeXF4oWLQojIyNkz54dHh4eeP/+vcp6ly5dQr169WBtbQ1jY2PkyZMH3bp1k5an7g+fN28eFi5ciNy5c8PY2BjVqlXDrVu30tRz9OhRVKlSBaamprCwsECzZs1w584dlXUmTZoEhUKBkJAQdOnSBRYWFjA3N0fXrl3x4cMHlXWDgoJQuXJlWFhYIEuWLChYsCDGjBmjss73vtY/48sxKqm7DE6dOoUBAwbAxsYGFhYW8PDwQFJSEiIjI9G5c2dYWlrC0tISI0aMwJcnitfUa/Q1CoUCa9euRVxcnLRrI3VMw8ePHzF16lTky5cPhoaGcHJywpgxY9L8vZycnNC4cWMEBgaibNmyMDY2xooVK/71cc+fP4/69evD3NwcJiYmqFatGk6fPq2yTlhYGPr27YuCBQvC2NgY2bJlw59//onQ0NA09xcZGYnBgwfDyckJhoaGcHBwQOfOndMEB6VSienTp8PBwQFGRkaoVasWQkJC/rXWz7Vr1w7v3r1DUFCQ1JaUlITt27ejffv2X71NXFwchg4dCkdHRxgaGqJgwYKYN29emtc5MTERgwcPho2NjfT58OzZs6/ep6Y/H0hDBNH/K1SokOjevbsQQogTJ04IAOLChQvS8m7dugkLCwuRmJiocrv169cLAOLixYtCCCFSUlJE3bp1hYmJiRg0aJBYsWKF6Nevn9DT0xPNmjVTuS0AUbhwYWFjYyMmT54sli1bJq5evSqEEMLBwUH07dtXLF26VCxYsED88ccfAoDYt2+fyn24ubkJAKJTp05i2bJlws3NTZQoUUIAEBMnTpTWe/XqlXBwcBCOjo5iypQpwtvbWzRt2lQAEAsXLvzPv0/u3LlFo0aNVNqSk5OFnZ2dyJ49u0p7jx49hJ6enujZs6fw8fERI0eOFKampqJcuXIiKSlJCCHE69evhaWlpShQoICYO3euWLlypRg7dqwoXLiwdD+PHz8WAETx4sWFk5OTmD17tpg8ebKwsrISNjY24tWrV9K6QUFBQk9PTxQoUEDMmTNHTJ48WVhbWwtLS0vx+PFjab2JEycKAKJUqVKiZcuWYvny5aJHjx4CgBgxYoS03q1bt4SBgYEoW7asWLRokfDx8RHDhg0TVatWldZR57X+L0WLFhXVqlX75t/e3d1dur527VoBQJQsWVLUr19fLFu2THTq1El6DpUrVxbt27cXy5cvF40bNxYAxPr169PlNfoaPz8/UaVKFWFoaCj8/PyEn5+fePjwoRBCCHd3dwFAtG7dWixbtkx07txZABDNmzdP85ydnZ2FpaWlGDVqlPDx8RHBwcHffMwjR44IAwMDUaFCBTF//nyxcOFC4eLiIgwMDMT58+el9bZt2yZKlCghJkyYIHx9fcWYMWOEpaWlyJ07t4iLi5PWi4mJEcWKFRO6urqiZ8+ewtvbW0ydOlWUK1dOeo8GBwdL21KZMmXEwoULxaRJk4SJiYn4448//vVv9PnrePHiRVGxYkXRqVMnadmuXbuEjo6OeP78eZr3nlKpFDVr1hQKhUL06NFDLF26VDRp0kQAEIMGDVJ5jI4dOwoAon379mLp0qWiZcuWwsXF5Yc/H1Lfk2vXrv3P50eawaBCQgghLl26JACIoKAgIcSnDwIHBwcxcOBAaZ3AwEABQOzdu1fltg0bNhR58+aVrvv5+QkdHR1x8uRJlfV8fHwEAHH69GmpDYDQ0dERt2/fTlPThw8fVK4nJSWJYsWKiZo1a0ptly9f/uqHU5cuXdJ8EHXv3l3kyJFDvH37VmXdtm3bCnNz8zSP96XcuXOLunXrijdv3og3b96ImzdvSl+Onp6e0nonT54UAIS/v7/K7Q8ePKjSvnPnTpWA9zWpH4rGxsbi2bNnUvv58+cFADF48GCprWTJksLW1la8e/dOart+/brQ0dERnTt3ltpSg0q3bt1UHqtFixYiW7Zs0vWFCxcKAOLNmzffrE+d1/q//EhQqVevnlAqlVJ7hQoVhEKhEL1795baPn78KBwcHFTuW5Ov0be4u7sLU1NTlbZr164JAKJHjx4q7cOGDRMAxNGjR1WeMwBx8ODB/3wspVIp8ufPn+bv8eHDB5EnTx5Rp04dlbYvnT17VgAQGzZskNomTJggAIiAgICvPp4Q/wsqhQsXVvkBs2jRIgFA3Lx581/r/jyoLF26VGTNmlWq788//xQ1atSQ/hafB5Vdu3YJAGLatGkq99e6dWuhUChESEiIEOJ/f+++ffuqrNe+ffsf/nxgUPn1uOuHAHza7ZM9e3bUqFEDwKeu6zZt2mDz5s1ISUkBANSsWRPW1tbYsmWLdLv3798jKCgIbdq0kdq2bduGwoULo1ChQnj79q10qVmzJgAgODhY5bGrVauGIkWKpKnp833x79+/R1RUFKpUqYIrV65I7am7ifr27aty2/79+6tcF0Jgx44daNKkCYQQKnXVq1cPUVFRKvf7LYcOHYKNjQ1sbGxQvHhx+Pn5oWvXrpg7d67K8zc3N0edOnVUHqdMmTLIkiWL9PwtLCwAAPv27UNycvK/Pm7z5s2RM2dO6foff/yB8uXL4++//wbwaWDvtWvX0KVLF1hZWUnrubi4oE6dOtJ6n+vdu7fK9SpVquDdu3eIjo5WqW/37t1QKpVfrUvd11rTunfvrjIzqHz58hBCoHv37lKbrq4uypYti0ePHqnUrenX6Hukvg5DhgxRaU8doL1//36V9jx58qBevXr/eb/Xrl3DgwcP0L59e7x79056PnFxcahVqxZOnDghvYafv6+Sk5Px7t07ODs7w8LCQuU9sGPHDpQoUQItWrRI83hfzsbq2rWrylicKlWqAIDK3/y/uLm5IT4+Hvv27UNMTAz27dv3zd0+f//9N3R1dTFgwACV9qFDh0IIgQMHDkjrAUiz3qBBg1Sua+rzgdJHhgkqJ06cQJMmTWBvbw+FQoFdu3al+2M+f/4cHTt2lMZQFC9eHJcuXUr3x9W0lJQUbN68GTVq1MDjx48REhKCkJAQlC9fHq9fv8aRI0cAAHp6emjVqhV2794t7U8PCAhAcnKySlB58OABbt++LX2hp14KFCgA4NMgw8/lyZPnq3Xt27cPrq6uMDIygpWVFWxsbODt7Y2oqChpnbCwMOjo6KS5jy9nK7158waRkZHw9fVNU1fquJsv6/qa8uXLIygoCAcPHsS8efNgYWGB9+/fq3xIP3jwAFFRUbC1tU3zWLGxsdLjVKtWDa1atcLkyZNhbW2NZs2aYe3atV8d25E/f/40bQUKFJDGFYSFhQEAChYsmGa9woULS19an8uVK5fKdUtLSwCQxmi0adMGlSpVQo8ePZA9e3a0bdsWW7duVQkt6r7WmvblczA3NwcAODo6pmn/fOxJerxG3yN1e/1y+7Szs4OFhYX0Oqb61nvjSw8ePAAAuLu7p3k+q1atQmJiovS+iY+Px4QJE6SxHdbW1rCxsUFkZKTKe+vhw4coVqzYdz3+f21L38PGxga1a9fGxo0bERAQgJSUFLRu3fqr64aFhcHe3h5Zs2ZVaS9cuLC0PPVfHR0d5MuXT2W9L98nmvp8oPSRYWb9xMXFoUSJEujWrRtatmyZ7o/3/v17VKpUCTVq1MCBAwdgY2ODBw8eSG/Q38nRo0fx8uVLbN68GZs3b06z3N/fH3Xr1gUAtG3bFitWrMCBAwfQvHlzbN26FYUKFUKJEiWk9ZVKJYoXL44FCxZ89fG+/BL52iyGkydPomnTpqhatSqWL1+OHDlyQF9fH2vXrsXGjRvVfo6pX64dO3aEu7v7V9dxcXH5z/uxtrZG7dq1AQD16tVDoUKF0LhxYyxatEj6laxUKmFra6syGPlzNjY2ACAdKOvcuXPYu3cvAgMD0a1bN8yfPx/nzp1DlixZ1H6e6tDV1f1qu/j/wYjGxsY4ceIEgoODsX//fhw8eBBbtmxBzZo1cejQIejq6qr9Wmvat57D19rFZ4Mstf0afe/xYb53hk/q9j137txvHlgutdb+/ftj7dq1GDRoECpUqABzc3MoFAq0bdv2mz1n/+W/tqXv1b59e/Ts2ROvXr1CgwYNpB6t9KapzwdKHxkmqDRo0AANGjT45vLExESMHTsWmzZtQmRkJIoVK4bZs2ejevXqP/R4s2fPhqOjI9auXSu1fe+vH7nx9/eHra0tli1blmZZQEAAdu7cCR8fHxgbG6Nq1arIkSMHtmzZgsqVK+Po0aNpjnuRL18+XL9+HbVq1frhA3bt2LEDRkZGCAwMVJma+vnfGwBy584NpVKJx48fq/Q6fDnjIHXEf0pKihQ0NKFRo0aoVq0aZsyYAQ8PD5iamiJfvnw4fPgwKlWq9F1fNK6urnB1dcX06dOxceNGdOjQAZs3b0aPHj2kdVJ/MX/u/v37cHJyAvDp7wB8Ot7Il+7evQtra2uYmpqq/fx0dHRQq1Yt1KpVCwsWLMCMGTMwduxYBAcHo3bt2hp5rbUhPV6j75G6vT548ED69Q8Ar1+/RmRkpPQ6qiu1x8DMzOw/t+/t27fD3d0d8+fPl9oSEhIQGRmZ5j6/NrMsPbVo0QIeHh44d+6cyi7mL+XOnRuHDx9GTEyMSq/K3bt3peWp/yqVSjx8+FClF+XL90l6fT6QZmSYXT//pV+/fjh79iw2b96MGzdu4M8//0T9+vW/+gXwPfbs2YOyZcvizz//hK2tLUqVKoWVK1dquOr0Fx8fj4CAADRu3BitW7dOc+nXrx9iYmKwZ88eAJ++uFq3bo29e/fCz88PHz9+VNntA3za1/z8+fOv/j3i4+PT7IL4Gl1dXSgUCml8DPBpqu6Xu/RS998vX75cpX3JkiVp7q9Vq1bYsWPHVz9837x58581fcvIkSPx7t076fm6ubkhJSUFU6dOTbPux48fpS+E9+/fp/nFmfpr+MtdC7t27cLz58+l6xcuXMD58+elcJ4jRw6ULFkS69evV/nCuXXrFg4dOoSGDRuq/bwiIiLStH1ZnyZea21Ij9foe6S+Dl5eXirtqT1SjRo1Uvs+AaBMmTLIly8f5s2bh9jY2DTLP9++dXV10zynJUuWqLzXAKBVq1a4fv06du7cmeb+1O0p+V5ZsmSBt7c3Jk2ahCZNmnxzvYYNGyIlJQVLly5VaV+4cCEUCoX0vkj9d/HixSrrffn3T8/PB/p5GaZH5d88efIEa9euxZMnT2Bvbw8AGDZsGA4ePPjDh7Z+9OgRvL29MWTIEIwZMwYXL17EgAEDYGBg8M2uQznas2cPYmJi0LRp068ud3V1hY2NDfz9/aVA0qZNGyxZsgQTJ05E8eLFVX4ZAkCnTp2wdetW9O7dG8HBwahUqRJSUlJw9+5dbN26VTouxL9p1KgRFixYgPr166N9+/YIDw/HsmXL4OzsjBs3bkjrlSlTBq1atYKXlxfevXsHV1dXHD9+HPfv3weg2sU+a9YsBAcHo3z58ujZsyeKFCmCiIgIXLlyBYcPH/7qF/P3aNCgAYoVK4YFCxbA09MT1apVg4eHB2bOnIlr166hbt260NfXx4MHD7Bt2zYsWrQIrVu3xvr167F8+XK0aNEC+fLlQ0xMDFauXAkzM7M0wcLZ2RmVK1dGnz59kJiYCC8vL2TLlg0jRoyQ1pk7dy4aNGiAChUqoHv37oiPj8eSJUtgbm7+Q+c0mTJlCk6cOIFGjRohd+7cCA8Px/Lly+Hg4CAdQVQTr7U2pMdr9D1KlCgBd3d3+Pr6IjIyEtWqVcOFCxewfv16NG/eXBrMri4dHR2sWrUKDRo0QNGiRdG1a1fkzJkTz58/R3BwMMzMzLB3714AQOPGjeHn5wdzc3MUKVIEZ8+exeHDh5EtWzaV+xw+fDi2b9+OP//8E926dUOZMmUQERGBPXv2wMfHR2V3ryZ9z+dnkyZNUKNGDYwdOxahoaEoUaIEDh06hN27d2PQoEFSD1PJkiXRrl07LF++HFFRUahYsSKOHDny1WO8pNfnA2mAVuYapTMAYufOndL1ffv2CQDC1NRU5aKnpyfc3NyEEELcuXNHAPjXy8iRI6X71NfXFxUqVFB53P79+wtXV9df8hw1pUmTJsLIyEjl+Alf6tKli9DX15em7SmVSuHo6PjV6YGpkpKSxOzZs0XRokWFoaGhsLS0FGXKlBGTJ08WUVFR0nr4Ymrv51avXi3y588vDA0NRaFChcTatWulqbWfi4uLE56ensLKykpkyZJFNG/eXNy7d08AELNmzVJZ9/Xr18LT01M4OjoKfX19YWdnJ2rVqiV8fX3/82/1teOopFq3bl2aKYu+vr6iTJkywtjYWGTNmlUUL15cjBgxQrx48UIIIcSVK1dEu3btRK5cuYShoaGwtbUVjRs3FpcuXZLuI3Uq5Ny5c8X8+fOFo6OjMDQ0FFWqVBHXr19PU8fhw4dFpUqVhLGxsTAzMxNNmjQR//zzj8o6qX/DL6cdp04VTT3mypEjR0SzZs2Evb29MDAwEPb29qJdu3bi/v37Krf73tf6v/zI9OQvpw1/67l9baqwEJp5jb7lW4+ZnJwsJk+eLPLkySP09fWFo6OjGD16tEhISEjznL+1vX3L1atXRcuWLUW2bNmEoaGhyJ07t3BzcxNHjhyR1nn//r3o2rWrsLa2FlmyZBH16tUTd+/eTfM3FkKId+/eiX79+omcOXMKAwMD4eDgINzd3aXPgtTpydu2bVO53fdO4f3W6/ilr/0tYmJixODBg4W9vb3Q19cX+fPnF3PnzlWZni2EEPHx8WLAgAEiW7ZswtTUVDRp0kQ8ffo0zfRkIb7v84HTk389hRDp1IenRQqFAjt37kTz5s0BAFu2bEGHDh1w+/btNIO+smTJAjs7OyQlJf3nVLps2bJJg+xy586NOnXqYNWqVdJyb29vTJs2TaWLnrTj2rVrKFWqFP766y906NBB2+X8sNDQUOTJkwdz585VOfIvEVFmkSl2/ZQqVQopKSkIDw+X5vd/ycDAAIUKFfru+6xUqVKaAVn379//4cFw9OPi4+PTDIj08vKCjo4OqlatqqWqiIhIEzJMUImNjVXZ7/j48WNcu3YNVlZWKFCgADp06IDOnTtj/vz5KFWqFN68eYMjR47AxcXlhwawDR48GBUrVsSMGTPg5uaGCxcuwNfXF76+vpp8WvQd5syZg8uXL6NGjRrQ09PDgQMHcODAAfTq1Svdp8cSEVE60/a+J01J3Vf65SV1n2tSUpKYMGGCcHJyEvr6+iJHjhyiRYsW4saNGz/8mHv37hXFihWTxlB8zzgH0rxDhw6JSpUqCUtLS6Gvry/y5csnJk2aJJKTk7Vd2k/7fIwKEVFmlCHHqBAREVHGkGmOo0JERES/HwYVIiIikq3fejCtUqnEixcvkDVr1t/q8N1ERESZmRACMTExsLe3h47Ov/eZ/NZB5cWLF5zVQURE9Jt6+vQpHBwc/nWd3zqopJ6M6unTpzAzM9NyNURERPQ9oqOj4ejoqHJSyW/5rYNK6u4eMzMzBhUiIqLfzPcM2+BgWiIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki09bRdARETy5TRqv7ZLIC0LndVIq4/PHhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki3ZBJVZs2ZBoVBg0KBB2i6FiIiIZEIWQeXixYtYsWIFXFxctF0KERERyYjWg0psbCw6dOiAlStXwtLSUtvlEBERkYxoPah4enqiUaNGqF279n+um5iYiOjoaJULERERZVx62nzwzZs348qVK7h48eJ3rT9z5kxMnjw5nasiIiIiudBaj8rTp08xcOBA+Pv7w8jI6LtuM3r0aERFRUmXp0+fpnOVREREpE1a61G5fPkywsPDUbp0aaktJSUFJ06cwNKlS5GYmAhdXV2V2xgaGsLQ0PBXl0pERERaorWgUqtWLdy8eVOlrWvXrihUqBBGjhyZJqQQERFR5qO1oJI1a1YUK1ZMpc3U1BTZsmVL005ERESZk9Zn/RARERF9i1Zn/Xzp2LFj2i6BiIiIZIQ9KkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWz8UVB4+fIhx48ahXbt2CA8PBwAcOHAAt2/f1mhxRERElLmpHVSOHz+O4sWL4/z58wgICEBsbCwA4Pr165g4caLGCyQiIqLMS+2gMmrUKEybNg1BQUEwMDCQ2mvWrIlz585ptDgiIiLK3NQOKjdv3kSLFi3StNva2uLt27caKYqIiIgI+IGgYmFhgZcvX6Zpv3r1KnLmzKmRooiIiIiAHwgqbdu2xciRI/Hq1SsoFAoolUqcPn0aw4YNQ+fOndOjRiIiIsqk1A4qM2bMQKFCheDo6IjY2FgUKVIEVatWRcWKFTFu3Lj0qJGIiIgyKT11b2BgYICVK1di/PjxuHXrFmJjY1GqVCnkz58/PeojIiKiTEztoJIqV65cyJUrlyZrISIiIlKhdlARQmD79u0IDg5GeHg4lEqlyvKAgACNFUdERESZm9pBZdCgQVixYgVq1KiB7NmzQ6FQpEddREREROoHFT8/PwQEBKBhw4bpUQ8RERGRRO1ZP+bm5sibN2961EJERESkQu2gMmnSJEyePBnx8fHpUQ8RERGRRO1dP25ubti0aRNsbW3h5OQEfX19leVXrlzRWHFERESUuakdVNzd3XH58mV07NiRg2mJiIgoXakdVPbv34/AwEBUrlw5PeohIiIikqg9RsXR0RFmZmbpUQsRERGRCrWDyvz58zFixAiEhoamQzlERERE/6P2rp+OHTviw4cPyJcvH0xMTNIMpo2IiNBYcUSZndOo/dougbQsdFYjbZdApFVqBxUvL690KIOIiIgorR+a9UNERET0K3xXUImOjpYG0EZHR//ruhxoS0RERJryXUHF0tISL1++hK2tLSwsLL567BQhBBQKBVJSUjReJBEREWVO3xVUjh49CisrKwBAcHBwuhZERERElOq7gkq1atWQN29eXLx4EdWqVUvvmoiIiIgAqHEcldDQUO7WISIiol9K7QO+aZK3tzdcXFxgZmYGMzMzVKhQAQcOHNBmSURERCQjak1PDgwMhLm5+b+u07Rp0+++PwcHB8yaNQv58+eHEALr169Hs2bNcPXqVRQtWlSd0oiIiCgDUiuo/NcxVNSd9dOkSROV69OnT4e3tzfOnTvHoEJERETqBZVXr17B1tY2XQpJSUnBtm3bEBcXhwoVKnx1ncTERCQmJkrX/+uYLkRERPR7++4xKl87doom3Lx5E1myZIGhoSF69+6NnTt3okiRIl9dd+bMmTA3N5cujo6O6VITERERycN3BxUhRLoUULBgQVy7dg3nz59Hnz594O7ujn/++eer644ePRpRUVHS5enTp+lSExEREcnDd+/6cXd3h7GxscYLMDAwgLOzMwCgTJkyuHjxIhYtWoQVK1akWdfQ0BCGhoYar4GIiIjk6buDytq1a9OzDolSqVQZh0JERESZl9pnT9ak0aNHo0GDBsiVKxdiYmKwceNGHDt2DIGBgdosi4iIiGRCq0ElPDwcnTt3xsuXL2Fubg4XFxcEBgaiTp062iyLiIiIZEKrQWX16tXafHgiIiKSuR8+hH5ISAgCAwMRHx8PIP1mBREREVHmpXZQeffuHWrXro0CBQqgYcOGePnyJQCge/fuGDp0qMYLJCIiosxL7aAyePBg6Onp4cmTJzAxMZHa27Rpg4MHD2q0OCIiIsrc1B6jcujQIQQGBsLBwUGlPX/+/AgLC9NYYURERERq96jExcWp9KSkioiI4MHYiIiISKPUDipVqlTBhg0bpOsKhQJKpRJz5sxBjRo1NFocERERZW5q7/qZM2cOatWqhUuXLiEpKQkjRozA7du3ERERgdOnT6dHjURERJRJqd2jUqxYMdy/fx+VK1dGs2bNEBcXh5YtW+Lq1avIly9fetRIREREmdQPHfDN3NwcY8eO1XQtRERERCrU7lE5ePAgTp06JV1ftmwZSpYsifbt2+P9+/caLY6IiIgyN7WDyvDhwxEdHQ0AuHnzJoYMGYKGDRvi8ePHGDJkiMYLJCIiosxL7V0/jx8/RpEiRQAAO3bsQJMmTTBjxgxcuXIFDRs21HiBRERElHmp3aNiYGCADx8+AAAOHz6MunXrAgCsrKyknhYiIiIiTVC7R6Vy5coYMmQIKlWqhAsXLmDLli0AgPv376c5Wi0RERHRz1C7R2Xp0qXQ09PD9u3b4e3tjZw5cwIADhw4gPr162u8QCIiIsq81O5RyZUrF/bt25emfeHChRopiIiIiCjVDx1HRalUIiQkBOHh4VAqlSrLqlatqpHCiIiIiNQOKufOnUP79u0RFhYGIYTKMoVCgZSUFI0VR0RERJmb2kGld+/eKFu2LPbv348cOXJAoVCkR11ERERE6geVBw8eYPv27XB2dk6PeoiIiIgkas/6KV++PEJCQtKjFiIiIiIVaveo9O/fH0OHDsWrV69QvHhx6Ovrqyx3cXHRWHFERESUuakdVFq1agUA6Natm9SmUCgghOBgWiIiItKoHzrXDxEREdGvoHZQyZ07d3rUQURERJTGDx3w7eHDh/Dy8sKdO3cAAEWKFMHAgQORL18+jRZHREREmZvaQSUwMBBNmzZFyZIlUalSJQDA6dOnUbRoUezduxd16tTReJHa4jRqv7ZLIC0LndVI2yUQEWVqageVUaNGYfDgwZg1a1aa9pEjR2aooEJERETapfZxVO7cuYPu3bunae/WrRv++ecfjRRFREREBPxAULGxscG1a9fStF+7dg22traaqImIiIgIwA/s+unZsyd69eqFR48eoWLFigA+jVGZPXs2hgwZovECiYiIKPNSO6iMHz8eWbNmxfz58zF69GgAgL29PSZNmoQBAwZovEAiIiLKvNQOKgqFAoMHD8bgwYMRExMDAMiaNavGCyMiIiL6oeOoAEB4eDju3bsHAChUqBBsbGw0VhQRERER8AODaWNiYtCpUyfY29ujWrVqqFatGuzt7dGxY0dERUWlR41ERESUSakdVHr06IHz589j//79iIyMRGRkJPbt24dLly7Bw8MjPWokIiKiTErtXT/79u1DYGAgKleuLLXVq1cPK1euRP369TVaHBEREWVuaveoZMuWDebm5mnazc3NYWlpqZGiiIiIiIAfCCrjxo3DkCFD8OrVK6nt1atXGD58OMaPH6/R4oiIiChzU3vXj7e3N0JCQpArVy7kypULAPDkyRMYGhrizZs3WLFihbTulStXNFcpERERZTpqB5XmzZunQxlEREREaakdVCZOnJgedRARERGlofYYladPn+LZs2fS9QsXLmDQoEHw9fXVaGFEREREageV9u3bIzg4GMCnQbS1a9fGhQsXMHbsWEyZMkXjBRIREVHmpXZQuXXrFv744w8AwNatW1G8eHGcOXMG/v7+WLdunabrIyIiokxM7aCSnJwMQ0NDAMDhw4fRtGlTAJ/O9/Py5UvNVkdERESZmtpBpWjRovDx8cHJkycRFBQkHY32xYsXyJYtm8YLJCIiosxL7aAye/ZsrFixAtWrV0e7du1QokQJAMCePXukXUJEREREmqD29OTq1avj7du3iI6OVjlkfq9evWBiYqLR4oiIiChzU7tHBQCEELh8+TJWrFiBmJgYAICBgQGDChEREWmU2j0qYWFhqF+/Pp48eYLExETUqVMHWbNmxezZs5GYmAgfH5/0qJOIiIgyIbV7VAYOHIiyZcvi/fv3MDY2ltpbtGiBI0eOaLQ4IiIiytzU7lE5efIkzpw5AwMDA5V2JycnPH/+XGOFEREREando6JUKpGSkpKm/dmzZ8iaNatGiiIiIiICfiCo1K1bF15eXtJ1hUKB2NhYTJw4EQ0bNtRkbURERJTJqb3rZ/78+ahXrx6KFCmChIQEtG/fHg8ePIC1tTU2bdqUHjUSERFRJqV2UHFwcMD169exZcsWXL9+HbGxsejevTs6dOigMriWiIiI6GepHVQAQE9PDx06dECHDh2ktpcvX2L48OFYunSpxoojIiKizE2toHL79m0EBwfDwMAAbm5usLCwwNu3bzF9+nT4+Pggb9686VUnERERZULfPZh2z549KFWqFAYMGIDevXujbNmyCA4ORuHChXHnzh3s3LkTt2/fTs9aiYiIKJP57qAybdo0eHp6Ijo6GgsWLMCjR48wYMAA/P333zh48KB0FmUiIiIiTfnuoHLv3j14enoiS5Ys6N+/P3R0dLBw4UKUK1cuPesjIiKiTOy7g0pMTAzMzMwAALq6ujA2NuaYFCIiIkpXag2mDQwMhLm5OYBPR6g9cuQIbt26pbJO06ZNNVcdERERZWpqBRV3d3eV6x4eHirXFQrFVw+vT0RERPQjvjuoKJXK9KyDiIiIKA21z/VDRERE9KtoNajMnDkT5cqVQ9asWWFra4vmzZvj3r172iyJiIiIZESrQeX48ePw9PTEuXPnEBQUhOTkZNStWxdxcXHaLIuIiIhk4ofO9aMpBw8eVLm+bt062Nra4vLly6hataqWqiIiIiK50GpQ+VJUVBQAwMrK6qvLExMTkZiYKF2Pjo7+JXURERGRdvzQrp/IyEisWrUKo0ePRkREBADgypUreP78+Q8XolQqMWjQIFSqVAnFihX76jozZ86Eubm5dHF0dPzhxyMiIiL5Uzuo3LhxAwUKFMDs2bMxb948REZGAgACAgIwevToHy7E09MTt27dwubNm7+5zujRoxEVFSVdnj59+sOPR0RERPKndlAZMmQIunTpggcPHsDIyEhqb9iwIU6cOPFDRfTr1w/79u1DcHAwHBwcvrmeoaEhzMzMVC5ERESUcak9RuXixYtYsWJFmvacOXPi1atXat2XEAL9+/fHzp07cezYMeTJk0fdcoiIiCgDUzuoGBoafnUQ6/3792FjY6PWfXl6emLjxo3YvXs3smbNKgUdc3NzGBsbq1saERERZTBq7/pp2rQppkyZguTkZACfzu/z5MkTjBw5Eq1atVLrvry9vREVFYXq1asjR44c0mXLli3qlkVEREQZkNpBZf78+YiNjYWtrS3i4+NRrVo1ODs7I2vWrJg+fbpa9yWE+OqlS5cu6pZFREREGZDau37Mzc0RFBSEU6dO4caNG4iNjUXp0qVRu3bt9KiPiIiIMrEfPuBb5cqVUblyZU3WQkRERKRC7aCyePHir7YrFAoYGRnB2dkZVatWha6u7k8XR0RERJmb2kFl4cKFePPmDT58+ABLS0sAwPv372FiYoIsWbIgPDwcefPmRXBwMI8cS0RERD9F7cG0M2bMQLly5fDgwQO8e/cO7969w/3791G+fHksWrQIT548gZ2dHQYPHpwe9RIREVEmonaPyrhx47Bjxw7ky5dPanN2dsa8efPQqlUrPHr0CHPmzFF7qjIRERHRl9TuUXn58iU+fvyYpv3jx4/SAdvs7e0RExPz89URERFRpqZ2UKlRowY8PDxw9epVqe3q1avo06cPatasCQC4efMmD4dPREREP03toLJ69WpYWVmhTJkyMDQ0hKGhIcqWLQsrKyusXr0aAJAlSxbMnz9f48USERFR5qL2GBU7OzsEBQXh7t27uH//PgCgYMGCKFiwoLROjRo1NFchERERZVo/fMC3QoUKoVChQpqshYiIiEjFDwWVZ8+eYc+ePXjy5AmSkpJUli1YsEAjhRERERGpHVSOHDmCpk2bIm/evLh79y6KFSuG0NBQCCFQunTp9KiRiIiIMim1B9OOHj0aw4YNw82bN2FkZIQdO3bg6dOnqFatGv7888/0qJGIiIgyKbWDyp07d9C5c2cAgJ6eHuLj45ElSxZMmTIFs2fP1niBRERElHmpHVRMTU2lcSk5cuTAw4cPpWVv377VXGVERESU6ak9RsXV1RWnTp1C4cKF0bBhQwwdOhQ3b95EQEAAXF1d06NGIiIiyqTUDioLFixAbGwsAGDy5MmIjY3Fli1bkD9/fs74ISIiIo1SK6ikpKTg2bNncHFxAfBpN5CPj0+6FEZERESk1hgVXV1d1K1bF+/fv0+veoiIiIgkag+mLVasGB49epQetRARERGpUDuoTJs2DcOGDcO+ffvw8uVLREdHq1yIiIiINEXtwbQNGzYEADRt2hQKhUJqF0JAoVAgJSVFc9URERFRpqZ2UAkODk6POoiIiIjSUDuoVKtWLT3qICIiIkpD7TEqAHDy5El07NgRFStWxPPnzwEAfn5+OHXqlEaLIyIiosxN7aCyY8cO1KtXD8bGxrhy5QoSExMBAFFRUZgxY4bGCyQiIqLM64dm/fj4+GDlypXQ19eX2itVqoQrV65otDgiIiLK3NQOKvfu3UPVqlXTtJubmyMyMlITNREREREB+IGgYmdnh5CQkDTtp06dQt68eTVSFBERERHwA0GlZ8+eGDhwIM6fPw+FQoEXL17A398fw4YNQ58+fdKjRiIiIsqk1J6ePGrUKCiVStSqVQsfPnxA1apVYWhoiGHDhqF///7pUSMRERFlUmoHFYVCgbFjx2L48OEICQlBbGwsihQpgixZsqRHfURERJSJqb3r56+//sKHDx9gYGCAIkWK4I8//mBIISIionShdlAZPHgwbG1t0b59e/z99988tw8RERGlG7WDysuXL7F582YoFAq4ubkhR44c8PT0xJkzZ9KjPiIiIsrE1A4qenp6aNy4Mfz9/REeHo6FCxciNDQUNWrUQL58+dKjRiIiIsqk1B5M+zkTExPUq1cP79+/R1hYGO7cuaOpuoiIiIh+7KSEHz58gL+/Pxo2bIicOXPCy8sLLVq0wO3btzVdHxEREWViaveotG3bFvv27YOJiQnc3Nwwfvx4VKhQIT1qIyIiokxO7aCiq6uLrVu3ol69etDV1VVZduvWLRQrVkxjxREREVHmpnZQ8ff3V7keExODTZs2YdWqVbh8+TKnKxMREZHG/NAYFQA4ceIE3N3dkSNHDsybNw81a9bEuXPnNFkbERERZXJq9ai8evUK69atw+rVqxEdHQ03NzckJiZi165dKFKkSHrVSERERJnUd/eoNGnSBAULFsSNGzfg5eWFFy9eYMmSJelZGxEREWVy392jcuDAAQwYMAB9+vRB/vz507MmIiIiIgBq9KicOnUKMTExKFOmDMqXL4+lS5fi7du36VkbERERZXLfHVRcXV2xcuVKvHz5Eh4eHti8eTPs7e2hVCoRFBSEmJiY9KyTiIiIMiG1Z/2YmpqiW7duOHXqFG7evImhQ4di1qxZsLW1RdOmTdOjRiIiIsqkfnh6MgAULFgQc+bMwbNnz7Bp0yZN1UREREQE4CeDSipdXV00b94ce/bs0cTdEREREQHQUFAhIiIiSg8MKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbDCpEREQkWwwqREREJFsMKkRERCRbWg0qJ06cQJMmTWBvbw+FQoFdu3ZpsxwiIiKSGa0Glbi4OJQoUQLLli3TZhlEREQkU3rafPAGDRqgQYMG2iyBiIiIZEyrQUVdiYmJSExMlK5HR0drsRoiIiJKb7/VYNqZM2fC3Nxcujg6Omq7JCIiIkpHv1VQGT16NKKioqTL06dPtV0SERERpaPfatePoaEhDA0NtV0GERER/SK/VY8KERERZS5a7VGJjY1FSEiIdP3x48e4du0arKyskCtXLi1WRkRERHKg1aBy6dIl1KhRQ7o+ZMgQAIC7uzvWrVunpaqIiIhILrQaVKpXrw4hhDZLICIiIhnjGBUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSIiIpItBhUiIiKSLVkElWXLlsHJyQlGRkYoX748Lly4oO2SiIiISAa0HlS2bNmCIUOGYOLEibhy5QpKlCiBevXqITw8XNulERERkZZpPagsWLAAPXv2RNeuXVGkSBH4+PjAxMQEa9as0XZpREREpGVaDSpJSUm4fPkyateuLbXp6Oigdu3aOHv2rBYrIyIiIjnQ0+aDv337FikpKciePbtKe/bs2XH37t006ycmJiIxMVG6HhUVBQCIjo5Ol/qUiR/S5X7p95Fe29b34jZI3AZJ29JjG0y9TyHEf66r1aCirpkzZ2Ly5Mlp2h0dHbVQDWUG5l7aroAyO26DpG3puQ3GxMTA3Nz8X9fRalCxtraGrq4uXr9+rdL++vVr2NnZpVl/9OjRGDJkiHRdqVQiIiIC2bJlg0KhSPd6M5Po6Gg4Ojri6dOnMDMz03Y5lAlxGyRt4zaYfoQQiImJgb29/X+uq9WgYmBggDJlyuDIkSNo3rw5gE/h48iRI+jXr1+a9Q0NDWFoaKjSZmFh8QsqzbzMzMz4BiWt4jZI2sZtMH38V09KKq3v+hkyZAjc3d1RtmxZ/PHHH/Dy8kJcXBy6du2q7dKIiIhIy7QeVNq0aYM3b95gwoQJePXqFUqWLImDBw+mGWBLREREmY/WgwoA9OvX76u7ekh7DA0NMXHixDS72oh+FW6DpG3cBuVBIb5nbhARERGRFmj9yLRERERE38KgQkRERLLFoEJERESyxaBCREREssWgQkRERLLFoEJERESyxaBCREREssWgQkRERLLFoEJERESyxaBCvyWlUqntEoiI6BdgUKHfko7Op0337du3AACeCYJ+tS/DMrdB0oYvt8OM+COOQYV+W4sWLULz5s3x8OFDKBQKbZdDmYyOjg6ioqIQGBgIANwGSSt0dHQQGRmJuXPn4v3799KPuIwk4z0jyrC+/MWqr68PY2NjGBgYaKkiysyUSiXmz58PDw8P7Nu3T9vlUCZ26NAhLFiwAEuXLtV2KemCZ0+m3050dDTMzMwAAFFRUTA3N9dyRZRZKJVKlV+sd+7cwerVqzF79mzo6upqsTLKTFJSUlS2t+TkZGzZsgXt2rXLkNshgwr9VgYPHoyUlBSMHj0aOXLk0HY5lAlFRkYiMjISjo6OKl8KX355EP2ML0Pxl969e4fTp0+jYsWKsLa2ltoz4nbIXT8ka1/maAcHB2zYsCHDvRHp9yCEwKhRo1C+fHmEhoaqLOM2ST/j5cuXePHiBd68eQPg09iTf+tH2Lp1K5o3b47jx4+rtGfE7ZA9KiQbqb8EhBBQKBTf/EXx/v17WFpaaqFCymj+61fr19YJCwvDuHHjsG7dugz5pUC/3tq1a7Fs2TI8ffoU+fLlQ+XKlTFnzhyVdb7WU+Ll5YV+/fpBT0/vV5b7yzGokFakhhHg0xtQCAE9PT08f/4cO3fuRNeuXWFqagrg0+4eS0tLTJgwIc1tiX7U5wHk6NGjePLkCZydnZE3b17Y29urrBMVFQWlUpkmIGfEbnb6tfbt2wc3NzcsX74cJiYmePToEebMmYOKFSti/fr1yJYtm/SZ9/btW4SEhMDV1VXlPj5+/Jihwwp3/dAvkZqHo6OjER8fD4VCgUOHDiEkJAS6urrQ09NDWFgYSpUqhRcvXkghJS4uDvr6+li4cCEiIiIYUkgjhBBSSBk1ahS6dOmCefPmoVevXhg2bBguXrwI4FP3e2JiIiZMmIDSpUvj3bt3KvfDkEI/6+LFi2jUqBG6dOkCNzc3jBgxAoGBgbhx4wY6dOgA4NPU9+TkZPj5+aFixYo4deqUyn1k5JACMKjQL/Tq1SsUL14cx48fx8aNG1G/fn38888/AD7tzilatChatGiB6dOnS7cxNTXFiBEj8ODBA1hZWTGkkEakbkfz5s3DX3/9hU2bNuHWrVto2bIl9u7di3HjxuHs2bMAAAMDA5QqVQq1atWChYWFFqumjOjx48d4+fKlSlu5cuWwZ88eXL58GT179gTw6XAMjRs3xvTp09P0qGR4gugX6tq1qzAzMxM6Ojpi5cqVUntSUpLYsmWLSElJkdqUSqU2SqRM4vXr16Jly5ZizZo1Qggh9uzZI8zMzETv3r1FqVKlRK1atcS5c+eEEKrb4sePH7VSL2VMgYGBInv27GLz5s1SW+r25u/vL5ydncXFixfT3C45OfmX1aht7FGhXyL1sM6enp6IiYmBgYEB7OzskJCQAODTrwU3NzeVQYvsPaH0ZGtrixEjRqB+/fq4evUqPD09MW3aNHh7e6NVq1Y4d+4cPD09cfnyZZVtkbt7SJMKFy6M6tWrw8/PD0eOHAHwv8++kiVLIjw8XDpVyOcy+u6ezzGo0C+RGkAcHR1x6tQpuLu7o23btti9ezfi4+PTrJ8Rz1dB2vOt7alUqVLIkSMHDhw4ABcXF/Tq1QsAYGVlBVdXVzRp0gSlSpX6laVSJuPo6IjevXsjMjISCxcuxJ49e6RlOXLkQJ48ebRYnTxknkhGWiH+f/Dry5cvkZycjFy5csHW1hYVK1ZEQkICunfvjnXr1qFx48YwMjKCj48PateuDWdnZ22XThmE+Gzg7KpVqxAeHg4DAwMMGzZMOv1CYmIinj9/jtDQUBQsWBCHDh1C06ZN0b9//3+dKk/0M1JnjVWvXh3Lly/HmDFjMHLkSAQGBsLFxQVbt26FQqFAnTp1tF2qVnF6MqW7gIAATJo0Ca9fv0ajRo3QokULNGnSBADQtWtX7Ny5E0OHDsXr16/h7e2NmzdvokiRIlqumjKaiRMnwsvLC+XKlcOFCxdQvnx5+Pn5wc7ODnv37sW0adPw/v176OvrQwiBGzduQE9PjzPNKF2kblcBAQFYvnw5Dh06hLt37yI4OBhLly6Fo6MjLCws4O/vD319/Uw9FZ5BhdLV7du3Ua9ePQwePBgmJibYtGkTDA0N4e7ujo4dOwIABg4ciCtXriAxMRG+vr4oWbKkdoumDOHzXpCPHz/C3d0d/fv3R6lSpRAaGopGjRrBzs4OO3fuhI2NDfbv34+QkBDExsZi5MiR0NPTy9RfDqQZqYFEfHHsKF1dXQQEBKBz585YsGCBtNsR+LS96ujoqGy/mWlMypcYVCjd3L17F9u2bUN8fDxmzJgBALh58yYmTJiA6OhodO3aVQorr169gqmpKbJmzarNkimD+Dyk3LlzB9HR0VixYgUmTJgAJycnAJ+mhdapUwfZs2fHrl27YGNjo3IfDCn0sz7fDt++fQuFQoFs2bIB+PSZV7p0aUyYMAG9e/eWbvNlDx579BhUKB0IIfD+/Xs0btwY//zzD5o0aQI/Pz9p+Y0bNzBhwgTEx8ejbdu26Nq1qxarpYxs+PDhUtf569evERAQgAYNGkgf/I8fP0aDBg0ghMDp06dVTu5G9DM+DxhTp07Frl27EB0dDWtra0yfPh01a9bE8+fPkTNnTi1XKn8cHUYap1AoYGVlhZkzZ6Jo0aK4cuUKgoKCpOUuLi6YOnUqkpOTpTcvkSZ8Prtn3759OHjwIBYvXozly5cjT548GDt2LK5fvy4dKTlPnjzYt28fSpYsyfNHkUalhpQpU6Zg0aJF0vR3a2trdOjQAevXr0/Ti0dfxx4V0ohvdU8eP34cY8aMgZ2dHTw9PVGzZk1p2e3bt2Fubg4HB4dfWSplAgEBAThz5gyyZcuG0aNHAwBiY2NRunRpmJmZYdWqVShRokSabZa7e0iT3r17h7p168LT0xPdunWT2nv16oW9e/ciODgYhQoV4u6d/8AeFfppqW+yM2fOYMGCBRg/fjxOnz6N5ORkVKtWDVOmTMGrV6+wdOlSHDt2TLpd0aJFGVJI4+Lj4zF+/HgsWLAAt2/fltqzZMmCK1euICYmBh4eHtL5fD7HkEKa9PHjR7x9+1bqrUs9wKWvry/s7e2xcOFCADy45X9hUKGf8vkUuwYNGuD06dPYs2cPxowZg+nTpyMpKQm1atXClClT8O7dO0ydOhUnT57UdtmUgRkbG+PkyZOoXbs2Ll++jD179iAlJQXA/8LK3bt3sWLFCi1XShnJ13ZOZM+eHXZ2dlizZg0AwMjICElJSQAAZ2dnBpTvxKBCPyW1J2XAgAFYsGABduzYgW3btuHy5cvYsmULxo0bJ4WVUaNGQV9fn0daJI35fEyKEEL6srCyssLGjRthaWmJuXPnIjAwUFpmamqKV69ewdfXVys1U8ajVCql0PHixQuEh4fjw4cPAIBJkybh7t270sye1IMMPnv2jCe5/E4co0I/JPWNqVAosHz5cly7dg2+vr54/PgxateujcqVK8PMzAzbtm2Dh4cHxowZA0NDQ3z48AEmJibaLp8ygM+nfi5ZsgTXr1/Ho0ePMGjQIJQuXRoODg548+YNmjVrBl1dXYwZMwb16tVTOcIsx6TQz/D394erqyvy5csHABg9ejQCAwMRFhaG2rVro2nTpujQoQNWrlyJqVOnIlu2bChWrBgePnyIyMhI6aCC9O8YVOi7pH4pfB40rl27hpIlSyI6OhpPnz6Fs7Mz6tevjzx58mDNmjWIioqSjjDbpUsXTJ8+nYPG6Kd9uQ2NHj0aq1evRq9evfDs2TOcPXsWzZo1Q69eveDs7Iw3b96gZcuWePPmDdatWwdXV1ctVk8ZxYEDB9C4cWOMHDkSgwYNwoEDBzBixAh4eXnh3bt3uHLlCgIDAzF+/Hj07t0bN2/ehJeXF3R0dGBpaYkZM2bwoILfK13PzUwZyqNHj0S7du3EP//8I7Zu3SoUCoW4cOGCdErymzdvikKFConz588LIYR4+PChaNy4sRgzZox48uSJNkunDCYlJUUIIYSfn5/IkyePuHz5shBCiJMnTwqFQiHy588vBg4cKB49eiSEEOLly5eiV69e4uPHj1qrmTKepUuXCgcHBzF16lTRr18/sXLlSmnZ06dPxZQpU4STk5M4ePDgV2+fnJz8q0r9rbHPib5bQkICTp48iS5duuDatWtYu3YtypUrJ+0GEkLg48ePOHv2LIoWLYoNGzYAAIYNG8ZjVNBP69SpE2xsbLBgwQLo6OggOTkZBgYG6N27N0qXLo1du3aha9euWLVqFV69eoVp06ZBR0cHPXv2ROHChaXBs/wFSz8rKSkJBgYG8PT0hImJCUaPHo2YmBhMmzZNWsfBwQGdO3fGoUOHcOnSJdSrVy/NyS252+c7aTsp0e8h9Resj4+P0NHRESVKlBBXr15VWScqKkp06dJF5MuXTzg5OQkbGxvply7Rz4iKihKTJ08WVlZWYtKkSVL78+fPxevXr8XLly9F2bJlxfz586X17e3tRY4cOcSiRYuEEELq+SPSlJkzZ4rw8HDh7+8vTExMRMOGDcX9+/dV1mnTpo1o2bKllirMGDjrh/6TEAI6OjoQQsDe3h7z58/Hx48fMW7cOJw6dUpaz8zMDPPmzcPy5csxceJEnD9/HqVLl9Zi5ZQRxMTEwMzMDH369MG4cePg5eWFiRMnAgDs7e1ha2uLly9f4v3799L4k+fPn6Nu3bqYMGECPD09AfBYFfTzxGdDOtevX4+pU6fiwYMHaN++PRYuXIgrV67Ax8cH9+7dAwBER0fj8ePHyJUrl7ZKzhDY70T/Svz/wMWjR4/i+PHjGDRoEJo0aYLatWvDzc0Ns2bNwpgxY1CxYkUAn046WLduXS1XTRnFiBEjsGLFCjx8+BA2Njbo2LEjhBCYOnUqAGDy5MkAPoUZXV1dnD59GkIIzJo1CyYmJtKUUO7uIU1IDbtHjhzB1atX4evrK3329erVC8nJyZg8eTIOHjyI0qVLIy4uDklJSZgzZ442y/79abM7h+Qttat8+/btwtzcXIwePVpcvHhRWn7jxg1RpEgR0bhxY/HXX3+JSZMmCYVCIZ4+fcpudtKI69evi6pVq4qCBQuKN2/eCCGECA8PF/PnzxcWFhZiwoQJ0rr9+vUT+fLlEw4ODsLV1VUkJSUJIbjLhzTr2LFjonjx4iJbtmxi165dQgghEhMTpeWrV68WWbJkEaVLlxYbNmyQBnBz4OyP4/Rk+lcXLlxA/fr1MXv2bPTs2VNqj46OhpmZGe7cuYOePXsiPj4eUVFR2Lp1K3f3kEacPXsWb968QZEiRdCmTRvExsZKZzh+8+YN/Pz8MHXqVOlkb8CnKfMKhQLFixeHjo4OPn78yAGL9FPEF9PhY2NjMXfuXPj6+qJ8+fLYtGkTjI2NkZycDH19fQDAggULcObMGWzbtg0KhYI9ej+JQYX+1dKlS7Fz504cOXIEUVFROHr0KP766y/cuXMHw4YNQ7du3RAeHo6oqCiYm5vD1tZW2yVTBtG5c2e8ePEChw8fRmhoKFq3bo2YmJg0YWXatGno168fpkyZonJ7fjmQJi1btgwODg5o1qwZ4uPjMW/ePOzcuRPVq1fHjBkzYGRkpBJWUgPOl0GH1MfBtPSv7OzscPnyZcycOROtW7fG2rVrYWRkhEaNGqFHjx64f/8+bG1tkT9/foYU0qhly5bh2bNnWLp0KZycnLBp0yaYm5ujUqVKePv2LWxsbNCpUydMmDAB06ZNw+rVq1Vuz5BCmvLmzRscPXoUffv2xcGDB2FsbIwhQ4agcePGOHPmDMaOHYuEhATo6+vj48ePAMCQokHsUSFJ6psqNjYWWbJkAQC8fv0aS5YswdatW1GzZk106dIFf/zxB16/fo2mTZti3bp1KFq0qJYrp4wmtTdk8eLFuHr1KhYsWABLS0vcvXsXnTt3RlRUlNSz8urVKxw/fhytWrXibh7SiC+PdwIA169fx+LFi3H48GH4+PigQYMGiIuLw5w5c3D48GEULlwYy5cvl87lQ5rDHhWSKBQK7N+/H+3atUP16tWxbt066OnpYdq0aTh//jx8fHzg6uoKHR0dLFmyBHFxcexFoXSR2htSvXp1nDhxAvv37wcAFCxYEH5+frC0tETVqlXx+vVr2NnZoU2bNtDT05N+zRL9jNSQ8urVK6mtRIkSGDhwIGrUqIHevXvj4MGDMDU1xYgRI/DHH39AR0dH2u1DGqalQbwkQ6dPnxZGRkZi+PDhon79+sLFxUV4eHiIkJAQaZ3g4GDRq1cvYWVlleaAb0Q/KvWAgl/j4+MjChQoIO7duye13bt3Tzg5OYm2bdv+ivIok/h8O9y8ebPImzevykxHIYS4du2aaNasmciVK5c4duyYEEKI+Ph4aXbZv23L9GPYo0IAgLCwMAQFBWH69OmYM2cODhw4gF69euHGjRuYOXMmHj16hLi4OJw9exbh4eE4fvw4SpYsqe2yKQP4vJv9woULOHPmDI4fPy4tb9q0KcqXL4/g4GCprUCBAjhx4gT++uuvX14vZUyJiYnSdpiUlIR8+fKhUKFC8PT0xOXLl6X1SpQogebNm+Pp06eoW7cuzpw5AyMjI2lMype7jOjn8S+aCS1duhR///23dP3evXto06YN1qxZAyMjI6nd09MTHTp0wO3btzFnzhxERkZi+PDhWL9+PYoVK6aN0imD+fyDfcyYMejSpQu6desGd3d3tGnTBtHR0ciRI4e0/z85OVm6raOjI3R1dZGSkqKt8imDOHDgAPz8/AAAPXv2RM2aNVG2bFkMHToUdnZ28PDwwKVLl6T1c+XKhbZt22L+/PkoX7681M6Bs+lE21069Gs9fvxYtG/fXjx48EClfdSoUcLW1la0bNlSOrBWKm9vb1GwYEExYMAAHrSI0sW8efNEtmzZxPnz50VKSoqYMWOGUCgU4tSpU9I6lSpVEh4eHlqskjKqdu3aCScnJ1GvXj1hbW0trl+/Li07evSoaN68uShWrJg4cOCAePz4sWjevLkYOnSotA7Pyp2+GFQyobi4OCGEEOfOnRPbt2+X2idMmCCKFy8uxo0bJ16/fq1ym5UrV4rHjx//yjIpk1AqlcLd3V34+voKIYTYsWOHsLCwED4+PkIIIWJiYoQQQhw4cEA0bdpU3LhxQ2u1UsZVsmRJoVAoVE56merkyZOiU6dOQqFQiAIFCggXFxfpRxuPfJz+OJcvEzI2NkZkZCRmzpyJ58+fQ1dXF82bN8fkyZORnJyM/fv3QwiBgQMHwsbGBgDQo0cPLVdNGVVCQgLOnz+P6tWr49ixY3B3d8fcuXPh4eGBjx8/Ys6cOahQoQJcXV0xZcoUXLhwAcWLF9d22ZRBJCUlISEhAc7OzsiVKxe2bNmCnDlzom3bttJhGipXrozy5cujZ8+eSE5ORrVq1aCrq8sjH/8iHKOSCSkUClhYWGDo0KHIkycPvLy8EBAQAACYMWMG6tevj6CgIMyYMQNv377VcrWUkdy4cQPPnj0DAAwePBjHjx+HsbEx2rdvj7/++gsNGzbEwoULpZMJvn//HpcuXcK9e/dgaWkJPz8/5M6dW5tPgTIYAwMDmJmZYdu2bdi9ezfKlSuHOXPmYPPmzYiJiZHWS0hIQJUqVVCzZk1pbBRDyq/BoJIJiU+7/FClShUMHjwYlpaWWLx4sUpYcXV1xdWrV1VOa070o4QQuH//PmrUqIE1a9agd+/eWLRoESwtLQEArq6uCAsLQ/ny5VGhQgUAwIsXL9ClSxdERkaiX79+AIB8+fKhdu3aWnselPEIIaBUKqXr69evR8WKFbFw4UJs2LABT548Qc2aNfHnn39K6wM88vGvxCPTZkKpR/2MioqCiYkJbty4genTp+P9+/cYOHAgmjdvDuDTYaNTd/0QacLKlSsxYsQIJCQkYPfu3ahbt650ROQtW7ZgypQpEEJAT08PxsbGUCqVOHPmDPT19XnuHvppERERsLKyUmlL3f62bduGoKAg+Pr6AgB69eqFY8eOISUlBVZWVjh9+jSPOqsl7FHJZD5+/AhdXV2EhoaievXqOHToEMqUKYNhw4bBxsYGkydPxr59+wCAIYU0JvUXq6OjIwwNDWFmZoZz584hNDRUmtLZpk0bbNiwAVOmTIGbmxtGjhyJc+fOSedPYUihn7Fo0SKUK1dOZXcOACmkdOnSBSVKlJDafX19sWLFCixZsgTnzp2DgYEBj3ysLdoZw0u/wrdGo4eEhIjs2bOLHj16qEyrO3bsmOjUqZMIDQ39VSVSBvflNpiUlCTi4+OFt7e3yJkzpxgzZsx/bm+c+kk/a8WKFcLQ0FBs3LgxzbInT56I4sWLi6VLl0ptX9vmuB1qD3f9ZFDi/7szz549izt37iAkJASdO3dGjhw5sH79ely6dAnr169Pc4bPhIQElYO+Ef2oz484GxERgZiYGJWBsF5eXpg3bx66d++Orl27wsnJCU2aNMHYsWPh6uqqrbIpg1m5ciX69+8PPz8//Pnnn4iMjERcXBwSEhJga2uLrFmz4sGDB8ifP7+2S6VvYFDJwHbs2IFevXpJJ2978+YN2rRpg5EjRyJr1qzaLo8ysM9DypQpU3Do0CHcunULbm5uaNGiBRo0aADgU1jx8vJCsWLF8O7dOzx58gShoaE8uRtpxKNHj+Ds7Aw3Nzds3rwZt27dQt++ffHmzRuEhYWhRo0a6NOnDxo3bqztUulfcG5VBnXr1i0MHjwY8+fPR5cuXRAdHQ0LCwsYGxszpFC6Sw0pEyZMgK+vL+bOnQsnJyf07t0bDx48QGRkJNq1a4dBgwbB2toa169fR0JCAk6ePCmdBZlTP+ln2djYYPbs2ZgwYQKGDRuGQ4cOoUqVKmjWrBmio6Oxfft2jBs3DtbW1uzFkzNt7ncizTh69Kh4+PBhmrYKFSoIIYS4c+eOyJ07t+jRo4e0/OHDh9znSunq6NGjomjRouLEiRNCCCHOnDkjDAwMRJEiRUT58uXFtm3bpHU/PzUDT9NAmpSQkCDmzZsndHR0RLdu3URSUpK07NKlS6JgwYJi2bJlWqyQ/gtn/fzGhBC4evUqGjRoAG9vb4SFhUnLnj9/DiEEYmNjUb9+fdStWxcrVqwAAAQFBcHb2xvv37/XVumUAYkv9iLnzJkTffr0QZUqVXDo0CE0btwYvr6+CAoKwsOHD7F48WKsXr0aAFR6T9iTQppkaGiI3r17Y8eOHejRowf09fWlbbVMmTIwMjLC06dPtVwl/RsGld+YQqFAqVKlMH/+fGzduhXe3t549OgRAKBRo0Z4/fo1zMzM0KhRI/j6+krd8YGBgbhx4wane5LGKJVKaUD2o0ePEBcXh/z586Ndu3ZISEjAokWLMGDAAHTq1An29vYoWrQoQkJCcOfOHS1XTpmBqakpGjRoIB1MMHVbDQ8Ph7GxMYoWLarN8ug/8KfLbyx1P76npycAYO7cudDV1UWPHj2QJ08ejB8/HjNmzMDHjx/x4cMHhISEYNOmTVi1ahVOnTolHRWU6Gd8PnB2woQJOHv2LIYPH44aNWrAysoKcXFxePnyJUxMTKCjo4PExEQ4OTlhxIgRqF+/vparp4xIfDaTMZWhoaH0/5SUFLx9+xY9e/aEQqFAu3btfnWJpAYGld9Yao/IoUOHoKOjg+TkZHh5eSEhIQEjR46Em5sb4uPjMWPGDGzfvh3Zs2eHgYEBgoODUaxYMS1XTxnF5yFlxYoV8PX1RalSpaSZO4mJibCyssKpU6ekAbPv3r3DmjVroKOjoxJ0iH5EWFgYIiIikC1bNtjZ2f3rEWSTk5Ph5+eHTZs2ISIiAufOnZPO3cNeZnni9OTfXGBgoHQiN1NTUzx48ACLFy9G3759MXLkSNjY2CAmJgbHjx+Hk5MTbG1tYWtrq+2y6Tf3Zbi4f/8+mjdvjtmzZ6NJkyZp1rt48SLGjRuH2NhYWFlZISAgAPr6+gwp9NM2bNiA+fPnIzw8HNbW1ujfv7/UU5Lqy+0sKCgIt2/fRr9+/TjL7DfAoPIbUyqV6NChAxQKBTZu3Ci1L1myBCNGjICnpyf69u2LvHnzarFKymhatmyJMWPGoGzZslLbtWvXUL9+fRw/fhwFCxb86kEEExISIISAkZERFAoFvxzop23YsAGenp7S4fFnzJiBR48e4fTp09K2lRpSIiMjcejQIbi5uancB3tS5I8/ZX5jqb8QUrvYk5KSAAD9+/eHh4cH1q5di8WLF6vMBiL6Webm5nBxcVFpMzIywvv373Hr1i2pLfX8PmfPnsWOHTugo6MDY2NjKBQKKJVKhhT6KZcuXcLUqVOxdOlSdOvWDcWLF8fgwYPh7OyMM2fO4Pbt24iOjpZ2i69fvx59+/bFX3/9pXI/DCnyx6DyG3rx4oX0/4IFC2Lv3r0IDw+HgYEBkpOTAQAODg4wMTFBcHAwjI2NtVUqZSDPnz8HAKxduxYGBgZYvHgxDh06hKSkJDg7O6NNmzaYO3cuDh8+DIVCAR0dHaSkpGD69OkIDg5WGTfA3T30sxITEzFo0CA0atRIaps0aRKOHDmCdu3aoXPnzmjbti0iIiKgr6+Phg0bYtiwYRw4+xvirp/fzPXr19GvXz+0b98effr0QVJSEmrWrIm3b9/i2LFjsLOzAwCMHDkSRYsWRePGjdOc1pxIXT179gQAjB49WtqV6OLigrdv32Lz5s2oWrUqTp48iYULF+LmzZvo0KEDDAwMcOTIEbx58wZXrlxhDwpplFKpxJs3b5A9e3YAQOfOnXH48GHs2bMHjo6OOH78OKZNm4aRI0eiffv2KmNWuLvn98KfNb8ZExMTWFhYYPv27Vi3bh0MDAywYsUK2NjYoHDhwmjevDnq1q2LRYsWoWzZsgwppBEuLi44ePAgvL29ERISAgC4ceMGChYsiA4dOuDEiROoUqUKpkyZgs6dO8PPzw9Hjx5Frly5cPnyZWnAIpGm6OjoSCEFAIYNG4bz58+jbNmyyJ49Oxo0aICIiAi8fv06zVRlhpTfC3tUfkMhISEYM2YMXr16hZ49e6JTp05ISUnBvHnzEBYWBiEE+vfvjyJFimi7VMpA1qxZgwkTJqBt27bo2bMnChYsCACoWrUqHj9+DH9/f1StWhUA8OHDB5iYmEi35cBZ+tWePXuGjh07YtiwYTzp4G+OQeU3cOXKFbx8+VJlX2xISAjGjRuH0NBQ9O/fHx06dNBihZSRfT61c/Xq1ZgwYQLatWuXJqyEhYVhw4YNqFChgsp4lK8dfItIHZ9vQ6n/T/33zZs3sLGxUVk/Li4O7dq1Q1RUFI4ePcoelN8cg4rMxcTEoFGjRtDV1cWIESPQoEEDaVloaCjq168PExMT9OjRA3379tVipZTRfOsYJytXrsTkyZPRpk0b9OrVSworNWvWxOnTp3Hu3DmUKlXqV5dLGdTXtsPUtoCAAGzatAmLFi2Cvb094uPjsXv3bvj5+eH58+e4ePEi9PX1OSblN8cxKjKVmh+zZs2KOXPmQE9PD0uXLsX+/fuldZycnFCjRg28evUKR44cQWRkpJaqpYzm8y+HM2fOIDg4GNevXwfwaWDt+PHjsXnzZvj6+uLevXsAgKNHj6JHjx5ppi4T/ahTp05JJwwcMmQIZs2aBeDT+JQtW7agc+fOqF27Nuzt7QF8OqHl48ePkTdvXly6dAn6+vr4+PEjQ8pvjj0qMpPanZn6CyD1C+P8+fMYNWoUTE1N0adPH2k30NChQ5E3b160bNkSOXLk0HL1lBF83s0+ZMgQbNmyBbGxsXBwcECuXLlw4MABAMCKFSswbdo0tG3bFu7u7iqnZeAvWPoZQghERUXB1tYWDRo0gLW1NQICAnDy5EkUK1YMkZGRcHV1haenJ/r37y/d5vPPToDbYUbBoCIjqW+04OBg7NmzBxEREahcuTL+/PNPWFhY4Ny5cxg/fjwSExORN29emJiYYMuWLbh+/TocHBy0XT5lAJ+HlEOHDmHQoEHw9fWFhYUF/vnnH0ycOBGmpqa4dOkSgE9jVjw8PODl5YV+/fpps3TKgMLDw5E3b16kpKRgx44daNiwobTsa2NTvjaWhX5/3PUjIwqFAjt37kSTJk3w4cMHfPjwAX5+fujTpw8iIiLg6uqKefPmoVq1aggJCcGjR49w9OhRhhTSmNQP9j179mDz5s2oXbs2KleujGLFiqF169bYsGEDYmNj0adPHwBA9+7dsXv3buk6kaYkJibi1atXMDExga6uLtasWSNNjQcAa2tr6f+pR0H+PJgwpGQc7FGRkUuXLqFt27YYNWoUevTogbCwMJQuXRrGxsYoWbIkNmzYACsrK+ncKV9OASXShIiICDRu3BjXr19HjRo1sG/fPpXlY8aMwenTp/H333/D1NRUamc3O/2sbw3gDg0NhYuLC2rUqIEFCxYgX758WqiOtIU9Kloyc+ZMjB07VvolAHw6RLmrqyt69OiB0NBQ1KpVC82bN8e4ceNw8eJF9O3bFxERETAyMgIAhhTSiM+3QQCwsrLC+vXrUadOHVy9ehVr165VWZ4/f368e/cO8fHxKu0MKfQzPg8px44dw8aNG3H9+nU8f/4cTk5OOH36NIKDgzFixAhpAHeLFi2wZMkSbZZNvwB7VLRkyZIlGDhwIGbMmIERI0ZIb9A7d+6gYMGCaNasmfSFoVQqUbJkSYSEhKBRo0bYsmULz5VCGvH5l8PDhw+hUChgYmICOzs7PH78GJ6enoiLi8Off/4JDw8PvH79Gu7u7jAyMsK+ffvYvU4aN2zYMKxfvx56enrIkiUL7OzssHDhQpQtWxY3b95EjRo14OTkhKSkJHz8+BHXr1+XTsxKGZSgX06pVAohhFi5cqXQ0dERU6dOFcnJydLyp0+fisKFC4t9+/YJIYSIiIgQ7dq1E0uWLBHPnj3TSs2U8aRuh0IIMXHiRFG8eHFRqFAhkSNHDuHr6yuEECIkJEQ0bNhQGBkZiYIFC4oWLVqIevXqifj4eCGEECkpKVqpnTKOz7fDoKAgUaJECXHy5EkREREhdu/eLVq0aCGcnZ3FlStXhBBCPHjwQEyZMkVMnz5d+tz8/POTMh4GlV9MqVRKb0ylUin++usvoaOjI6ZNmyZ96IeHh4uSJUsKDw8PERoaKsaMGSPKlSsnXr9+rc3SKYOaMmWKsLGxEYGBgSI2Nla0aNFCWFhYiNu3bwshhHj06JFo1KiRKFmypFi4cKF0u4SEBC1VTBnR+vXrRb9+/USvXr1U2i9evCjq168v3N3dRWxsrBBCNdwwpGR83H+gBQqFAocPH8bQoUNRpkwZ6Rwqs2bNghAClpaW6NChA44fPw5XV1ds2LABPj4+sLW11XbplAF8PiZFqVTiwoULWLhwIerWrYugoCAcO3YMM2bMQJEiRZCcnIw8efJg/vz5yJ49O/bv34+AgAAAgKGhobaeAmUA4otRB7t27cKyZctw7do1JCYmSu1ly5ZFlSpVcOrUKaSkpABQndHDc0hlAtpOSpnRjh07hLGxsZg6daq4ePGiEEIIX19faTeQEEIkJiaK27dvi6CgIPH06VNtlksZ1IQJE8SsWbNEzpw5xb1790RwcLDIkiWL8Pb2FkII8eHDBzF27FgRGhoqhBDi/v37onHjxqJs2bIiICBAm6XTb+7zHhF/f3+xYcMGIYQQ/fr1ExYWFmLZsmUiKipKWicwMFAUKlRI2hYpc2FQ+cXu3bsn8uTJI5YvX55m2YoVK6TdQESa9vl4ks2bNwtHR0dx69Yt0bFjR1GvXj1hYmIiVq9eLa3z/PlzUaVKFbFhwwbptnfu3BGtW7cWYWFhv7x+yhg+3w5v3bolSpUqJUqUKCF2794thBDC3d1d5M+fX0yfPl2EhISIkJAQUatWLVGtWjWVgEOZB/vMfrEnT55AX19f5QiLqTMvevXqBVNTU3Tq1AmGhoYYNmyYFiuljCZ1ds/x48dx7NgxDB06FEWLFpUOJFirVi1069YNwKeTYfbo0QO6urpo3749dHR0oFQqUahQIWzcuJGzLOiHpW6Hw4cPx+PHj2FsbIy7d+9i8ODB+PjxI9atW4du3bph3LhxWLJkCSpVqoQsWbJgy5YtUCgU3zzWCmVcDCq/WGxsrMrxJ5RKpbS/9dixYyhTpgy2bNmict4UIk159eoVunfvjvDwcIwZMwYA0Lt3bzx8+BBHjx5FqVKlkD9/fjx58gQJCQm4ePEidHV1VQ7mxjEB9LPWrVuHVatW4ciRI8iTJw8SExPh7u6OmTNnQkdHB2vWrIGJiQm2bt2K+vXro23btjA0NERSUhIMDAy0XT79Yoylv1iJEiXw9u1b+Pr6Avj06yI1qOzevRsbN25Ey5YtUbhwYW2WSRmUnZ0dAgICkD17duzduxeXL1+Grq4u5s6diylTpqBmzZqws7NDmzZtvnn2WR47hX5WSEgIihUrhpIlS8Lc3Bx2dnZYs2YNdHV1MXjwYOzcuRNLly5F7dq1sWDBAuzZswcxMTEMKZkUfxr9Ynny5MHSpUvRu3dvJCcno3PnztDV1cW6deuwbt06nD17lkf4pHTl4uKCHTt2wN3dHT4+Pujfvz9cXFzQtGlTNG3aVGXdlJQU9qCQxoj/P1GgoaEhEhISkJSUBCMjIyQnJyNnzpyYOXMmGjduDC8vLxgbG2Pjxo1o3749hg0bBj09Pbi5uWn7KZAW8Mi0WqBUKrFjxw54eHjA1NQURkZG0NXVxaZNm1CqVCltl0eZxNWrV9GjRw+UKVMGAwcORNGiRbVdEmUSN2/eRKlSpTB+/HhMnDhRag8MDMTKlSvx/v17pKSk4NixYwCArl27Yvz48cibN6+WKiZtYlDRohcvXiAsLAwKhQJ58uRB9uzZtV0SZTJXr16Fh4cHcufOjTlz5iBPnjzaLokyiXXr1qFXr14YNGgQ2rRpA0tLSwwYMAAVK1ZEixYtULRoUezfvx8NGjTQdqmkZQwqRJnchQsX4OPjg1WrVnE2Bf1SO3bsQN++fWFgYAAhBGxtbXHmzBm8fv0aderUwfbt2+Hi4qLtMknLGFSISBo7wKmf9Ks9f/4cT58+RXJyMipVqgQdHR2MHj0au3btQnBwMOzs7LRdImkZgwoRAfhfWCHSltu3b2P27Nn4+++/cfjwYZQsWVLbJZEMcDg/EQHgtGPSro8fPyIpKQm2trY4fvw4B3eThD0qREQkG8nJyTzyMalgUCEiIiLZ4qg5IiIiki0GFSIiIpItBhUiIiKSLQYVIiIiki0GFSL6rRw7dgwKhQKRkZHffRsnJyd4eXmlW01ElH4YVIhIo7p06QKFQoHevXunWebp6QmFQoEuXbr8+sKI6LfEoEJEGufo6IjNmzcjPj5eaktISMDGjRuRK1cuLVZGRL8bBhUi0rjSpUvD0dERAQEBUltAQABy5cqFUqVKSW2JiYkYMGAAbG1tYWRkhMqVK+PixYsq9/X333+jQIECMDY2Ro0aNRAaGprm8U6dOoUqVarA2NgYjo6OGDBgAOLi4tLt+RHRr8OgQkTpolu3bli7dq10fc2aNejatavKOiNGjMCOHTuwfv16XLlyBc7OzqhXrx4iIiIAAE+fPkXLli3RpEkTXLt2DT169MCoUaNU7uPhw4eoX78+WrVqhRs3bmDLli04deoU+vXrl/5PkojSHYMKEaWLjh074tSpUwgLC0NYWBhOnz6Njh07Ssvj4uLg7e2NuXPnokGDBihSpAhWrlwJY2NjrF69GgDg7e2NfPnyYf78+ShYsCA6dOiQZnzLzJkz0aFDBwwaNAj58+dHxYoVsXjxYmzYsAEJCQm/8ikTUTrgSQmJKF3Y2NigUaNGWLduHYQQaNSoEaytraXlDx8+RHJyMipVqiS16evr448//sCdO3cAAHfu3EH58uVV7rdChQoq169fv44bN27A399fahNCQKlU4vHjxyhcuHB6PD0i+kUYVIgo3XTr1k3aBbNs2bJ0eYzY2Fh4eHhgwIABaZZx4C7R749BhYjSTf369ZGUlASFQoF69eqpLMuXLx8MDAxw+vRp5M6dG8CnM+devHgRgwYNAgAULlwYe/bsUbnduXPnVK6XLl0a//zzD5ydndPviRCR1nCMChGlG11dXdy5cwf//PMPdHV1VZaZmpqiT58+GD58OA4ePIh//vkHPXv2xIcPH9C9e3cAQO/evfHgwQMMHz4c9+7dw8aNG7Fu3TqV+xk5ciTOnDmDfv364dq1a3jw4AF2797NwbREGQSDChGlKzMzM5iZmX112axZs9CqVSt06tQJpUuXRkhICAIDA2FpaQng066bHTt2YNeuXShRogR8fHwwY8YMlftwcXHB8ePHcf/+fVSpUgWlSpXChAkTYG9vn+7PjYjSn0IIIbRdBBEREdHXsEeFiIiIZItBhYiIiGSLQYWIiIhki0GFiIiIZItBhYiIiGSLQYWIiIhki0GFiIiIZItBhYiIiGSLQYWIiIhki0GFiIiIZItBhYiIiGSLQYWIiIhk6/8AHoK08GWUizwAAAAASUVORK5CYII=",
- "text/plain": [
- "
"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "import matplotlib.pyplot as plt\n",
- "\n",
- "## calculate avg response time\n",
- "unique_models = set(result[\"response\"]['model'] for result in result[\"results\"])\n",
- "model_dict = {model: {\"response_time\": []} for model in unique_models}\n",
- "for completion_result in result[\"results\"]:\n",
- " model_dict[completion_result[\"response\"][\"model\"]][\"response_time\"].append(completion_result[\"response_time\"])\n",
- "\n",
- "avg_response_time = {}\n",
- "for model, data in model_dict.items():\n",
- " avg_response_time[model] = sum(data[\"response_time\"]) / len(data[\"response_time\"])\n",
- "\n",
- "models = list(avg_response_time.keys())\n",
- "response_times = list(avg_response_time.values())\n",
- "\n",
- "plt.bar(models, response_times)\n",
- "plt.xlabel('Model', fontsize=10)\n",
- "plt.ylabel('Average Response Time')\n",
- "plt.title('Average Response Times for each Model')\n",
- "\n",
- "plt.xticks(models, [model[:15]+'...' if len(model) > 15 else model for model in models], rotation=45)\n",
- "plt.show()"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "inSDIE3_IRds"
- },
- "source": [
- "# Duration Test endpoint\n",
- "\n",
- "Run load testing for 2 mins. Hitting endpoints with 100+ queries every 15 seconds."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 20,
- "metadata": {
- "id": "ePIqDx2EIURH"
- },
- "outputs": [],
- "source": [
- "models=[\"gpt-3.5-turbo\", \"replicate/llama-2-70b-chat:58d078176e02c219e11eb4da5a02a7830a283b14cf8f94537af893ccff5ee781\", \"claude-instant-1\"]\n",
- "context = \"\"\"Paul Graham (/ɔrƦm/; born 1964)[3] is an English computer scientist, essayist, entrepreneur, venture capitalist, and author. He is best known for his work on the programming language Lisp, his former startup Viaweb (later renamed Yahoo! Store), cofounding the influential startup accelerator and seed capital firm Y Combinator, his essays, and Hacker News. He is the author of several computer programming books, including: On Lisp,[4] ANSI Common Lisp,[5] and Hackers & Painters.[6] Technology journalist Steven Levy has described Graham as a \"hacker philosopher\".[7] Graham was born in England, where he and his family maintain permanent residence. However he is also a citizen of the United States, where he was educated, lived, and worked until 2016.\"\"\"\n",
- "prompt = \"Where does Paul Graham live?\"\n",
- "final_prompt = context + prompt\n",
- "result = load_test_model(models=models, prompt=final_prompt, num_calls=100, interval=15, duration=120)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 27,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 552
- },
- "id": "k6rJoELM6t1K",
- "outputId": "f4968b59-3bca-4f78-a88b-149ad55e3cf7"
- },
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAIXCAYAAABghH+YAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABwdUlEQVR4nO3dd1QU198G8GfpoNKUooKCYuwIaiL2GrGLJnYFOxrsNZbYFTsYG2JDjV2xRKOIir33EhsWLBGwUaXJ3vcPX+bnCiYsLi6Oz+ecPbp37ux+lx3YZ+/cmVEIIQSIiIiIZEJH2wUQERERaRLDDREREckKww0RERHJCsMNERERyQrDDREREckKww0RERHJCsMNERERyQrDDREREckKww0RERHJCsMNEcmSg4MDunfvru0y1DZnzhyUKFECurq6cHFx0XY5GnfkyBEoFAps27ZN26WoTaFQYNKkSWqv9+jRIygUCgQFBWm8Jsoaww19tiVLlkChUKBatWraLiXPcXBwgEKhkG758uXDDz/8gLVr12q7tK9Oxodidm5fqwMHDmDUqFGoWbMmVq9ejRkzZmi7pDwnKChIep9PnDiRabkQAvb29lAoFGjRooUWKqS8QE/bBdDXb/369XBwcMC5c+cQHh4OJycnbZeUp7i4uGD48OEAgOfPn2PFihXw8vJCSkoK+vTpo+Xqvh5ly5bFunXrVNrGjBmD/PnzY9y4cZn637lzBzo6X9f3t8OHD0NHRwcrV66EgYGBtsvJ04yMjLBhwwbUqlVLpf3o0aN4+vQpDA0NtVQZ5QUMN/RZHj58iFOnTiE4OBje3t5Yv349Jk6c+EVrUCqVSE1NhZGR0Rd93uwqWrQounbtKt3v3r07SpQoAT8/P4YbNdjY2Kj8HAFg5syZKFSoUKZ2AF/lh1t0dDSMjY01FmyEEEhOToaxsbFGHi8vadasGbZu3Yrff/8denr/+yjbsGEDqlSpgpcvX2qxOtK2r+trDeU569evh4WFBZo3b46ff/4Z69evl5alpaXB0tISPXr0yLReXFwcjIyMMGLECKktJSUFEydOhJOTEwwNDWFvb49Ro0YhJSVFZV2FQoEBAwZg/fr1KF++PAwNDbF//34AwNy5c1GjRg0ULFgQxsbGqFKlSpb79pOSkjBo0CAUKlQIBQoUQKtWrfDs2bMs96k/e/YMPXv2hI2NDQwNDVG+fHmsWrUqxz8zKysrlClTBvfv31dpVyqV8Pf3R/ny5WFkZAQbGxt4e3vjzZs3Kv0uXLgAd3d3FCpUCMbGxnB0dETPnj2l5Rn79+fOnQs/Pz8UL14cxsbGqFu3Lm7cuJGpnsOHD6N27drIly8fzM3N0bp1a9y6dUulz6RJk6BQKBAeHo7u3bvD3NwcZmZm6NGjB96+favSNzQ0FLVq1YK5uTny58+P0qVLY+zYsSp9svtef46P59xk7M44ceIEBg0aBCsrK5ibm8Pb2xupqamIiYmBp6cnLCwsYGFhgVGjRkEIofKYmnqPsqJQKLB69WokJiZKu10y5mi8e/cOU6dORcmSJWFoaAgHBweMHTs208/LwcEBLVq0QEhICKpWrQpjY2MsW7bsX5/37NmzaNKkCczMzGBiYoK6devi5MmTKn0iIiLwyy+/oHTp0jA2NkbBggXRrl07PHr0KNPjxcTEYOjQoXBwcIChoSHs7Ozg6emZKWwolUpMnz4ddnZ2MDIyQsOGDREeHv6vtX6oU6dOePXqFUJDQ6W21NRUbNu2DZ07d85yncTERAwfPhz29vYwNDRE6dKlMXfu3Ezvc0pKCoYOHQorKyvp78PTp0+zfExN/30gDRFEn6FMmTKiV69eQgghjh07JgCIc+fOSct79uwpzM3NRUpKisp6a9asEQDE+fPnhRBCpKeni8aNGwsTExMxZMgQsWzZMjFgwAChp6cnWrdurbIuAFG2bFlhZWUlJk+eLBYvXiwuX74shBDCzs5O/PLLL2LRokVi/vz54ocffhAAxJ49e1Qeo3379gKA6Natm1i8eLFo3769qFSpkgAgJk6cKPWLjIwUdnZ2wt7eXkyZMkUsXbpUtGrVSgAQfn5+//nzKV68uGjevLlKW1pamrC1tRU2NjYq7b179xZ6enqiT58+IiAgQIwePVrky5dPfP/99yI1NVUIIURUVJSwsLAQ3333nZgzZ45Yvny5GDdunChbtqz0OA8fPhQARMWKFYWDg4OYNWuWmDx5srC0tBRWVlYiMjJS6hsaGir09PTEd999J2bPni0mT54sChUqJCwsLMTDhw+lfhMnThQAhKurq2jbtq1YsmSJ6N27twAgRo0aJfW7ceOGMDAwEFWrVhULFiwQAQEBYsSIEaJOnTpSH3Xe6/9Svnx5Ubdu3U/+7L28vKT7q1evFgCEi4uLaNKkiVi8eLHo1q2b9Bpq1aolOnfuLJYsWSJatGghAIg1a9bkynuUlXXr1onatWsLQ0NDsW7dOrFu3Tpx//59IYQQXl5eAoD4+eefxeLFi4Wnp6cAIDw8PDK9ZicnJ2FhYSF+/fVXERAQIMLCwj75nIcOHRIGBgaievXqYt68ecLPz084OzsLAwMDcfbsWanf1q1bRaVKlcSECRNEYGCgGDt2rLCwsBDFixcXiYmJUr/4+HhRoUIFoaurK/r06SOWLl0qpk6dKr7//nvpdzQsLEzalqpUqSL8/PzEpEmThImJifjhhx/+9Wf04ft4/vx5UaNGDdGtWzdp2c6dO4WOjo549uxZpt89pVIpGjRoIBQKhejdu7dYtGiRaNmypQAghgwZovIcXbt2FQBE586dxaJFi0Tbtm2Fs7Nzjv8+ZPxOrl69+j9fH2kGww3l2IULFwQAERoaKoR4/8fDzs5ODB48WOoTEhIiAIg///xTZd1mzZqJEiVKSPfXrVsndHR0xPHjx1X6BQQECADi5MmTUhsAoaOjI27evJmpprdv36rcT01NFRUqVBANGjSQ2i5evJjlH7Tu3btn+uPVq1cvUbhwYfHy5UuVvh07dhRmZmaZnu9jxYsXF40bNxYvXrwQL168ENevX5c+UH18fKR+x48fFwDE+vXrVdbfv3+/SvuOHTtUQmFWMv6QGhsbi6dPn0rtZ8+eFQDE0KFDpTYXFxdhbW0tXr16JbVdvXpV6OjoCE9PT6ktI9z07NlT5bnatGkjChYsKN338/MTAMSLFy8+WZ867/V/yUm4cXd3F0qlUmqvXr26UCgUol+/flLbu3fvhJ2dncpja/I9+hQvLy+RL18+lbYrV64IAKJ3794q7SNGjBAAxOHDh1VeMwCxf//+/3wupVIpSpUqlenn8fbtW+Ho6Ch+/PFHlbaPnT59WgAQa9euldomTJggAIjg4OAsn0+I/4WbsmXLqnzpWbBggQAgrl+//q91fxhuFi1aJAoUKCDV165dO1G/fn3pZ/FhuNm5c6cAIKZNm6byeD///LNQKBQiPDxcCPG/n/cvv/yi0q9z5845/vvAcPPlcbcU5dj69ethY2OD+vXrA3g/rN6hQwds2rQJ6enpAIAGDRqgUKFC2Lx5s7TemzdvEBoaig4dOkhtW7duRdmyZVGmTBm8fPlSujVo0AAAEBYWpvLcdevWRbly5TLV9OHcgjdv3iA2Nha1a9fGpUuXpPaMXVi//PKLyroDBw5UuS+EwPbt29GyZUsIIVTqcnd3R2xsrMrjfsqBAwdgZWUFKysrVKxYEevWrUOPHj0wZ84clddvZmaGH3/8UeV5qlSpgvz580uv39zcHACwZ88epKWl/evzenh4oGjRotL9H374AdWqVcNff/0F4P3k5itXrqB79+6wtLSU+jk7O+PHH3+U+n2oX79+Kvdr166NV69eIS4uTqW+Xbt2QalUZlmXuu+1pvXq1UvliKpq1apBCIFevXpJbbq6uqhatSoePHigUrem36PsyHgfhg0bptKeMUl97969Ku2Ojo5wd3f/z8e9cuUK7t27h86dO+PVq1fS60lMTETDhg1x7Ngx6T388PcqLS0Nr169gpOTE8zNzVV+B7Zv345KlSqhTZs2mZ7v46PYevTooTK3qHbt2gCg8jP/L+3bt0dSUhL27NmD+Ph47Nmz55O7pP766y/o6upi0KBBKu3Dhw+HEAL79u2T+gHI1G/IkCEq9zX194Fyxzcdbo4dO4aWLVuiSJEiUCgU2LlzZ64/57Nnz9C1a1dpTkjFihVx4cKFXH9eTUtPT8emTZtQv359PHz4EOHh4QgPD0e1atUQFRWFQ4cOAQD09PTw008/YdeuXdL8gODgYKSlpamEm3v37uHmzZtSCMi4fffddwDeT7T8kKOjY5Z17dmzB25ubjAyMoKlpSWsrKywdOlSxMbGSn0iIiKgo6OT6TE+PsrrxYsXiImJQWBgYKa6MuYRfVxXVqpVq4bQ0FDs378fc+fOhbm5Od68eaPyh/3evXuIjY2FtbV1pudKSEiQnqdu3br46aefMHnyZBQqVAitW7fG6tWrs5yrUqpUqUxt3333nTRPIiIiAgBQunTpTP3Kli0rfdB9qFixYir3LSwsAECac9KhQwfUrFkTvXv3ho2NDTp27IgtW7aoBB1132tN+/g1mJmZAQDs7e0ztX84lyY33qPsyNheP94+bW1tYW5uLr2PGT71u/Gxe/fuAQC8vLwyvZ4VK1YgJSVF+r1JSkrChAkTpLkqhQoVgpWVFWJiYlR+t+7fv48KFSpk6/n/a1vKDisrKzRq1AgbNmxAcHAw0tPT8fPPP2fZNyIiAkWKFEGBAgVU2suWLSstz/hXR0cHJUuWVOn38e+Jpv4+UO74po+WSkxMRKVKldCzZ0+0bds215/vzZs3qFmzJurXr499+/bBysoK9+7dk36pvyaHDx/G8+fPsWnTJmzatCnT8vXr16Nx48YAgI4dO2LZsmXYt28fPDw8sGXLFpQpUwaVKlWS+iuVSlSsWBHz58/P8vk+/uDJ6uiP48ePo1WrVqhTpw6WLFmCwoULQ19fH6tXr8aGDRvUfo0ZH8hdu3aFl5dXln2cnZ3/83EKFSqERo0aAQDc3d1RpkwZtGjRAgsWLJC+jSuVSlhbW6tMyP6QlZUVAEgnPztz5gz+/PNPhISEoGfPnpg3bx7OnDmD/Pnzq/061aGrq5tlu/j/CZnGxsY4duwYwsLCsHfvXuzfvx+bN29GgwYNcODAAejq6qr9Xmvap15DVu3ig4mm2n6Psnv+nuweGZWxfc+ZM+eTJwvMqHXgwIFYvXo1hgwZgurVq8PMzAwKhQIdO3b85Ajdf/mvbSm7OnfujD59+iAyMhJNmzaVRs5ym6b+PlDu+KbDTdOmTdG0adNPLk9JScG4ceOwceNGxMTEoEKFCpg1axbq1auXo+ebNWsW7O3tsXr1aqktu9+y8pr169fD2toaixcvzrQsODgYO3bsQEBAAIyNjVGnTh0ULlwYmzdvRq1atXD48OFM5yUpWbIkrl69ioYNG+b4JGzbt2+HkZERQkJCVA4D/vDnDQDFixeHUqnEw4cPVUY3Pj5SI+NIifT0dCmcaELz5s1Rt25dzJgxA97e3siXLx9KliyJgwcPombNmtn6cHJzc4ObmxumT5+ODRs2oEuXLti0aRN69+4t9cn4Zv6hu3fvwsHBAcD7nwPw/nwwH7t9+zYKFSqEfPnyqf36dHR00LBhQzRs2BDz58/HjBkzMG7cOISFhaFRo0Yaea+1ITfeo+zI2F7v3bsnjTIAQFRUFGJiYqT3UV0ZIxOmpqb/uX1v27YNXl5emDdvntSWnJyMmJiYTI+Z1RF5ualNmzbw9vbGmTNnVHZ/f6x48eI4ePAg4uPjVUZvbt++LS3P+FepVOL+/fsqozUf/57k1t8H0oxverfUfxkwYABOnz6NTZs24dq1a2jXrh2aNGmS5YdGduzevRtVq1ZFu3btYG1tDVdXVyxfvlzDVee+pKQkBAcHo0WLFvj5558z3QYMGID4+Hjs3r0bwPsPu59//hl//vkn1q1bh3fv3qnskgLe7zt/9uxZlj+PpKSkTLtHsqKrqwuFQiHN9wHeHxb98e7GjPkIS5YsUWlfuHBhpsf76aefsH379iz/YL948eI/a/qU0aNH49WrV9Lrbd++PdLT0zF16tRMfd+9eyd9iLx58ybTN9uMb90f7/bYuXMnnj17Jt0/d+4czp49KwX6woULw8XFBWvWrFH5kLpx4wYOHDiAZs2aqf26Xr9+nant4/o08V5rQ268R9mR8T74+/urtGeMfDVv3lztxwSAKlWqoGTJkpg7dy4SEhIyLf9w+9bV1c30mhYuXKjyuwYAP/30E65evYodO3Zkejx1R2SyK3/+/Fi6dCkmTZqEli1bfrJfs2bNkJ6ejkWLFqm0+/n5QaFQSL8XGf/+/vvvKv0+/vnn5t8H+nzf9MjNv3n8+DFWr16Nx48fo0iRIgCAESNGYP/+/Tk+LfqDBw+wdOlSDBs2DGPHjsX58+cxaNAgGBgYfHJYMy/avXs34uPj0apVqyyXu7m5wcrKCuvXr5dCTIcOHbBw4UJMnDgRFStWVPkGCgDdunXDli1b0K9fP4SFhaFmzZpIT0/H7du3sWXLFum8Hf+mefPmmD9/Ppo0aYLOnTsjOjoaixcvhpOTE65duyb1q1KlCn766Sf4+/vj1atXcHNzw9GjR3H37l0AqsP/M2fORFhYGKpVq4Y+ffqgXLlyeP36NS5duoSDBw9m+WGeHU2bNkWFChUwf/58+Pj4oG7duvD29oavry+uXLmCxo0bQ19fH/fu3cPWrVuxYMEC/Pzzz1izZg2WLFmCNm3aoGTJkoiPj8fy5cthamqaKYw4OTmhVq1a6N+/P1JSUuDv74+CBQti1KhRUp85c+agadOmqF69Onr16oWkpCQsXLgQZmZmObqGzpQpU3Ds2DE0b94cxYsXR3R0NJYsWQI7OzvpTLKaeK+1ITfeo+yoVKkSvLy8EBgYiJiYGNStWxfnzp3DmjVr4OHhIU3oV5eOjg5WrFiBpk2bonz58ujRoweKFi2KZ8+eISwsDKampvjzzz8BAC1atMC6detgZmaGcuXK4fTp0zh48CAKFiyo8pgjR47Etm3b0K5dO/Ts2RNVqlTB69evsXv3bgQEBKjsitak7Pz9bNmyJerXr49x48bh0aNHqFSpEg4cOIBdu3ZhyJAh0kiWi4sLOnXqhCVLliA2NhY1atTAoUOHsjwHT279fSAN0MoxWnkQALFjxw7p/p49ewQAkS9fPpWbnp6eaN++vRBCiFu3bgkA/3obPXq09Jj6+vqievXqKs87cOBA4ebm9kVeo6a0bNlSGBkZqZzf4mPdu3cX+vr60iGSSqVS2NvbZ3koZobU1FQxa9YsUb58eWFoaCgsLCxElSpVxOTJk0VsbKzUDx8dRv2hlStXilKlSglDQ0NRpkwZsXr1aukw5g8lJiYKHx8fYWlpKfLnzy88PDzEnTt3BAAxc+ZMlb5RUVHCx8dH2NvbC319fWFraysaNmwoAgMD//NnldV5bjIEBQVlOjw0MDBQVKlSRRgbG4sCBQqIihUrilGjRol//vlHCCHEpUuXRKdOnUSxYsWEoaGhsLa2Fi1atBAXLlyQHiPjsNM5c+aIefPmCXt7e2FoaChq164trl69mqmOgwcPipo1awpjY2NhamoqWrZsKf7++2+VPhk/w48P8c44LDfjnDiHDh0SrVu3FkWKFBEGBgaiSJEiolOnTuLu3bsq62X3vf4vOTkU/ONDtD/12rI6LFsIzbxHn/Kp50xLSxOTJ08Wjo6OQl9fX9jb24sxY8aI5OTkTK/5U9vbp1y+fFm0bdtWFCxYUBgaGorixYuL9u3bi0OHDkl93rx5I3r06CEKFSok8ufPL9zd3cXt27cz/YyFEOLVq1diwIABomjRosLAwEDY2dkJLy8v6W9BxqHgW7duVVkvu4dLf+p9/FhWP4v4+HgxdOhQUaRIEaGvry9KlSol5syZo3IovBBCJCUliUGDBomCBQuKfPnyiZYtW4onT55kOhRciOz9feCh4F+eQohcGiv8yigUCuzYsQMeHh4AgM2bN6NLly64efNmpolv+fPnh62tLVJTU//zsMWCBQtKEw2LFy+OH3/8EStWrJCWL126FNOmTVPZfUDaceXKFbi6uuKPP/5Aly5dtF1Ojj169AiOjo6YM2eOyhmgiYi+Fdwt9Qmurq5IT09HdHS0dP6FjxkYGKBMmTLZfsyaNWtmmpR29+7dHE8IpJxLSkrKNCnU398fOjo6qFOnjpaqIiIiTfimw01CQoLKftSHDx/iypUrsLS0xHfffYcuXbrA09MT8+bNg6urK168eIFDhw7B2dk5R5P4hg4diho1amDGjBlo3749zp07h8DAQAQGBmryZVE2zJ49GxcvXkT9+vWhp6eHffv2Yd++fejbt2+uH4pMRES5TNv7xbQpY9/vx7eMfcipqaliwoQJwsHBQejr64vChQuLNm3aiGvXruX4Of/8809RoUIFaU5IduZtkOYdOHBA1KxZU1hYWAh9fX1RsmRJMWnSJJGWlqbt0j7bh3NuiIi+RZxzQ0RERLLC89wQERGRrDDcEBERkax8cxOKlUol/vnnHxQoUOCrOvU7ERHRt0wIgfj4eBQpUgQ6Ov8+NvPNhZt//vmHR8MQERF9pZ48eQI7O7t/7fPNhZuMC6Y9efIEpqamWq6GiIiIsiMuLg729vYqFz79lG8u3GTsijI1NWW4ISIi+spkZ0oJJxQTERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGs6Gm7ACLSLIdf92q7BNKyRzOba/X5uQ2StrdBjtwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrGg13CxduhTOzs4wNTWFqakpqlevjn379v3rOlu3bkWZMmVgZGSEihUr4q+//vpC1RIREdHXQKvhxs7ODjNnzsTFixdx4cIFNGjQAK1bt8bNmzez7H/q1Cl06tQJvXr1wuXLl+Hh4QEPDw/cuHHjC1dOREREeZVCCCG0XcSHLC0tMWfOHPTq1SvTsg4dOiAxMRF79uyR2tzc3ODi4oKAgIBsPX5cXBzMzMwQGxsLU1NTjdVNlFfwooWk7YsWchuk3NgG1fn8zjNzbtLT07Fp0yYkJiaievXqWfY5ffo0GjVqpNLm7u6O06dPf/JxU1JSEBcXp3IjIiIi+dJ6uLl+/Try588PQ0ND9OvXDzt27EC5cuWy7BsZGQkbGxuVNhsbG0RGRn7y8X19fWFmZibd7O3tNVo/ERER5S1aDzelS5fGlStXcPbsWfTv3x9eXl74+++/Nfb4Y8aMQWxsrHR78uSJxh6biIiI8h49bRdgYGAAJycnAECVKlVw/vx5LFiwAMuWLcvU19bWFlFRUSptUVFRsLW1/eTjGxoawtDQULNFExERUZ6l9ZGbjymVSqSkpGS5rHr16jh06JBKW2ho6Cfn6BAREdG3R6sjN2PGjEHTpk1RrFgxxMfHY8OGDThy5AhCQkIAAJ6enihatCh8fX0BAIMHD0bdunUxb948NG/eHJs2bcKFCxcQGBiozZdBREREeYhWw010dDQ8PT3x/PlzmJmZwdnZGSEhIfjxxx8BAI8fP4aOzv8Gl2rUqIENGzZg/PjxGDt2LEqVKoWdO3eiQoUK2noJRERElMdoNdysXLnyX5cfOXIkU1u7du3Qrl27XKqIiIiIvnZ5bs4NERER0edguCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZYbghIiIiWdFquPH19cX333+PAgUKwNraGh4eHrhz586/rhMUFASFQqFyMzIy+kIVExERUV6n1XBz9OhR+Pj44MyZMwgNDUVaWhoaN26MxMTEf13P1NQUz58/l24RERFfqGIiIiLK6/S0+eT79+9XuR8UFARra2tcvHgRderU+eR6CoUCtra2uV0eERERfYXy1Jyb2NhYAIClpeW/9ktISEDx4sVhb2+P1q1b4+bNm5/sm5KSgri4OJUbERERyVeeCTdKpRJDhgxBzZo1UaFChU/2K126NFatWoVdu3bhjz/+gFKpRI0aNfD06dMs+/v6+sLMzEy62dvb59ZLICIiojwgz4QbHx8f3LhxA5s2bfrXftWrV4enpydcXFxQt25dBAcHw8rKCsuWLcuy/5gxYxAbGyvdnjx5khvlExERUR6h1Tk3GQYMGIA9e/bg2LFjsLOzU2tdfX19uLq6Ijw8PMvlhoaGMDQ01ESZRERE9BXQ6siNEAIDBgzAjh07cPjwYTg6Oqr9GOnp6bh+/ToKFy6cCxUSERHR10arIzc+Pj7YsGEDdu3ahQIFCiAyMhIAYGZmBmNjYwCAp6cnihYtCl9fXwDAlClT4ObmBicnJ8TExGDOnDmIiIhA7969tfY6iIiIKO/QarhZunQpAKBevXoq7atXr0b37t0BAI8fP4aOzv8GmN68eYM+ffogMjISFhYWqFKlCk6dOoVy5cp9qbKJiIgoD9NquBFC/GefI0eOqNz38/ODn59fLlVEREREX7s8MaFYThx+3avtEkjLHs1sru0SiIi+aXnmUHAiIiIiTWC4ISIiIllhuCEiIiJZYbghIiIiWWG4ISIiIllhuCEiIiJZyVG4uX//PsaPH49OnTohOjoaALBv3z7cvHlTo8URERERqUvtcHP06FFUrFgRZ8+eRXBwMBISEgAAV69excSJEzVeIBEREZE61A43v/76K6ZNm4bQ0FAYGBhI7Q0aNMCZM2c0WhwRERGRutQON9evX0ebNm0ytVtbW+Ply5caKYqIiIgop9QON+bm5nj+/Hmm9suXL6No0aIaKYqIiIgop9QONx07dsTo0aMRGRkJhUIBpVKJkydPYsSIEfD09MyNGomIiIiyTe1wM2PGDJQpUwb29vZISEhAuXLlUKdOHdSoUQPjx4/PjRqJiIiIsk3tq4IbGBhg+fLl+O2333Djxg0kJCTA1dUVpUqVyo36iIiIiNSidrjJUKxYMRQrVkyTtRARERF9NrXDjRAC27ZtQ1hYGKKjo6FUKlWWBwcHa6w4IiIiInWpHW6GDBmCZcuWoX79+rCxsYFCociNuoiIiIhyRO1ws27dOgQHB6NZs2a5UQ8RERHRZ1H7aCkzMzOUKFEiN2ohIiIi+mxqh5tJkyZh8uTJSEpKyo16iIiIiD6L2rul2rdvj40bN8La2hoODg7Q19dXWX7p0iWNFUdERESkLrXDjZeXFy5evIiuXbtyQjERERHlOWqHm7179yIkJAS1atXKjXqIiIiIPovac27s7e1hamqaG7UQERERfTa1w828efMwatQoPHr0KBfKISIiIvo8au+W6tq1K96+fYuSJUvCxMQk04Ti169fa6w4IiIiInWpHW78/f1zoQwiIiIizcjR0VJEREREeVW2wk1cXJw0iTguLu5f+3KyMREREWlTtsKNhYUFnj9/Dmtra5ibm2d5bhshBBQKBdLT0zVeJBEREVF2ZSvcHD58GJaWlgCAsLCwXC2IiIiI6HNkK9zUrVsXJUqUwPnz51G3bt3cromIiIgox7J9nptHjx5xlxMRERHleWqfxI+IiIgoL1PrUPCQkBCYmZn9a59WrVp9VkFEREREn0OtcPNf57jh0VJERESkbWrtloqMjIRSqfzkjcGGiIiItC3b4Sarc9sQERER5TXZDjdCiNysg4iIiEgjsh1uvLy8YGxsnJu1EBEREX22bE8oXr16dW7WQURERKQRPM8NERERyQrDDREREckKww0RERHJSo7DTXh4OEJCQpCUlAQgZ0dT+fr64vvvv0eBAgVgbW0NDw8P3Llz5z/X27p1K8qUKQMjIyNUrFgRf/31l9rPTURERPKkdrh59eoVGjVqhO+++w7NmjXD8+fPAQC9evXC8OHD1Xqso0ePwsfHB2fOnEFoaCjS0tLQuHFjJCYmfnKdU6dOoVOnTujVqxcuX74MDw8PeHh44MaNG+q+FCIiIpIhtcPN0KFDoaenh8ePH8PExERq79ChA/bv36/WY+3fvx/du3dH+fLlUalSJQQFBeHx48e4ePHiJ9dZsGABmjRpgpEjR6Js2bKYOnUqKleujEWLFqn7UoiIiEiG1Lq2FAAcOHAAISEhsLOzU2kvVaoUIiIiPquY2NhYAIClpeUn+5w+fRrDhg1TaXN3d8fOnTuz7J+SkoKUlBTpflxc3GfVSERERHmb2iM3iYmJKiM2GV6/fg1DQ8McF6JUKjFkyBDUrFkTFSpU+GS/yMhI2NjYqLTZ2NggMjIyy/6+vr4wMzOTbvb29jmukYiIiPI+tcNN7dq1sXbtWum+QqGAUqnE7NmzUb9+/RwX4uPjgxs3bmDTpk05foysjBkzBrGxsdLtyZMnGn18IiIiylvU3i01e/ZsNGzYEBcuXEBqaipGjRqFmzdv4vXr1zh58mSOihgwYAD27NmDY8eOZdrd9TFbW1tERUWptEVFRcHW1jbL/oaGhp81okRERERfF7VHbipUqIC7d++iVq1aaN26NRITE9G2bVtcvnwZJUuWVOuxhBAYMGAAduzYgcOHD8PR0fE/16levToOHTqk0hYaGorq1aur9dxEREQkT2qP3ACAmZkZxo0b99lP7uPjgw0bNmDXrl0oUKCANG/GzMxMukinp6cnihYtCl9fXwDA4MGDUbduXcybNw/NmzfHpk2bcOHCBQQGBn52PURERPT1U3vkZv/+/Thx4oR0f/HixXBxcUHnzp3x5s0btR5r6dKliI2NRb169VC4cGHptnnzZqnP48ePpXPpAECNGjWwYcMGBAYGolKlSti2bRt27tz5r5OQiYiI6Nuh9sjNyJEjMWvWLADA9evXMWzYMAwfPhxhYWEYNmyYWlcPz85ZjY8cOZKprV27dmjXrl22n4eIiIi+HWqHm4cPH6JcuXIAgO3bt6Nly5aYMWMGLl26hGbNmmm8QCIiIiJ1qL1bysDAAG/fvgUAHDx4EI0bNwbw/sR7PEEeERERaZvaIze1atXCsGHDULNmTZw7d06aH3P37t3/PIybiIiIKLepPXKzaNEi6OnpYdu2bVi6dCmKFi0KANi3bx+aNGmi8QKJiIiI1KH2yE2xYsWwZ8+eTO1+fn4aKYiIiIjoc+ToPDdKpRLh4eGIjo6GUqlUWVanTh2NFEZERESUE2qHmzNnzqBz586IiIjIdCi3QqFAenq6xoojIiIiUpfa4aZfv36oWrUq9u7di8KFC0OhUORGXUREREQ5ona4uXfvHrZt2wYnJ6fcqIeIiIjos6h9tFS1atUQHh6eG7UQERERfTa1R24GDhyI4cOHIzIyEhUrVoS+vr7KcmdnZ40VR0RERKQutcPNTz/9BADo2bOn1KZQKCCE4IRiIiIi0rocXVuKiIiIKK9SO9wUL148N+ogIiIi0ogcncTv/v378Pf3x61btwAA5cqVw+DBg1GyZEmNFkdERESkLrWPlgoJCUG5cuVw7tw5ODs7w9nZGWfPnkX58uURGhqaGzUSERERZZvaIze//vorhg4dipkzZ2ZqHz16NH788UeNFUdERESkLrVHbm7duoVevXplau/Zsyf+/vtvjRRFRERElFNqhxsrKytcuXIlU/uVK1dgbW2tiZqIiIiIckzt3VJ9+vRB37598eDBA9SoUQMAcPLkScyaNQvDhg3TeIFERERE6lA73Pz2228oUKAA5s2bhzFjxgAAihQpgkmTJmHQoEEaL5CIiIhIHWqHG4VCgaFDh2Lo0KGIj48HABQoUEDjhRERERHlRI7OcwMA0dHRuHPnDgCgTJkysLKy0lhRRERERDml9oTi+Ph4dOvWDUWKFEHdunVRt25dFClSBF27dkVsbGxu1EhERESUbWqHm969e+Ps2bPYu3cvYmJiEBMTgz179uDChQvw9vbOjRqJiIiIsk3t3VJ79uxBSEgIatWqJbW5u7tj+fLlaNKkiUaLIyIiIlKX2iM3BQsWhJmZWaZ2MzMzWFhYaKQoIiIiopxSO9yMHz8ew4YNQ2RkpNQWGRmJkSNH4rffftNocURERETqUnu31NKlSxEeHo5ixYqhWLFiAIDHjx/D0NAQL168wLJly6S+ly5d0lylRERERNmgdrjx8PDIhTKIiIiINEPtcDNx4sTcqIOIiIhII9Sec/PkyRM8ffpUun/u3DkMGTIEgYGBGi2MiIiIKCfUDjedO3dGWFgYgPcTiRs1aoRz585h3LhxmDJlisYLJCIiIlKH2uHmxo0b+OGHHwAAW7ZsQcWKFXHq1CmsX78eQUFBmq6PiIiISC1qh5u0tDQYGhoCAA4ePIhWrVoBeH99qefPn2u2OiIiIiI1qR1uypcvj4CAABw/fhyhoaHSWYn/+ecfFCxYUOMFEhEREalD7XAza9YsLFu2DPXq1UOnTp1QqVIlAMDu3bul3VVERERE2qL2oeD16tXDy5cvERcXp3K5hb59+8LExESjxRERERGpS+2RGwAQQuDixYtYtmwZ4uPjAQAGBgYMN0RERKR1ao/cREREoEmTJnj8+DFSUlLw448/okCBApg1axZSUlIQEBCQG3USERERZYvaIzeDBw9G1apV8ebNGxgbG0vtbdq0waFDhzRaHBEREZG61B65OX78OE6dOgUDAwOVdgcHBzx79kxjhRERERHlhNojN0qlEunp6Znanz59igIFCmikKCIiIqKcUjvcNG7cGP7+/tJ9hUKBhIQETJw4Ec2aNdNkbURERERqU3u31Lx58+Du7o5y5cohOTkZnTt3xr1791CoUCFs3LgxN2okIiIiyja1R27s7Oxw9epVjBs3DkOHDoWrqytmzpyJy5cvw9raWq3HOnbsGFq2bIkiRYpAoVBg586d/9r/yJEjUCgUmW6RkZHqvgwiIiKSKbVHbgBAT08PXbp0QZcuXaS258+fY+TIkVi0aFG2HycxMRGVKlVCz5490bZt22yvd+fOHZiamkr31Q1VREREJF9qhZubN28iLCwMBgYGaN++PczNzfHy5UtMnz4dAQEBKFGihFpP3rRpUzRt2lStdYD3Ycbc3Fzt9YiIiEj+sr1bavfu3XB1dcWgQYPQr18/VK1aFWFhYShbtixu3bqFHTt24ObNm7lZq8TFxQWFCxfGjz/+iJMnT/5r35SUFMTFxanciIiISL6yHW6mTZsGHx8fxMXFYf78+Xjw4AEGDRqEv/76C/v375euDp6bChcujICAAGzfvh3bt2+Hvb096tWrh0uXLn1yHV9fX5iZmUk3e3v7XK+TiIiItCfb4ebOnTvw8fFB/vz5MXDgQOjo6MDPzw/ff/99btanonTp0vD29kaVKlVQo0YNrFq1CjVq1ICfn98n1xkzZgxiY2Ol25MnT75YvURERPTlZXvOTXx8vDSJV1dXF8bGxmrPsckNP/zwA06cOPHJ5YaGhjA0NPyCFREREZE2qTWhOCQkBGZmZgDen6n40KFDuHHjhkqfVq1aaa66bLhy5QoKFy78RZ+TiIiI8i61wo2Xl5fKfW9vb5X7CoUiy0szfEpCQgLCw8Ol+w8fPsSVK1dgaWmJYsWKYcyYMXj27BnWrl0LAPD394ejoyPKly+P5ORkrFixAocPH8aBAwfUeRlEREQkY9kON0qlUuNPfuHCBdSvX1+6P2zYMADvQ1RQUBCeP3+Ox48fS8tTU1MxfPhwPHv2DCYmJnB2dsbBgwdVHoOIiIi+bTk6iZ+m1KtXD0KITy4PCgpSuT9q1CiMGjUql6siIiKir5nal18gIiIiyssYboiIiEhWGG6IiIhIVhhuiIiISFZyFG5iYmKwYsUKjBkzBq9fvwYAXLp0Cc+ePdNocURERETqUvtoqWvXrqFRo0YwMzPDo0eP0KdPH1haWiI4OBiPHz+WzklDREREpA1qj9wMGzYM3bt3x71792BkZCS1N2vWDMeOHdNocURERETqUjvcnD9/PtOZiQGgaNGiiIyM1EhRRERERDmldrgxNDREXFxcpva7d+/CyspKI0URERER5ZTa4aZVq1aYMmUK0tLSALy/ntTjx48xevRo/PTTTxovkIiIiEgdaoebefPmISEhAdbW1khKSkLdunXh5OSEAgUKYPr06blRIxEREVG2qX20lJmZGUJDQ3HixAlcu3YNCQkJqFy5Mho1apQb9RERERGpJccXzqxVqxZq1aqlyVqIiIiIPpva4eb333/Psl2hUMDIyAhOTk6oU6cOdHV1P7s4IiIiInWpHW78/Pzw4sULvH37FhYWFgCAN2/ewMTEBPnz50d0dDRKlCiBsLAw2Nvba7xgIiIion+j9oTiGTNm4Pvvv8e9e/fw6tUrvHr1Cnfv3kW1atWwYMECPH78GLa2thg6dGhu1EtERET0r9QeuRk/fjy2b9+OkiVLSm1OTk6YO3cufvrpJzx48ACzZ8/mYeFERESkFWqP3Dx//hzv3r3L1P7u3TvpDMVFihRBfHz851dHREREpCa1w039+vXh7e2Ny5cvS22XL19G//790aBBAwDA9evX4ejoqLkqiYiIiLJJ7XCzcuVKWFpaokqVKjA0NIShoSGqVq0KS0tLrFy5EgCQP39+zJs3T+PFEhEREf0Xtefc2NraIjQ0FLdv38bdu3cBAKVLl0bp0qWlPvXr19dchURERERqyPFJ/MqUKYMyZcposhYiIiKiz5ajcPP06VPs3r0bjx8/Rmpqqsqy+fPna6QwIiIiopxQO9wcOnQIrVq1QokSJXD79m1UqFABjx49ghAClStXzo0aiYiIiLJN7QnFY8aMwYgRI3D9+nUYGRlh+/btePLkCerWrYt27drlRo1ERERE2aZ2uLl16xY8PT0BAHp6ekhKSkL+/PkxZcoUzJo1S+MFEhEREalD7XCTL18+aZ5N4cKFcf/+fWnZy5cvNVcZERERUQ6oPefGzc0NJ06cQNmyZdGsWTMMHz4c169fR3BwMNzc3HKjRiIiIqJsUzvczJ8/HwkJCQCAyZMnIyEhAZs3b0apUqV4pBQRERFpnVrhJj09HU+fPoWzszOA97uoAgICcqUwIiIiopxQa86Nrq4uGjdujDdv3uRWPURERESfRe0JxRUqVMCDBw9yoxYiIiKiz6Z2uJk2bRpGjBiBPXv24Pnz54iLi1O5EREREWmT2hOKmzVrBgBo1aoVFAqF1C6EgEKhQHp6uuaqIyIiIlKT2uEmLCwsN+ogIiIi0gi1w03dunVzow4iIiIijVB7zg0AHD9+HF27dkWNGjXw7NkzAMC6detw4sQJjRZHREREpC61w8327dvh7u4OY2NjXLp0CSkpKQCA2NhYzJgxQ+MFEhEREakjR0dLBQQEYPny5dDX15faa9asiUuXLmm0OCIiIiJ1qR1u7ty5gzp16mRqNzMzQ0xMjCZqIiIiIsoxtcONra0twsPDM7WfOHECJUqU0EhRRERERDmldrjp06cPBg8ejLNnz0KhUOCff/7B+vXrMWLECPTv3z83aiQiIiLKNrUPBf/111+hVCrRsGFDvH37FnXq1IGhoSFGjBiBgQMH5kaNRERERNmmdrhRKBQYN24cRo4cifDwcCQkJKBcuXLInz9/btRHREREpBa1d0v98ccfePv2LQwMDFCuXDn88MMPDDZERESUZ6gdboYOHQpra2t07twZf/3112ddS+rYsWNo2bIlihQpAoVCgZ07d/7nOkeOHEHlypVhaGgIJycnBAUF5fj5iYiISH7UDjfPnz/Hpk2boFAo0L59exQuXBg+Pj44deqU2k+emJiISpUqYfHixdnq//DhQzRv3hz169fHlStXMGTIEPTu3RshISFqPzcRERHJk9pzbvT09NCiRQu0aNECb9++xY4dO7BhwwbUr18fdnZ2uH//frYfq2nTpmjatGm2+wcEBMDR0RHz5s0DAJQtWxYnTpyAn58f3N3d1X0pREREJENqh5sPmZiYwN3dHW/evEFERARu3bqlqbqydPr0aTRq1Eilzd3dHUOGDPnkOikpKdIlIgAgLi4ut8ojIiKiPCBHF858+/Yt1q9fj2bNmqFo0aLw9/dHmzZtcPPmTU3XpyIyMhI2NjYqbTY2NoiLi0NSUlKW6/j6+sLMzEy62dvb52qNREREpF1qh5uOHTvC2toaQ4cORYkSJXDkyBGEh4dj6tSpKFOmTG7U+FnGjBmD2NhY6fbkyRNtl0RERES5SO3dUrq6utiyZQvc3d2hq6ursuzGjRuoUKGCxor7mK2tLaKiolTaoqKiYGpqCmNj4yzXMTQ0hKGhYa7VRERERHmL2uFm/fr1Kvfj4+OxceNGrFixAhcvXvysQ8P/S/Xq1fHXX3+ptIWGhqJ69eq59pxERET0dcnRnBvg/TlqvLy8ULhwYcydOxcNGjTAmTNn1HqMhIQEXLlyBVeuXAHw/lDvK1eu4PHjxwDe71Ly9PSU+vfr1w8PHjzAqFGjcPv2bSxZsgRbtmzB0KFDc/oyiIiISGbUGrmJjIxEUFAQVq5cibi4OLRv3x4pKSnYuXMnypUrp/aTX7hwAfXr15fuDxs2DADg5eWFoKAgPH/+XAo6AODo6Ii9e/di6NChWLBgAezs7LBixQoeBk5ERESSbIebli1b4tixY2jevDn8/f3RpEkT6OrqIiAgIMdPXq9ePQghPrk8q7MP16tXD5cvX87xcxIREZG8ZTvc7Nu3D4MGDUL//v1RqlSp3KyJiIiIKMeyPefmxIkTiI+PR5UqVVCtWjUsWrQIL1++zM3aiIiIiNSW7XDj5uaG5cuX4/nz5/D29samTZtQpEgRKJVKhIaGIj4+PjfrJCIiIsoWtY+WypcvH3r27IkTJ07g+vXrGD58OGbOnAlra2u0atUqN2okIiIiyrYcHwoOAKVLl8bs2bPx9OlTbNy4UVM1EREREeXYZ4WbDLq6uvDw8MDu3bs18XBEREREOaaRcENERESUVzDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGs5Ilws3jxYjg4OMDIyAjVqlXDuXPnPtk3KCgICoVC5WZkZPQFqyUiIqK8TOvhZvPmzRg2bBgmTpyIS5cuoVKlSnB3d0d0dPQn1zE1NcXz58+lW0RExBesmIiIiPIyrYeb+fPno0+fPujRowfKlSuHgIAAmJiYYNWqVZ9cR6FQwNbWVrrZ2Nh8wYqJiIgoL9NquElNTcXFixfRqFEjqU1HRweNGjXC6dOnP7leQkICihcvDnt7e7Ru3Ro3b978ZN+UlBTExcWp3IiIiEi+tBpuXr58ifT09EwjLzY2NoiMjMxyndKlS2PVqlXYtWsX/vjjDyiVStSoUQNPnz7Nsr+vry/MzMykm729vcZfBxEREeUdWt8tpa7q1avD09MTLi4uqFu3LoKDg2FlZYVly5Zl2X/MmDGIjY2Vbk+ePPnCFRMREdGXpKfNJy9UqBB0dXURFRWl0h4VFQVbW9tsPYa+vj5cXV0RHh6e5XJDQ0MYGhp+dq1ERET0ddDqyI2BgQGqVKmCQ4cOSW1KpRKHDh1C9erVs/UY6enpuH79OgoXLpxbZRIREdFXRKsjNwAwbNgweHl5oWrVqvjhhx/g7++PxMRE9OjRAwDg6emJokWLwtfXFwAwZcoUuLm5wcnJCTExMZgzZw4iIiLQu3dvbb4MIiIiyiO0Hm46dOiAFy9eYMKECYiMjISLiwv2798vTTJ+/PgxdHT+N8D05s0b9OnTB5GRkbCwsECVKlVw6tQplCtXTlsvgYiIiPIQrYcbABgwYAAGDBiQ5bIjR46o3Pfz84Ofn98XqIqIiIi+Rl/d0VJERERE/4bhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkheGGiIiIZIXhhoiIiGSF4YaIiIhkJU+Em8WLF8PBwQFGRkaoVq0azp0796/9t27dijJlysDIyAgVK1bEX3/99YUqJSIiorxO6+Fm8+bNGDZsGCZOnIhLly6hUqVKcHd3R3R0dJb9T506hU6dOqFXr164fPkyPDw84OHhgRs3bnzhyomIiCgv0nq4mT9/Pvr06YMePXqgXLlyCAgIgImJCVatWpVl/wULFqBJkyYYOXIkypYti6lTp6Jy5cpYtGjRF66ciIiI8iKthpvU1FRcvHgRjRo1ktp0dHTQqFEjnD59Ost1Tp8+rdIfANzd3T/Zn4iIiL4tetp88pcvXyI9PR02NjYq7TY2Nrh9+3aW60RGRmbZPzIyMsv+KSkpSElJke7HxsYCAOLi4j6n9E9SprzNlcelr0dubVvZxW2QuA2StuXGNpjxmEKI/+yr1XDzJfj6+mLy5MmZ2u3t7bVQDX0LzPy1XQF967gNkrbl5jYYHx8PMzOzf+2j1XBTqFAh6OrqIioqSqU9KioKtra2Wa5ja2urVv8xY8Zg2LBh0n2lUonXr1+jYMGCUCgUn/kK6ENxcXGwt7fHkydPYGpqqu1y6BvEbZC0jdtg7hFCID4+HkWKFPnPvloNNwYGBqhSpQoOHToEDw8PAO/Dx6FDhzBgwIAs16levToOHTqEIUOGSG2hoaGoXr16lv0NDQ1haGio0mZubq6J8ukTTE1N+UtNWsVtkLSN22Du+K8Rmwxa3y01bNgweHl5oWrVqvjhhx/g7++PxMRE9OjRAwDg6emJokWLwtfXFwAwePBg1K1bF/PmzUPz5s2xadMmXLhwAYGBgdp8GURERJRHaD3cdOjQAS9evMCECRMQGRkJFxcX7N+/X5o0/PjxY+jo/O+grho1amDDhg0YP348xo4di1KlSmHnzp2oUKGCtl4CERER5SEKkZ1px0TZkJKSAl9fX4wZMybTrkCiL4HbIGkbt8G8geGGiIiIZEXrZygmIiIi0iSGGyIiIpIVhhsiIiKSFYYbIiIikhWGGyIiIpIVhhsiIiKSFYYbIiIikhWGGyIiIpIVhhsiIiKSFYYb+mYolUptl0BERF8Aww19MzIuwPry5UsAAK88Ql/axwGb2yBpw8fboRy/+DHc0DdlwYIF8PDwwP3796FQKLRdDn1jdHR0EBsbi5CQEADgNkhaoaOjg5iYGMyZMwdv3ryRvvjJifxeEdEHPv5mrK+vD2NjYxgYGGipIvqWKZVKzJs3D97e3tizZ4+2y6Fv2IEDBzB//nwsWrRI26XkCl4VnL4JcXFxMDU1BQDExsbCzMxMyxXRt0KpVKp8M7516xZWrlyJWbNmQVdXV4uV0bckPT1dZXtLS0vD5s2b0alTJ1luhww3JHtDhw5Feno6xowZg8KFC2u7HPoGxcTEICYmBvb29iofJB9/4BB9jo+D9MdevXqFkydPokaNGihUqJDULsftkLulSHY+zut2dnZYu3at7H556esghMCvv/6KatWq4dGjRyrLuE3S53j+/Dn++ecfvHjxAsD7uTT/Nl6xZcsWeHh44OjRoyrtctwOOXJDX7WMbxxCCCgUik9+c3nz5g0sLCy0UCHJzX99O86qT0REBMaPH4+goCBZfpDQl7d69WosXrwYT548QcmSJVGrVi3Mnj1bpU9WIzL+/v4YMGAA9PT0vmS5XxzDDX01MgIM8P6XVggBPT09PHv2DDt27ECPHj2QL18+AO93RVlYWGDChAmZ1iXKqQ9Dy+HDh/H48WM4OTmhRIkSKFKkiEqf2NhYKJXKTKFajrsA6Mvas2cP2rdvjyVLlsDExAQPHjzA7NmzUaNGDaxZswYFCxaU/ua9fPkS4eHhcHNzU3mMd+/eyTrgcLcU5VkZuTsuLg5JSUlQKBQ4cOAAwsPDoaurCz09PURERMDV1RX//POPFGwSExOhr68PPz8/vH79msGGNEIIIQWbX3/9Fd27d8fcuXPRt29fjBgxAufPnwfwftdASkoKJkyYgMqVK+PVq1cqj8NgQ5/r/PnzaN68Obp374727dtj1KhRCAkJwbVr19ClSxcA708zkJaWhnXr1qFGjRo4ceKEymPIOdgADDeUx0VGRqJixYo4evQoNmzYgCZNmuDvv/8G8H5XU/ny5dGmTRtMnz5dWidfvnwYNWoU7t27B0tLSwYb0oiM7Wju3Ln4448/sHHjRty4cQNt27bFn3/+ifHjx+P06dMAAAMDA7i6uqJhw4YwNzfXYtUkRw8fPsTz589V2r7//nvs3r0bFy9eRJ8+fQC8P/VFixYtMH369EwjN7IniPK4Hj16CFNTU6GjoyOWL18utaemporNmzeL9PR0qU2pVGqjRPpGREVFibZt24pVq1YJIYTYvXu3MDU1Ff369ROurq6iYcOG4syZM0II1W3x3bt3WqmX5CkkJETY2NiITZs2SW0Z29v69euFk5OTOH/+fKb10tLSvliN2saRG8qzMk4J7uPjg/j4eBgYGMDW1hbJyckA3n8rad++vcrETY7SUG6ytrbGqFGj0KRJE1y+fBk+Pj6YNm0ali5dip9++glnzpyBj48PLl68qLItclcUaVLZsmVRr149rFu3DocOHQLwv799Li4uiI6Oli4z8yG574r6EMMN5VkZocXe3h4nTpyAl5cXOnbsiF27diEpKSlTfzleH4W051Pbk6urKwoXLox9+/bB2dkZffv2BQBYWlrCzc0NLVu2hKur65cslb4x9vb26NevH2JiYuDn54fdu3dLywoXLgxHR0ctVpc3fDsxjr4a4v8nAD9//hxpaWkoVqwYrK2tUaNGDSQnJ6NXr14ICgpCixYtYGRkhICAADRq1AhOTk7aLp1kQnwweXjFihWIjo6GgYEBRowYIV26IyUlBc+ePcOjR49QunRpHDhwAK1atcLAgQP/9bQERJ8j42i7evXqYcmSJRg7dixGjx6NkJAQODs7Y8uWLVAoFPjxxx+1XapW8VBwypOCg4MxadIkREVFoXnz5mjTpg1atmwJAOjRowd27NiB4cOHIyoqCkuXLsX169dRrlw5LVdNcjNx4kT4+/vj+++/x7lz51CtWjWsW7cOtra2+PPPPzFt2jS8efMG+vr6EELg2rVr0NPT4xF6lCsytqvg4GAsWbIEBw4cwO3btxEWFoZFixbB3t4e5ubmWL9+PfT19b/p0w4w3FCec/PmTbi7u2Po0KEwMTHBxo0bYWhoCC8vL3Tt2hUAMHjwYFy6dAkpKSkIDAyEi4uLdosmWfhwtOXdu3fw8vLCwIED4erqikePHqF58+awtbXFjh07YGVlhb179yI8PBwJCQkYPXo09PT0vukPFNKMjBAjPjq3l66uLoKDg+Hp6Yn58+dLu0SB99urjo6Oyvb7Lc2x+RjDDeUpt2/fxtatW5GUlIQZM2YAAK5fv44JEyYgLi4OPXr0kAJOZGQk8uXLhwIFCmizZJKJD4PNrVu3EBcXh2XLlmHChAlwcHAA8P4Q3B9//BE2NjbYuXMnrKysVB6DwYY+14fb4cuXL6FQKFCwYEEA7//mVa5cGRMmTEC/fv2kdT4eKeTIIcMN5RFCCLx58wYtWrTA33//jZYtW2LdunXS8mvXrmHChAlISkpCx44d0aNHDy1WS3I2cuRIaVg/KioKwcHBaNq0qfRh8fDhQzRt2hRCCJw8eVLlAoREn+PDUDJ16lTs3LkTcXFxKFSoEKZPn44GDRrg2bNnKFq0qJYrzfs4243yBIVCAUtLS/j6+qJ8+fK4dOkSQkNDpeXOzs6YOnUq0tLSpF94Ik348KioPXv2YP/+/fj999+xZMkSODo6Yty4cbh69ap0xmxHR0fs2bMHLi4uvF4ZaVRGsJkyZQoWLFggnWqgUKFC6NKlC9asWZNptJCyxpEb0ppPDZ0ePXoUY8eOha2tLXx8fNCgQQNp2c2bN2FmZgY7O7svWSp9A4KDg3Hq1CkULFgQY8aMAQAkJCSgcuXKMDU1xYoVK1CpUqVM2yx3RZEmvXr1Co0bN4aPjw969uwptfft2xd//vknwsLCUKZMGe56+g8cuSGtyPjFPHXqFObPn4/ffvsNJ0+eRFpaGurWrYspU6YgMjISixYtwpEjR6T1ypcvz2BDGpeUlITffvsN8+fPx82bN6X2/Pnz49KlS4iPj4e3t7d0/agPMdiQJr179w4vX76URgUzTloaGBiIIkWKwM/PDwBPWPpfGG7oi/vwcMamTZvi5MmT2L17N8aOHYvp06cjNTUVDRs2xJQpU/Dq1StMnToVx48f13bZJGPGxsY4fvw4GjVqhIsXL2L37t1IT08H8L+Ac/v2bSxbtkzLlZKcZLXjxMbGBra2tli1ahUAwMjICKmpqQAAJycnhppsYrihLy5jxGbQoEGYP38+tm/fjq1bt+LixYvYvHkzxo8fLwWcX3/9Ffr6+jzjJmnMh3NshBDSB4ylpSU2bNgACwsLzJkzByEhIdKyfPnyITIyEoGBgVqpmeRHqVRKQeWff/5BdHQ03r59CwCYNGkSbt++LR0RlXHiyKdPn/JCrNnEOTf0xWT8MisUCixZsgRXrlxBYGAgHj58iEaNGqFWrVowNTXF1q1b4e3tjbFjx8LQ0BBv376FiYmJtssnGfjwMNuFCxfi6tWrePDgAYYMGYLKlSvDzs4OL168QOvWraGrq4uxY8fC3d1d5UzDnGNDn2P9+vVwc3NDyZIlAQBjxoxBSEgIIiIi0KhRI7Rq1QpdunTB8uXLMXXqVBQsWBAVKlTA/fv3ERMTI50okv4dww3lmowPkg/DyZUrV+Di4oK4uDg8efIETk5OaNKkCRwdHbFq1SrExsZKZxru3r07pk+fzolz9Nk+3obGjBmDlStXom/fvnj69ClOnz6N1q1bo2/fvnBycsKLFy/Qtm1bvHjxAkFBQXBzc9Ni9SQX+/btQ4sWLTB69GgMGTIE+/btw6hRo+Dv749Xr17h0qVLCAkJwW+//YZ+/frh+vXr8Pf3h46ODiwsLDBjxgyeKDK7cvWa4/TNe/DggejUqZP4+++/xZYtW4RCoRDnzp0TSqVSCCHE9evXRZkyZcTZs2eFEELcv39ftGjRQowdO1Y8fvxYm6WTzKSnpwshhFi3bp1wdHQUFy9eFEIIcfz4caFQKESpUqXE4MGDxYMHD4QQQjx//lz07dtXvHv3Tms1k/wsWrRI2NnZialTp4oBAwaI5cuXS8uePHkipkyZIhwcHMT+/fuzXD8tLe1LlfpV49gW5ark5GQcP34c3bt3x5UrV7B69Wp8//330i4qIQTevXuH06dPo3z58li7di0AYMSIETyHCH22bt26wcrKCvPnz4eOjg7S0tJgYGCAfv36oXLlyti5cyd69OiBFStWIDIyEtOmTYOOjg769OmDsmXLShOI+U2ZPldqaioMDAzg4+MDExMTjBkzBvHx8Zg2bZrUx87ODp6enjhw4AAuXLgAd3f3TBdg5S6pbNJ2uiL5yvimHBAQIHR0dESlSpXE5cuXVfrExsaK7t27i5IlSwoHBwdhZWUlfaMm+hyxsbFi8uTJwtLSUkyaNElqf/bsmYiKihLPnz8XVatWFfPmzZP6FylSRBQuXFgsWLBACCGkEUYiTfH19RXR0dFi/fr1wsTERDRr1kzcvXtXpU+HDh1E27ZttVShPPBoKcoVQgjo6OhACIEiRYpg3rx5ePfuHcaPH48TJ05I/UxNTTF37lwsWbIEEydOxNmzZ1G5cmUtVk5yEB8fD1NTU/Tv3x/jx4+Hv78/Jk6cCAAoUqQIrK2t8fz5c7x580aaT/Ps2TM0btwYEyZMgI+PDwCeS4Q+n/hgWuuaNWswdepU3Lt3D507d4afnx8uXbqEgIAA3LlzBwAQFxeHhw8folixYtoqWRY4vkUaJ/5/8ubhw4dx9OhRDBkyBC1btkSjRo3Qvn17zJw5E2PHjkWNGjUAvL8wZuPGjbVcNcnFqFGjsGzZMty/fx9WVlbo2rUrhBCYOnUqAGDy5MkA3gcgXV1dnDx5EkIIzJw5EyYmJtLht9wVRZqQEZAPHTqEy5cvIzAwUPrb17dvX6SlpWHy5MnYv38/KleujMTERKSmpmL27NnaLPvrp81hI5KfjGH8bdu2CTMzMzFmzBhx/vx5afm1a9dEuXLlRIsWLcQff/whJk2aJBQKhXjy5Al3AZBGXL16VdSpU0eULl1avHjxQgghRHR0tJg3b54wNzcXEyZMkPoOGDBAlCxZUtjZ2Qk3NzeRmpoqhODuKNKsI0eOiIoVK4qCBQuKnTt3CiGESElJkZavXLlS5M+fX1SuXFmsXbtWmsTOycM5x0PBSePOnTuHJk2aYNasWejTp4/UHhcXB1NTU9y6dQt9+vRBUlISYmNjsWXLFu6KIo04ffo0Xrx4gXLlyqFDhw5ISEiQrtz94sULrFu3DlOnTpUuSAi8Pz2BQqFAxYoVoaOjg3fv3nHSJn0W8dGpBxISEjBnzhwEBgaiWrVq2LhxI4yNjZGWlgZ9fX0AwPz583Hq1Cls3boVCoWCI4efieGGNG7RokXYsWMHDh06hNjYWBw+fBh//PEHbt26hREjRqBnz56Ijo5GbGwszMzMYG1tre2SSSY8PT3xzz//4ODBg3j06BF+/vlnxMfHZwo406ZNw4ABAzBlyhSV9fmBQpq0ePFi2NnZoXXr1khKSsLcuXOxY8cO1KtXDzNmzICRkZFKwMkIRR+HI1IfJxSTxtna2uLixYvw9fXFzz//jNWrV8PIyAjNmzdH7969cffuXVhbW6NUqVIMNqRRixcvxtOnT7Fo0SI4ODhg48aNMDMzQ82aNfHy5UtYWVmhW7dumDBhAqZNm4aVK1eqrM9gQ5ry4sULHD58GL/88gv2798PY2NjDBs2DC1atMCpU6cwbtw4JCcnQ19fH+/evQMABhsN4sgNfZaMX8SEhATkz58fABAVFYWFCxdiy5YtaNCgAbp3744ffvgBUVFRaNWqFYKCglC+fHktV05ykzHq8vvvv+Py5cuYP38+LCwscPv2bXh6eiI2NlYawYmMjMTRo0fx008/cRcUacTH56MBgKtXr+L333/HwYMHERAQgKZNmyIxMRGzZ8/GwYMHUbZsWSxZskS6dhRpDkdu6LMoFArs3bsXnTp1Qr169RAUFAQ9PT1MmzYNZ8+eRUBAANzc3KCjo4OFCxciMTGRozWUKzJGXerVq4djx45h7969AIDSpUtj3bp1sLCwQJ06dRAVFQVbW1t06NABenp60rdmos+REWwiIyOltkqVKmHw4MGoX78++vXrh/379yNfvnwYNWoUfvjhB+jo6Ei7pEjDtDSRmWTi5MmTwsjISIwcOVI0adJEODs7C29vbxEeHi71CQsLE3379hWWlpaZTuJHlFMZJ4nMSkBAgPjuu+/EnTt3pLY7d+4IBwcH0bFjxy9RHn0jPtwON23aJEqUKKFyhKgQQly5ckW0bt1aFCtWTBw5ckQIIURSUpJ0VN6/bcuUMxy5oRyLiIhAaGgopk+fjtmzZ2Pfvn3o27cvrl27Bl9fXzx48ACJiYk4ffo0oqOjcfToUbi4uGi7bJKBD3cBnDt3DqdOncLRo0el5a1atUK1atUQFhYmtX333Xc4duwY/vjjjy9eL8lTSkqKtB2mpqaiZMmSKFOmDHx8fHDx4kWpX6VKleDh4YEnT56gcePGOHXqFIyMjKQ5Nh/vzqLPx58oZcuiRYvw119/Sffv3LmDDh06YNWqVTAyMpLafXx80KVLF9y8eROzZ89GTEwMRo4ciTVr1qBChQraKJ1k5sMPg7Fjx6J79+7o2bMnvLy80KFDB8TFxaFw4cLSfIa0tDRpXXt7e+jq6iI9PV1b5ZNM7Nu3D+vWrQMA9OnTBw0aNEDVqlUxfPhw2NrawtvbGxcuXJD6FytWDB07dsS8efNQrVo1qZ2Th3OJtoeOKO97+PCh6Ny5s7h3755K+6+//iqsra1F27ZtpZOlZVi6dKkoXbq0GDRoEE9ERbli7ty5omDBguLs2bMiPT1dzJgxQygUCnHixAmpT82aNYW3t7cWqyS56tSpk3BwcBDu7u6iUKFC4urVq9Kyw4cPCw8PD1GhQgWxb98+8fDhQ+Hh4SGGDx8u9eHV5nMXww1lS2JiohBCiDNnzoht27ZJ7RMmTBAVK1YU48ePF1FRUSrrLF++XDx8+PBLlknfCKVSKby8vERgYKAQQojt27cLc3NzERAQIIQQIj4+XgghxL59+0SrVq3EtWvXtFYryZeLi4tQKBQqF2bNcPz4cdGtWzehUCjEd999J5ydnaUvejwDdu7jMZCULcbGxoiJiYGvry+ePXsGXV1deHh4YPLkyUhLS8PevXshhMDgwYNhZWUFAOjdu7eWqya5Sk5OxtmzZ1GvXj0cOXIEXl5emDNnDry9vfHu3TvMnj0b1atXh5ubG6ZMmYJz586hYsWK2i6bZCI1NRXJyclwcnJCsWLFsHnzZhQtWhQdO3aUTolRq1YtVKtWDX369EFaWhrq1q0LXV1dngH7C+GcG8oWhUIBc3NzDB8+HI6OjvD390dwcDAAYMaMGWjSpAlCQ0MxY8YMvHz5UsvVkpxcu3YNT58+BQAMHToUR48ehbGxMTp37ow//vgDzZo1g5+fn3TByzdv3uDChQu4c+cOLCwssG7dOhQvXlybL4FkxsDAAKampti6dSt27dqF77//HrNnz8amTZsQHx8v9UtOTkbt2rXRoEEDaa4Xg82XwXBD2SLe78JE7dq1MXToUFhYWOD3339XCThubm64fPkyBM8LSRoghMDdu3dRv359rFq1Cv369cOCBQtgYWEBAHBzc0NERASqVauG6tWrAwD++ecfdO/eHTExMRgwYAAAoGTJkmjUqJHWXgfJjxACSqVSur9mzRrUqFEDfn5+WLt2LR4/fowGDRqgXbt2Un+AZ8D+kniGYsqWjLO/xsbGwsTEBNeuXcP06dPx5s0bDB48GB4eHgDen3I8Y7cUkSYsX74co0aNQnJyMnbt2oXGjRtLZ8bevHkzpkyZAiEE9PT0YGxsDKVSiVOnTkFfX5/XiqLP9vr1a1haWqq0ZWx/W7duRWhoKAIDAwEAffv2xZEjR5Ceng5LS0ucPHmSZx/WEo7c0H969+4ddHV18ejRI9SrVw8HDhxAlSpVMGLECFhZWWHy5MnYs2cPADDYkMZkfDO2t7eHoaEhTE1NcebMGTx69Eg6fLZDhw5Yu3YtpkyZgvbt22P06NE4c+aMdL0eBhv6HAsWLMD333+vsqsJgBRsunfvjkqVKkntgYGBWLZsGRYuXIgzZ87AwMCAZ8DWFu3MY6a86lOz+MPDw4WNjY3o3bu3yiGMR44cEd26dROPHj36UiWSzH28DaampoqkpCSxdOlSUbRoUTF27Nj/3N54mC19rmXLlglDQ0OxYcOGTMseP34sKlasKBYtWiS1ZbXNcTvUHu6WIon4/6HW06dP49atWwgPD4enpycKFy6MNWvW4MKFC1izZk2mK9cmJyernMiPKKc+PPPw69evER8frzIZ2N/fH3PnzkWvXr3Qo0cPODg4oGXLlhg3bhzc3Ny0VTbJzPLlyzFw4ECsW7cO7dq1Q0xMDBITE5GcnAxra2sUKFAA9+7dQ6lSpbRdKn0Cww2p2L59O/r27StdYPDFixfo0KEDRo8ejQIFCmi7PJKxD4PNlClTcODAAdy4cQPt27dHmzZt0LRpUwDvA46/vz8qVKiAV69e4fHjx3j06BEvQEga8eDBAzg5OaF9+/bYtGkTbty4gV9++QUvXrxAREQE6tevj/79+6NFixbaLpX+BY9JI8mNGzcwdOhQzJs3D927d0dcXBzMzc1hbGzMYEO5LiPYTJgwAYGBgZgzZw4cHBzQr18/3Lt3DzExMejUqROGDBmCQoUK4erVq0hOTsbx48elq3vzMFv6XFZWVpg1axYmTJiAESNG4MCBA6hduzZat26NuLg4bNu2DePHj0ehQoU4WpiXaXOfGGnP4cOHxf379zO1Va9eXQghxK1bt0Tx4sVF7969peX379/nPmTKVYcPHxbly5cXx44dE0IIcerUKWFgYCDKlSsnqlWrJrZu3Sr1/fCyHrzEB2lScnKymDt3rtDR0RE9e/YUqamp0rILFy6I0qVLi8WLF2uxQvovPFrqGyOEwOXLl9G0aVMsXboUERER0rJnz55BCIGEhAQ0adIEjRs3xrJlywAAoaGhWLp0Kd68eaOt0kmGxEd7xYsWLYr+/fujdu3aOHDgAFq0aIHAwECEhobi/v37+P3337Fy5UoAUBml4YgNaZKhoSH69euH7du3o3fv3tDX15e21SpVqsDIyAhPnjzRcpX0bxhuvjEKhQKurq6YN28etmzZgqVLl+LBgwcAgObNmyMqKgqmpqZo3rw5AgMDpV0FISEhuHbtGg+tJY1RKpXSpPQHDx4gMTERpUqVQqdOnZCcnIwFCxZg0KBB6NatG4oUKYLy5csjPDwct27d0nLl9C3Ily8fmjZtKp0gMmNbjY6OhrGxMcqXL6/N8ug/8OvONyZjXoKPjw8AYM6cOdDV1UXv3r3h6OiI3377DTNmzMC7d+/w9u1bhIeHY+PGjVixYgVOnDghnR2W6HN8OHl4woQJOH36NEaOHIn69evD0tISiYmJeP78OUxMTKCjo4OUlBQ4ODhg1KhRaNKkiZarJzkSHxwBmsHQ0FD6f3p6Ol6+fIk+ffpAoVCgU6dOX7pEUgPDzTcmY+TlwIED0NHRQVpaGvz9/ZGcnIzRo0ejffv2SEpKwowZM7Bt2zbY2NjAwMAAYWFhqFChgparJ7n4MNgsW7YMgYGBcHV1lY54SklJgaWlJU6cOCFNGn716hVWrVoFHR0dlXBElBMRERF4/fo1ChYsCFtb2389k3BaWhrWrVuHjRs34vXr1zhz5ox0rSiOZudNPBT8GxQSEiJdbDBfvny4d+8efv/9d/zyyy8YPXo0rKysEB8fj6NHj8LBwQHW1tawtrbWdtn0lfs4kNy9exceHh6YNWsWWrZsmanf+fPnMX78eCQkJMDS0hLBwcHQ19dnsKHPtnbtWsybNw/R0dEoVKgQBg4cKI3IZPh4OwsNDcXNmzcxYMAAHp33FWC4+cYolUp06dIFCoUCGzZskNoXLlyIUaNGwcfHB7/88gtKlCihxSpJbtq2bYuxY8eiatWqUtuVK1fQpEkTHD16FKVLl87yxJDJyckQQsDIyAgKhYIfKPTZ1q5dCx8fH+nSCjNmzMCDBw9w8uRJadvKCDYxMTE4cOAA2rdvr/IYHLHJ+/j15xuT8U0kY/g/NTUVADBw4EB4e3tj9erV+P3331WOoiL6XGZmZnB2dlZpMzIywps3b3Djxg2pLeN6UqdPn8b27duho6MDY2NjKBQKKJVKBhv6LBcuXMDUqVOxaNEi9OzZExUrVsTQoUPh5OSEU6dO4ebNm4iLi5N22a9Zswa//PIL/vjjD5XHYbDJ+xhuvhH//POP9P/SpUvjzz//RHR0NAwMDJCWlgYAsLOzg4mJCcLCwmBsbKytUklGnj17BgBYvXo1DAwM8Pvvv+PAgQNITU2Fk5MTOnTogDlz5uDgwYNQKBTQ0dFBeno6pk+fjrCwMJV5ENwVRZ8rJSUFQ4YMQfPmzaW2SZMm4dChQ+jUqRM8PT3RsWNHvH79Gvr6+mjWrBlGjBjBycNfIe6W+gZcvXoVAwYMQOfOndG/f3+kpqaiQYMGePnyJY4cOQJbW1sAwOjRo1G+fHm0aNEClpaWWq6avnZ9+vQBAIwZM0bazens7IyXL19i06ZNqFOnDo4fPw4/Pz9cv34dXbp0gYGBAQ4dOoQXL17g0qVLHKkhjVIqlXjx4gVsbGwAAJ6enjh48CB2794Ne3t7HD16FNOmTcPo0aPRuXNnlTk43BX1deFXoW+AiYkJzM3NsW3bNgQFBcHAwADLli2DlZUVypYtCw8PDzRu3BgLFixA1apVGWxII5ydnbF//34sXboU4eHhAIBr166hdOnS6NKlC44dO4batWtjypQp8PT0xLp163D48GEUK1YMFy9elCZtEmmKjo6OFGwAYMSIETh79iyqVq0KGxsbNG3aFK9fv0ZUVFSmw8IZbL4uHLn5RoSHh2Ps2LGIjIxEnz590K1bN6Snp2Pu3LmIiIiAEAIDBw5EuXLltF0qyciqVaswYcIEdOzYEX369EHp0qUBAHXq1MHDhw+xfv161KlTBwDw9u1bmJiYSOty8jB9aU+fPkXXrl0xYsQIXhjzK8dwI1OXLl3C8+fPVfYth4eHY/z48Xj06BEGDhyILl26aLFCkrMPD6NduXIlJkyYgE6dOmUKOBEREVi7di2qV6+uMr8mqxOqEanjw20o4/8Z/7548QJWVlYq/RMTE9GpUyfExsbi8OHDHKn5yjHcyFB8fDyaN28OXV1djBo1Ck2bNpWWPXr0CE2aNIGJiQl69+6NX375RYuVktx86hw0y5cvx+TJk9GhQwf07dtXCjgNGjTAyZMncebMGbi6un7pckmmstoOM9qCg4OxceNGLFiwAEWKFEFSUhJ27dqFdevW4dmzZzh//jz09fU5x+Yrxzk3MpKRUwsUKIDZs2dDT08PixYtwt69e6U+Dg4OqF+/PiIjI3Ho0CHExMRoqVqSmw8/UE6dOoWwsDBcvXoVwPvJxb/99hs2bdqEwMBA3LlzBwBw+PBh9O7dO9Nh4kQ5deLECemilsOGDcPMmTMBvJ9vs3nzZnh6eqJRo0YoUqQIgPcXXX348CFKlCiBCxcuQF9fH+/evWOw+cpx5EYGMoZaM75pZHzInD17Fr/++ivy5cuH/v37S7uohg8fjhIlSqBt27YoXLiwlqsnOfhwF8CwYcOwefNmJCQkwM7ODsWKFcO+ffsAAMuWLcO0adPQsWNHeHl5qVzSg9+U6XMIIRAbGwtra2s0bdoUhQoVQnBwMI4fP44KFSogJiYGbm5u8PHxwcCBA6V1PvzbCXA7lAuGm69cxi9nWFgYdu/ejdevX6NWrVpo164dzM3NcebMGfz2229ISUlBiRIlYGJigs2bN+Pq1auws7PTdvkkAx8GmwMHDmDIkCEIDAyEubk5/v77b0ycOBH58uXDhQsXALyfg+Pt7Q1/f38MGDBAm6WTDEVHR6NEiRJIT0/H9u3b0axZM2lZVnNtspqbQ18/7pb6yikUCuzYsQMtW7bE27dv8fbtW6xbtw79+/fH69ev4ebmhrlz56Ju3boIDw/HgwcPcPjwYQYb0piMD4Pdu3dj06ZNaNSoEWrVqoUKFSrg559/xtq1a5GQkID+/fsDAHr16oVdu3ZJ94k0JSUlBZGRkTAxMYGuri5WrVolnYYAAAoVKiT9P+Ns2B+GGQYb+eDIzVfuwoUL6NixI3799Vf07t0bERERqFy5MoyNjeHi4oK1a9fC0tJSulbPx4fbEmnC69ev0aJFC1y9ehX169fHnj17VJaPHTsWJ0+exF9//YV8+fJJ7dwFQJ/rU5PYHz16BGdnZ9SvXx/z589HyZIltVAdaQtHbr4ivr6+GDdunPSNA3h/ens3Nzf07t0bjx49QsOGDeHh4YHx48fj/Pnz+OWXX/D69WsYGRkBAIMNacSH2yAAWFpaYs2aNfjxxx9x+fJlrF69WmV5qVKl8OrVKyQlJam0M9jQ5/gw2Bw5cgQbNmzA1atX8ezZMzg4OODkyZMICwvDqFGjpEnsbdq0wcKFC7VZNn0BHLn5iixcuBCDBw/GjBkzMGrUKOmX+tatWyhdujRat24tfcgolUq4uLggPDwczZs3x+bNm3ltHtKIDz9Q7t+/D4VCARMTE9ja2uLhw4fw8fFBYmIi2rVrB29vb0RFRcHLywtGRkbYs2cPh/5J40aMGIE1a9ZAT08P+fPnh62tLfz8/FC1alVcv34d9evXh4ODA1JTU/Hu3TtcvXpVungwyZSgr4JSqRRCCLF8+XKho6Mjpk6dKtLS0qTlT548EWXLlhV79uwRQgjx+vVr0alTJ7Fw4ULx9OlTrdRM8pOxHQohxMSJE0XFihVFmTJlROHChUVgYKAQQojw8HDRrFkzYWRkJEqXLi3atGkj3N3dRVJSkhBCiPT0dK3UTvLx4XYYGhoqKlWqJI4fPy5ev34tdu3aJdq0aSOcnJzEpUuXhBBC3Lt3T0yZMkVMnz5d+rv54d9Pkh+Gm6+AUqmUfpmVSqX4448/hI6Ojpg2bZr0QREdHS1cXFyEt7e3ePTokRg7dqz4/vvvRVRUlDZLJ5maMmWKsLKyEiEhISIhIUG0adNGmJubi5s3bwohhHjw4IFo3ry5cHFxEX5+ftJ6ycnJWqqY5GjNmjViwIABom/fvirt58+fF02aNBFeXl4iISFBCKEaiBhs5I/7Kb4SCoUCBw8exPDhw1GlShXpmj0zZ86EEAIWFhbo0qULjh49Cjc3N6xduxYBAQGwtrbWdukkAx/OsVEqlTh37hz8/PzQuHFjhIaG4siRI5gxYwbKlSuHtLQ0ODo6Yt68ebCxscHevXsRHBwMADA0NNTWSyAZEB/Noti5cycWL16MK1euICUlRWqvWrUqateujRMnTiA9PR2A6pFQvGbZN0Db6YqyZ/v27cLY2FhMnTpVnD9/XgghRGBgoLSLSgghUlJSxM2bN0VoaKh48uSJNsslmZowYYKYOXOmKFq0qLhz544ICwsT+fPnF0uXLhVCCPH27Vsxbtw48ejRIyGEEHfv3hUtWrQQVatWFcHBwdosnb5yH468rF+/Xqxdu1YIIcSAAQOEubm5WLx4sYiNjZX6hISEiDJlykjbIn1bGG6+Anfu3BGOjo5iyZIlmZYtW7ZM2kVFpGkfzo/ZtGmTsLe3Fzdu3BBdu3YV7u7uwsTERKxcuVLq8+zZM1G7dm2xdu1aad1bt26Jn3/+WURERHzx+kkePtwOb9y4IVxdXUWlSpXErl27hBBCeHl5iVKlSonp06eL8PBwER4eLho2bCjq1q2rEoro28Gxua/A48ePoa+vr3KmzYwjVvr27Yt8+fKhW7duMDQ0xIgRI7RYKclNxlFRR48exZEjRzB8+HCUL19eOjlkw4YN0bNnTwDvL9jau3dv6OrqonPnztDR0YFSqUSZMmWwYcMGHp1COZaxHY4cORIPHz6EsbExbt++jaFDh+Ldu3cICgpCz549MX78eCxcuBA1a9ZE/vz5sXnzZigUik+eC4fki+HmK5CQkKByfhClUintPz5y5AiqVKmCzZs3q1ynh0hTIiMj0atXL0RHR2Ps2LEAgH79+uH+/fs4fPgwXF1dUapUKTx+/BjJyck4f/48dHV1VU7QxzkO9LmCgoKwYsUKHDp0CI6OjkhJSYGXlxd8fX2ho6ODVatWwcTEBFu2bEGTJk3QsWNHGBoaIjU1FQYGBtoun74wRtmvQKVKlfDy5UsEBgYCeP8tJiPc7Nq1Cxs2bEDbtm1RtmxZbZZJMmVra4vg4GDY2Njgzz//xMWLF6Grq4s5c+ZgypQpaNCgAWxtbdGhQ4dPXlWZ57ahzxUeHo4KFSrAxcUFZmZmsLW1xapVq6Crq4uhQ4dix44dWLRoERo1aoT58+dj9+7diI+PZ7D5RvHr1FfA0dERixYtQr9+/ZCWlgZPT0/o6uoiKCgIQUFBOH36NM/0SrnK2dkZ27dvh5eXFwICAjBw4EA4OzujVatWaNWqlUrf9PR0jtSQxoj/v5iloaEhkpOTkZqaCiMjI6SlpaFo0aLw9fVFixYt4O/vD2NjY2zYsAGdO3fGiBEjoKenh/bt22v7JZAW8AzFXwmlUont27fD29sb+fLlg5GREXR1dbFx40a4urpquzz6Rly+fBm9e/dGlSpVMHjwYJQvX17bJdE34vr163B1dcVvv/2GiRMnSu0hISFYvnw53rx5g/T0dBw5cgQA0KNHD/z2228oUaKEliombWK4+cr8888/iIiIgEKhgKOjI2xsbLRdEn1jLl++DG9vbxQvXhyzZ8+Go6Ojtkuib0RQUBD69u2LIUOGoEOHDrCwsMCgQYNQo0YNtGnTBuXLl8fevXvRtGlTbZdKWsZwQ0RqO3fuHAICArBixQoehUJf1Pbt2/HLL7/AwMAAQghYW1vj1KlTiIqKwo8//oht27bB2dlZ22WSljHcEFGOZMyF4GG29KU9e/YMT548QVpaGmrWrAkdHR2MGTMGO3fuRFhYGGxtbbVdImkZww0R5VhGwCHSlps3b2LWrFn466+/cPDgQbi4uGi7JMoDeEgDEeUYgw1p07t375Camgpra2scPXqUE9xJwpEbIiL6qqWlpfEM2KSC4YaIiIhkhbMAiYiISFYYboiIiEhWGG6IiIhIVhhuiIiISFYYbohI9o4cOQKFQoGYmJhsr+Pg4AB/f/9cq4mIcg/DDRFpXffu3aFQKNCvX79My3x8fKBQKNC9e/cvXxgRfZUYbogoT7C3t8emTZuQlJQktSUnJ2PDhg0oVqyYFisjoq8Nww0R5QmVK1eGvb09goODpbbg4GAUK1YMrq6uUltKSgoGDRoEa2trGBkZoVatWjh//rzKY/3111/47rvvYGxsjPr16+PRo0eZnu/EiROoXbs2jI2NYW9vj0GDBiExMTHXXh8RfTkMN0SUZ/Ts2ROrV6+W7q9atQo9evRQ6TNq1Chs374da9aswaVLl+Dk5AR3d3e8fv0aAPDkyRO0bdsWLVu2xJUrV9C7d2/8+uuvKo9x//59NGnSBD/99BOuXbuGzZs348SJExgwYEDuv0giynUMN0SUZ3Tt2hUnTpxAREQEIiIicPLkSXTt2lVanpiYiKVLl2LOnDlo2rQpypUrh+XLl8PY2BgrV64EACxduhQlS5bEvHnzULp0aXTp0iXTfB1fX1906dIFQ4YMQalSpVCjRg38/vvvWLt2LZKTk7/kSyaiXMALZxJRnmFlZYXmzZsjKCgIQgg0b94chQoVkpbfv38faWlpqFmzptSmr6+PH374Abdu3QIA3Lp1C9WqVVN53OrVq6vcv3r1Kq5du4b169dLbUIIKJVKPHz4EGXLls2Nl0dEXwjDDRHlKT179pR2Dy1evDhXniMhIQHe3t4YNGhQpmWcvEz09WO4IaI8pUmTJkhNTYVCoYC7u7vKspIlS8LAwAAnT55E8eLFAby/IvT58+cxZMgQAEDZsmWxe/dulfXOnDmjcr9y5cr4+++/4eTklHsvhIi0hnNuiChP0dXVxa1bt/D3339DV1dXZVm+fPnQv39/jBw5Evv378fff/+NPn364O3bt+jVqxcAoF+/frh37x5GjhyJO3fuYMOGDQgKClJ5nNGjR+PUqVMYMGAArly5gnv37mHXrl2cUEwkEww3RJTnmJqawtTUNMtlM2fOxE8//YRu3bqhcuXKCA8PR0hICCwsLAC83620fft27Ny5E5UqVUJAQABmzJih8hjOzs44evQo7t69i9q1a8PV1RUTJkxAkSJFcv21EVHuUwghhLaLICIiItIUjtwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGsMNwQERGRrDDcEBERkaww3BAREZGs/B+XLE52CERTBAAAAABJRU5ErkJggg==",
- "text/plain": [
- "
\n"
- ]
- },
- "metadata": {},
- "execution_count": 6
- }
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "input_texts = data['Input'].values"
- ],
- "metadata": {
- "id": "0DbL-kirWUyn"
- },
- "execution_count": 7,
- "outputs": []
- },
- {
- "cell_type": "code",
- "source": [
- "messages = [[{\"role\": \"user\", \"content\": input_text}] for input_text in input_texts]"
- ],
- "metadata": {
- "id": "cqpAvy8hWXyC"
- },
- "execution_count": 8,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "source": [
- "# Running Inference"
- ],
- "metadata": {
- "id": "SugCyom0Xy8U"
- }
- },
- {
- "cell_type": "code",
- "source": [
- "from litellm import batch_completion\n",
- "model_name = \"facebook/opt-125m\"\n",
- "provider = \"vllm\"\n",
- "response_list = batch_completion(\n",
- " model=model_name,\n",
- " custom_llm_provider=provider, # can easily switch to huggingface, replicate, together ai, sagemaker, etc.\n",
- " messages=messages,\n",
- " temperature=0.2,\n",
- " max_tokens=80,\n",
- " )"
- ],
- "metadata": {
- "id": "qpikx3uxHns3"
- },
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "code",
- "source": [
- "response_list"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "QDPikHtwKJJ2",
- "outputId": "06f47c44-e258-452a-f9db-232a5b6d2810"
- },
- "execution_count": 10,
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/plain": [
- "[ JSON: {\n",
- " \"choices\": [\n",
- " {\n",
- " \"finish_reason\": \"stop\",\n",
- " \"index\": 0,\n",
- " \"message\": {\n",
- " \"content\": \".\\n\\nThe query input is the query input that is used to query the data.\\n\\nThe query input is the query input that is used to query the data.\\n\\nThe query input is the query input that is used to query the data.\\n\\nThe query input is the query input that is used to query the data.\\n\\nThe query input is the query input that is\",\n",
- " \"role\": \"assistant\",\n",
- " \"logprobs\": null\n",
- " }\n",
- " }\n",
- " ],\n",
- " \"created\": 1694053363.6139505,\n",
- " \"model\": \"facebook/opt-125m\",\n",
- " \"usage\": {\n",
- " \"prompt_tokens\": 9,\n",
- " \"completion_tokens\": 80,\n",
- " \"total_tokens\": 89\n",
- " }\n",
- " }]"
- ]
- },
- "metadata": {},
- "execution_count": 10
- }
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "response_values = [response['choices'][0]['message']['content'] for response in response_list]"
- ],
- "metadata": {
- "id": "SYqTcCiJbQDF"
- },
- "execution_count": 11,
- "outputs": []
- },
- {
- "cell_type": "code",
- "source": [
- "response_values"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "wqs-Oy9FbiPo",
- "outputId": "16a6a7b7-97c8-4b5b-eff8-09ea5eb5ad06"
- },
- "execution_count": 12,
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/plain": [
- "['.\\n\\nThe query input is the query input that is used to query the data.\\n\\nThe query input is the query input that is used to query the data.\\n\\nThe query input is the query input that is used to query the data.\\n\\nThe query input is the query input that is used to query the data.\\n\\nThe query input is the query input that is']"
- ]
- },
- "metadata": {},
- "execution_count": 12
- }
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "data[f\"{model_name}_output\"] = response_values"
- ],
- "metadata": {
- "id": "mElNbBehbkrz"
- },
- "execution_count": 13,
- "outputs": []
- },
- {
- "cell_type": "code",
- "source": [
- "data.to_csv('model_responses.csv', index=False)"
- ],
- "metadata": {
- "id": "F06NXssDc45k"
- },
- "execution_count": 14,
- "outputs": []
- }
- ]
-}
\ No newline at end of file
diff --git a/cookbook/ai_coding_tool_guides/chatgpt_subscription_passthrough.md b/cookbook/ai_coding_tool_guides/chatgpt_subscription_passthrough.md
deleted file mode 100644
index 538a5f8a404..00000000000
--- a/cookbook/ai_coding_tool_guides/chatgpt_subscription_passthrough.md
+++ /dev/null
@@ -1,114 +0,0 @@
-# Use your ChatGPT subscription through LiteLLM
-
-Sign into ChatGPT in Codex or Pi and send model requests through your existing LiteLLM proxy. The client keeps your login and refreshes it. Each request uses that client's ChatGPT account, while a separate LiteLLM virtual key controls gateway access and usage attribution.
-
-An administrator adds a `chatgpt/` model using the usual model configuration. No ChatGPT login needs to be added to the proxy. This uses the same separate-key convention as provider passthrough, including Claude Code subscription forwarding.
-
-## Add a model to the proxy
-
-Reuse your existing proxy. If you need one, follow the [proxy quickstart](https://docs.litellm.ai/docs/proxy/docker_quick_start). Merge this entry into its model list and restart the proxy:
-
-```yaml
-model_list:
- - model_name: gpt-6-luna
- litellm_params:
- model: chatgpt/gpt-6-luna
- model_info:
- mode: responses
-```
-
-Use a model available to the caller's subscription. The virtual key must allow the exposed model name. Leave saved ChatGPT logins and API credentials unset for a client-only deployment. Explicit cross-provider fallbacks can incur separate API charges; do not configure them for subscription-only traffic.
-
-## Configure Codex
-
-Use your existing LiteLLM URL and a virtual key assigned to your gateway user. Export that key as `LITELLM_API_KEY` in the environment that launches Codex. Keep the value out of committed configuration.
-
-Add this provider to `~/.codex/config.toml`, preserving your other settings:
-
-```toml
-model = "gpt-6-luna"
-model_provider = "litellm_subscription"
-
-[model_providers.litellm_subscription]
-name = "LiteLLM subscription"
-base_url = "https://litellm.example.com/v1"
-wire_api = "responses"
-supports_websockets = false
-requires_openai_auth = true
-env_http_headers = { "x-litellm-api-key" = "LITELLM_API_KEY" }
-```
-
-Replace `https://litellm.example.com` with your proxy URL. A local proxy normally uses `http://localhost:4000/v1`. Launch Codex, sign into ChatGPT if needed, and send a prompt with a model available to your subscription. The selected model must match the name exposed by the gateway.
-
-The `lite` CLI can set up this provider each time it starts Codex, with no `config.toml` edit. Export `LITELLM_PROXY_URL` and `LITELLM_PROXY_API_KEY`, then run `lite codex --subscription`. Run `lite config set codex_subscription true` to make it the default.
-
-For a managed workspace rollout, use OpenAI's [gateway configuration guide](https://learn.chatgpt.com/docs/enterprise/sign-in-with-chatgpt-through-a-gateway).
-
-## Native compatibility and Pi
-
-For a native Codex model catalog and request format, change Codex's `base_url` to `https://litellm.example.com/backend-api/codex`. This route uses the same credentials. A ChatGPT model keeps its upstream name and needs no deployment of its own.
-
-### Mix ChatGPT and LiteLLM models
-
-On the native route, Codex's model picker shows OpenAI's catalog first and then the other chat and responses models the virtual key can use. Pick one of those and Codex's requests go through LiteLLM's `/v1/responses` with the key's usual routing, budgets, guardrails, and logging. The proxy removes the ChatGPT bearer and account header from that request, so the other provider and your callbacks never receive them. A name is routed this way when another provider serves it as an exact `model_name`, through a router `model_group_alias`, or as a team's public model name. A ChatGPT model, a name that only a wildcard deployment matches, or a name OpenAI's catalog lists for the caller's ChatGPT account keeps the passthrough, so a model OpenAI adds later still uses the subscription even when a deployment of another provider has the same name. The proxy learns the account's list from the catalog Codex fetches, or asks OpenAI with the caller's login. When it cannot get the list, for example because OpenAI is down, the request times out, or the client is not Codex, every request keeps the passthrough for the next minute, as it did before LiteLLM models were added to the picker. When the key can use many models, the picker lists the ones that fit in Codex's catalog size limit, and the proxy logs a warning with how many it left out.
-
-Codex's picker sends a `none` reasoning effort for a LiteLLM model whose catalog entry lists no effort levels. With `litellm_settings: drop_params: true`, the proxy drops it for a model that does not support `none`, and the provider applies its own default. A model whose cost map entry or `model_info` sets `supports_none_reasoning_effort: true` keeps it. Without `drop_params`, the effort reaches the provider unchanged.
-
-Codex also sends its ChatGPT app tools with every turn, which on a LiteLLM model adds a few hundred thousand prompt tokens to each turn. Set `apps = false` under `[features]` in Codex's `config.toml` to turn them off.
-
-Only a key that may call `/v1/responses` gets these models. A key whose `allowed_routes` covers only the native routes sees OpenAI's catalog unchanged, and every request it sends uses the subscription.
-
-Codex sends earlier reasoning items back with each turn. The proxy tags the reasoning that a LiteLLM model returns on this route, OpenAI and Azure deployments included. When a session switches to a ChatGPT model, the proxy drops those items before forwarding the request to OpenAI, which would reject them. Messages, tool calls, and ChatGPT's own reasoning are kept, and a LiteLLM model still gets its own reasoning back.
-
-When the catalog adds LiteLLM models for a key, its `ETag` is a digest of OpenAI's ETag and those models, and that key's responses carry the same value in `X-Models-Etag`. Codex keeps its cached catalog while the two match and fetches it again when OpenAI's catalog or the key's models change. A key whose catalog adds no LiteLLM models keeps OpenAI's ETags.
-
-### Restrict native models with an access group
-
-A key with a model allowlist needs a rule for ChatGPT names that OpenAI adds over time. Put the subscription wildcards in an access group under `chatgpt/` names:
-
-```yaml
-model_list:
- - model_name: chatgpt/gpt-*
- litellm_params:
- model: chatgpt/gpt-*
- model_info:
- access_groups: ["chatgpt-native"]
- - model_name: chatgpt/codex-*
- litellm_params:
- model: chatgpt/codex-*
- model_info:
- access_groups: ["chatgpt-native"]
-```
-
-Then add `chatgpt-native` to the key's models, next to any other model names it should reach. On the native route, a key that holds only the `chatgpt/` form of a name is checked as that form, so a request for `gpt-5.6-sol` is allowed when the key may call `chatgpt/gpt-5.6-sol`. The key can call every ChatGPT model that matches the wildcards, including ones the proxy configuration does not name.
-
-On `/v1` these wildcards match only a request that asks for a `chatgpt/` name, so a request for a plain `gpt-` or `codex-` name still goes where it went before, such as a `*` catch-all deployment. A `/v1` request for a `chatgpt/` name without a ChatGPT login falls back to the proxy's own saved login if it has one, so keep that login unset as described above.
-
-### Configure Pi
-
-Sign into the built-in `openai-codex` provider in Pi. Merge this override into `~/.pi/agent/models.json` and export your LiteLLM virtual key as `LITELLM_API_KEY` before launching Pi:
-
-```json
-{
- "providers": {
- "openai-codex": {
- "baseUrl": "https://litellm.example.com/backend-api/codex",
- "headers": {
- "x-litellm-api-key": "LITELLM_API_KEY"
- }
- }
- }
-}
-```
-
-Keep Pi's subscription login as the provider credential. The header value names an environment variable, not a key to paste into the file.
-
-## Verify account usage
-
-Send a prompt and check the completed request in LiteLLM's Logs page. Its gateway user/key and token counts should match the caller. Give each person their own LiteLLM virtual key for separate attribution. The gateway credential and ChatGPT login are independently authenticated; LiteLLM does not establish that both identities belong to the same person.
-
-The native routes are `GET /backend-api/codex/models` and `POST /backend-api/codex/responses`. Model requests use the upstream model name, so a restricted LiteLLM key needs the `chatgpt-native` access group above or that exact name. OpenAI's model catalog is returned in its native format; seeing a model there does not override gateway permissions.
-
-The proxy forwards the ChatGPT bearer and account header to the fixed OpenAI Codex endpoint. It removes gateway credentials and cookies from the upstream request and redacts credential headers in logging callbacks. The client remains responsible for refresh and reauthentication. Native passthrough returns authentication and quota errors without switching subscriptions. Standard model routes retain the caller login across ChatGPT retries; explicitly configured cross-provider fallbacks use their own credentials. An explicit workspace header is preserved for OpenAI to validate.
-
-This flow does not connect accounts through the dashboard or pool subscription allowances. Use a Codex build that supports ChatGPT authentication with a custom model provider. The integration forwards HTTP and SSE requests; it does not add native WebSocket or other Codex backend routes.
diff --git a/cookbook/ai_coding_tool_guides/claude_code_quickstart/guide.md b/cookbook/ai_coding_tool_guides/claude_code_quickstart/guide.md
deleted file mode 100644
index cd27eed18a3..00000000000
--- a/cookbook/ai_coding_tool_guides/claude_code_quickstart/guide.md
+++ /dev/null
@@ -1,294 +0,0 @@
-# Claude Code with LiteLLM Quickstart
-
-This guide shows how to call Claude models (and any LiteLLM-supported model) through LiteLLM proxy from Claude Code.
-
-> **Note:** This integration is based on [Anthropic's official LiteLLM configuration documentation](https://docs.anthropic.com/en/docs/claude-code/llm-gateway#litellm-configuration). It allows you to use any LiteLLM supported model through Claude Code with centralized authentication, usage tracking, and cost controls.
-
-## Video Walkthrough
-
-Watch the full tutorial: https://www.loom.com/embed/3c17d683cdb74d36a3698763cc558f56
-
-## Prerequisites
-
-- [Claude Code](https://docs.anthropic.com/en/docs/claude-code/overview) installed
-- API keys for your chosen providers
-
-## Installation
-
-First, install LiteLLM with proxy support:
-
-```bash
-pip install 'litellm[proxy]'
-```
-
-## Step 1: Setup config.yaml
-
-Create a secure configuration using environment variables:
-
-```yaml
-model_list:
- # Claude models
- - model_name: claude-3-5-sonnet-20241022
- litellm_params:
- model: anthropic/claude-3-5-sonnet-20241022
- api_key: os.environ/ANTHROPIC_API_KEY
-
- - model_name: claude-3-5-haiku-20241022
- litellm_params:
- model: anthropic/claude-3-5-haiku-20241022
- api_key: os.environ/ANTHROPIC_API_KEY
-
-
-litellm_settings:
- master_key: os.environ/LITELLM_MASTER_KEY
-```
-
-Set your environment variables:
-
-```bash
-export ANTHROPIC_API_KEY="your-anthropic-api-key"
-export LITELLM_MASTER_KEY="sk-$(openssl rand -hex 32)"
-```
-
-## Step 2: Start Proxy
-
-```bash
-litellm --config /path/to/config.yaml
-
-# RUNNING on http://0.0.0.0:4000
-```
-
-## Step 3: Verify Setup
-
-Test that your proxy is working correctly:
-
-```bash
-curl -X POST http://0.0.0.0:4000/v1/messages \
--H "Authorization: Bearer $LITELLM_MASTER_KEY" \
--H "Content-Type: application/json" \
--d '{
- "model": "claude-3-5-sonnet-20241022",
- "max_tokens": 1000,
- "messages": [{"role": "user", "content": "What is the capital of France?"}]
-}'
-```
-
-## Step 4: Configure Claude Code
-
-### Method 1: Unified Endpoint (Recommended)
-
-Configure Claude Code to use LiteLLM's unified endpoint. Either a virtual key or master key can be used here:
-
-```bash
-export ANTHROPIC_BASE_URL="http://0.0.0.0:4000"
-export ANTHROPIC_AUTH_TOKEN="$LITELLM_MASTER_KEY"
-```
-
-> **Tip:** LITELLM_MASTER_KEY gives Claude access to all proxy models, whereas a virtual key would be limited to the models set in the UI.
-
-### Method 2: Provider-specific Pass-through Endpoint
-
-Alternatively, use the Anthropic pass-through endpoint:
-
-```bash
-export ANTHROPIC_BASE_URL="http://0.0.0.0:4000/anthropic"
-export ANTHROPIC_AUTH_TOKEN="$LITELLM_MASTER_KEY"
-```
-
-## Step 5: Use Claude Code
-
-### Choosing Your Model
-
-You have two options for specifying which model Claude Code uses:
-
-#### Option 1: Command Line / Session Model Selection
-
-Specify the model directly when starting Claude Code or during a session:
-
-```bash
-# Specify model at startup
-claude --model claude-3-5-sonnet-20241022
-
-# Or change model during a session
-/model claude-3-5-haiku-20241022
-```
-
-This method uses the exact model you specify.
-
-#### Option 2: Environment Variables
-
-Configure default models using environment variables:
-
-```bash
-# Tell Claude Code which models to use by default
-export ANTHROPIC_DEFAULT_SONNET_MODEL=claude-3-5-sonnet-20241022
-export ANTHROPIC_DEFAULT_HAIKU_MODEL=claude-3-5-haiku-20241022
-export ANTHROPIC_DEFAULT_OPUS_MODEL=claude-opus-3-5-20240229
-
-claude # Will use the models specified above
-```
-
-**Note:** Claude Code may cache the model from a previous session. If environment variables don't take effect, use Option 1 to explicitly set the model.
-
-**Important:** The `model_name` in your LiteLLM config must match what Claude Code requests (either from env vars or command line).
-
-### Using 1M Context Window
-
-Claude Code supports extended context (1 million tokens) using the `[1m]` suffix with Claude 4+ models:
-
-```bash
-# Use Sonnet 4.5 with 1M context (requires quotes for shell)
-claude --model 'claude-sonnet-4-5-20250929[1m]'
-
-# Inside a Claude Code session (no quotes needed)
-/model claude-sonnet-4-5-20250929[1m]
-```
-
-**Important:** When using `--model` with `[1m]` in the shell, you must use quotes to prevent the shell from interpreting the brackets.
-
-Alternatively, set as default with environment variables:
-
-```bash
-export ANTHROPIC_DEFAULT_SONNET_MODEL='claude-sonnet-4-5-20250929[1m]'
-claude
-```
-
-**How it works:**
-- Claude Code strips the `[1m]` suffix before sending to LiteLLM
-- Claude Code automatically adds the header `anthropic-beta: context-1m-2025-08-07`
-- Your LiteLLM config should **NOT** include `[1m]` in model names
-
-**Verify 1M context is active:**
-```bash
-/context
-# Should show: 21k/1000k tokens (2%)
-```
-
-**Pricing:** Models using 1M context have different pricing. Input tokens above 200k are charged at a higher rate.
-
-## Troubleshooting
-
-Common issues and solutions:
-
-**Claude Code not connecting:**
-- Verify your proxy is running: `curl http://0.0.0.0:4000/health`
-- Check that `ANTHROPIC_BASE_URL` is set correctly
-- Ensure your `ANTHROPIC_AUTH_TOKEN` matches your LiteLLM master key
-
-**Authentication errors:**
-- Verify your environment variables are set: `echo $LITELLM_MASTER_KEY`
-- Check that your API keys are valid and have sufficient credits
-- Ensure the `ANTHROPIC_AUTH_TOKEN` matches your LiteLLM master key
-
-**Model not found:**
-- Check what model Claude Code is requesting in LiteLLM logs
-- Ensure your `config.yaml` has a matching `model_name` entry
-- If using environment variables, verify they're set: `echo $ANTHROPIC_DEFAULT_SONNET_MODEL`
-
-**1M context not working (showing 200k instead of 1000k):**
-- Verify you're using the `[1m]` suffix: `/model your-model-name[1m]`
-- Check LiteLLM logs for the header `context-1m-2025-08-07` in the request
-- Ensure your model supports 1M context (only certain Claude models do)
-- Your LiteLLM config should **NOT** include `[1m]` in the `model_name`
-
-## Using Multiple Models and Providers
-
-You can configure LiteLLM to route to any supported provider. Here's an example with multiple providers:
-
-```yaml
-model_list:
- # OpenAI models
- - model_name: codex-mini
- litellm_params:
- model: openai/codex-mini
- api_key: os.environ/OPENAI_API_KEY
- api_base: https://api.openai.com/v1
-
- - model_name: o3-pro
- litellm_params:
- model: openai/o3-pro
- api_key: os.environ/OPENAI_API_KEY
- api_base: https://api.openai.com/v1
-
- - model_name: gpt-4o
- litellm_params:
- model: openai/gpt-4o
- api_key: os.environ/OPENAI_API_KEY
- api_base: https://api.openai.com/v1
-
- # Anthropic models
- - model_name: claude-3-5-sonnet-20241022
- litellm_params:
- model: anthropic/claude-3-5-sonnet-20241022
- api_key: os.environ/ANTHROPIC_API_KEY
-
- - model_name: claude-3-5-haiku-20241022
- litellm_params:
- model: anthropic/claude-3-5-haiku-20241022
- api_key: os.environ/ANTHROPIC_API_KEY
-
- # AWS Bedrock
- - model_name: claude-bedrock
- litellm_params:
- model: bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
- aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
- aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
- aws_region_name: us-east-1
-
-litellm_settings:
- master_key: os.environ/LITELLM_MASTER_KEY
-```
-
-**Note:** The `model_name` can be anything you choose. Claude Code will request whatever model you specify (via env vars or command line), and LiteLLM will route to the `model` configured in `litellm_params`.
-
-Switch between models seamlessly:
-
-```bash
-# Use environment variables to set defaults
-export ANTHROPIC_DEFAULT_SONNET_MODEL=claude-3-5-sonnet-20241022
-export ANTHROPIC_DEFAULT_HAIKU_MODEL=claude-3-5-haiku-20241022
-
-# Or specify directly
-claude --model claude-3-5-sonnet-20241022 # Complex reasoning
-claude --model claude-3-5-haiku-20241022 # Fast responses
-claude --model claude-bedrock # Bedrock deployment
-```
-
-## Default Models Used by Claude Code
-
-If you **don't** set environment variables, Claude Code uses these default model names:
-
-| Purpose | Default Model Name (v2.1.14) |
-|---------|------------------------------|
-| Main model | `claude-sonnet-4-5-20250929` |
-| Light tasks (subagents, summaries) | `claude-haiku-4-5-20251001` |
-| Planning mode | `claude-opus-4-5-20251101` |
-
-Your LiteLLM config should include these model names if you want Claude Code to work without setting environment variables:
-
-```yaml
-model_list:
- - model_name: claude-sonnet-4-5-20250929
- litellm_params:
- # Can be any provider - Anthropic, Bedrock, Vertex AI, etc.
- model: anthropic/claude-sonnet-4-5-20250929
- api_key: os.environ/ANTHROPIC_API_KEY
-
- - model_name: claude-haiku-4-5-20251001
- litellm_params:
- model: anthropic/claude-haiku-4-5-20251001
- api_key: os.environ/ANTHROPIC_API_KEY
-
- - model_name: claude-opus-4-5-20251101
- litellm_params:
- model: anthropic/claude-opus-4-5-20251101
- api_key: os.environ/ANTHROPIC_API_KEY
-```
-
-**Warning:** These default model names may change with new Claude Code versions. Check LiteLLM proxy logs for "model not found" errors to identify what Claude Code is requesting.
-
-## Additional Resources
-
-- [LiteLLM Documentation](https://docs.litellm.ai/)
-- [Claude Code Documentation](https://docs.anthropic.com/en/docs/claude-code/overview)
-- [Anthropic's LiteLLM Configuration Guide](https://docs.anthropic.com/en/docs/claude-code/llm-gateway#litellm-configuration)
diff --git a/cookbook/ai_coding_tool_guides/index.json b/cookbook/ai_coding_tool_guides/index.json
deleted file mode 100644
index 3e71670d623..00000000000
--- a/cookbook/ai_coding_tool_guides/index.json
+++ /dev/null
@@ -1,134 +0,0 @@
-[{
- "title": "Claude Code Quickstart",
- "description": "This is a quickstart guide to using Claude Code with LiteLLM.",
- "url": "https://docs.litellm.ai/docs/tutorials/claude_responses_api",
- "date": "2026-01-15",
- "version": "1.0.0",
- "tags": [
- "Claude Code",
- "LiteLLM"
- ]
-},
-{
- "title": "Claude Code with MCPs",
- "description": "This is a guide to using Claude Code with MCPs via LiteLLM Proxy.",
- "url": "https://docs.litellm.ai/docs/tutorials/claude_mcp",
- "date": "2026-01-15",
- "version": "1.0.0",
- "tags": [
- "Claude Code",
- "LiteLLM",
- "MCP"
- ]
-},
-{
- "title": "Claude Code with Non-Anthropic Models",
- "description": "This is a guide to using Claude Code with non-Anthropic models via LiteLLM Proxy.",
- "url": "https://docs.litellm.ai/docs/tutorials/claude_non_anthropic_models",
- "date": "2026-01-16",
- "version": "1.0.0",
- "tags": [
- "Claude Code",
- "LiteLLM",
- "OpenAI",
- "Gemini"
- ]
-},
-{
- "title": "Cursor Quickstart",
- "description": "This is a quickstart guide to using Cursor with LiteLLM.",
- "url": "https://docs.litellm.ai/docs/tutorials/cursor_integration",
- "date": "2026-01-16",
- "version": "1.0.0",
- "tags": [
- "Cursor",
- "LiteLLM",
- "Quickstart"
- ]
-},
-{
- "title": "Github Copilot Quickstart",
- "description": "This is a quickstart guide to using Github Copilot with LiteLLM.",
- "url": "https://docs.litellm.ai/docs/tutorials/github_copilot_integration",
- "date": "2026-01-16",
- "version": "1.0.0",
- "tags": [
- "Github Copilot",
- "LiteLLM",
- "Quickstart"
- ]
-},
-{
- "title": "LiteLLM Gemini CLI Quickstart",
- "description": "This is a quickstart guide to using LiteLLM Gemini CLI.",
- "url": "https://docs.litellm.ai/docs/tutorials/litellm_gemini_cli",
- "date": "2026-01-16",
- "version": "1.0.0",
- "tags": [
- "Gemini CLI",
- "Gemini",
- "LiteLLM",
- "Quickstart"
- ]
-},
-{
- "title": "OpenAI Codex CLI Quickstart",
- "description": "This is a quickstart guide to using OpenAI Codex CLI.",
- "url": "https://docs.litellm.ai/docs/tutorials/openai_codex",
- "date": "2026-01-16",
- "version": "1.0.0",
- "tags": [
- "OpenAI Codex CLI",
- "OpenAI",
- "LiteLLM",
- "Quickstart"
- ]
-},
-{
- "title": "OpenWebUI Quickstart",
- "description": "This is a quickstart guide to using OpenWebUI with LiteLLM.",
- "url": "https://docs.litellm.ai/docs/tutorials/openweb_ui",
- "date": "2026-01-16",
- "version": "1.0.0",
- "tags": [
- "OpenWebUI",
- "LiteLLM",
- "Quickstart"
- ]
-},
-{
- "title": "AI Coding Tool Usage Tracking",
- "description": "This is a guide to tracking usage for AI coding tools monitor the use of Claude Code , Google Antigravity, OpenAI Codex, Roo Code etc. through LiteLLM.",
- "url": "https://docs.litellm.ai/docs/tutorials/cost_tracking_coding",
- "date": "2026-01-17",
- "version": "1.0.0",
- "tags": [
- "Claude Code",
- "Gemini CLI",
- "OpenAI Codex",
- "LiteLLM"
- ]
-},
-{
- "title": "Use Web Search with Claude Code (across Bedrock/OpenAI/Gemini/etc.)",
- "description": "This is a guide for using Web Search with Claude Code via LiteLLM.",
- "url": "https://docs.litellm.ai/docs/tutorials/claude_code_websearch",
- "date": "2026-01-17",
- "version": "1.0.0",
- "tags": [
- "Claude Code",
- "LiteLLM",
- "Web Search"
- ]
-},
-{
- "title": "Track Claude Code Usage per user via Custom Headers",
- "description": "This is a guide for tracking claude code user usage by passing a customer ID header.",
- "url": "https://docs.litellm.ai/docs/tutorials/claude_code_customer_tracking",
- "date": "2026-01-17",
- "version": "1.0.0",
- "tags": [
- "Claude Code",
- "LiteLLM"
- ]
-}]
\ No newline at end of file
diff --git a/cookbook/anthropic_agent_sdk/README.md b/cookbook/anthropic_agent_sdk/README.md
deleted file mode 100644
index 4521aee310a..00000000000
--- a/cookbook/anthropic_agent_sdk/README.md
+++ /dev/null
@@ -1,150 +0,0 @@
-# Claude Agent SDK with LiteLLM Gateway
-
-A simple example showing how to use Claude's Agent SDK with LiteLLM as a proxy. This lets you use any LLM provider (OpenAI, Bedrock, Azure, etc.) through the Agent SDK.
-
-## Quick Start
-
-### 1. Install dependencies
-
-```bash
-pip install anthropic claude-agent-sdk litellm
-```
-
-### 2. Start LiteLLM proxy
-
-```bash
-# Simple start with Claude
-litellm --model claude-sonnet-4-20250514
-
-# Or with a config file
-litellm --config config.yaml
-```
-
-Generate a master key before starting the proxy:
-
-```bash
-export LITELLM_MASTER_KEY="sk-$(openssl rand -hex 32)"
-```
-
-### 3. Run the chat
-
-**Basic Agent (no MCP):**
-
-```bash
-python main.py
-```
-
-**Agent with MCP (DeepWiki2 for research):**
-
-```bash
-python agent_with_mcp.py
-```
-
-If MCP connection fails, you can disable it:
-
-```bash
-USE_MCP=false python agent_with_mcp.py
-```
-
-That's it! You can now chat with the agent in your terminal.
-
-### Chat Commands
-
-While chatting, you can use these commands:
-- `models` - List all available models (fetched from your LiteLLM proxy)
-- `model` - Switch to a different model
-- `clear` - Start a new conversation
-- `quit` or `exit` - End the chat
-
-The chat automatically fetches available models from your LiteLLM proxy's `/models` endpoint, so you'll always see what's currently configured.
-
-## Configuration
-
-Set these environment variables if needed:
-
-```bash
-export LITELLM_PROXY_URL="http://localhost:4000"
-export LITELLM_API_KEY="$LITELLM_MASTER_KEY"
-export LITELLM_MODEL="bedrock-claude-sonnet-4.5"
-```
-
-Or just use the defaults - it'll connect to `http://localhost:4000` by default.
-
-## Files
-
-- `main.py` - Basic interactive agent without MCP
-- `agent_with_mcp.py` - Agent with MCP server integration (DeepWiki2)
-- `common.py` - Shared utilities and functions
-- `config.example.yaml` - Example LiteLLM configuration
-- `requirements.txt` - Python dependencies
-
-## Example Config File
-
-If you want to use multiple models, create a `config.yaml` (see `config.example.yaml`):
-
-```yaml
-model_list:
- - model_name: bedrock-claude-sonnet-4
- litellm_params:
- model: "bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0"
- aws_region_name: "us-east-1"
-
- - model_name: bedrock-claude-sonnet-4.5
- litellm_params:
- model: "bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0"
- aws_region_name: "us-east-1"
-```
-
-Then start LiteLLM with: `litellm --config config.yaml`
-
-## How It Works
-
-The key is pointing the Agent SDK to LiteLLM instead of directly to Anthropic:
-
-```python
-# Point to LiteLLM gateway (not Anthropic)
-os.environ["ANTHROPIC_BASE_URL"] = "http://localhost:4000"
-os.environ["ANTHROPIC_API_KEY"] = os.environ["LITELLM_API_KEY"]
-
-# Use any model configured in LiteLLM
-options = ClaudeAgentOptions(
- model="bedrock-claude-sonnet-4", # or gpt-4, or anything else
- system_prompt="You are a helpful assistant.",
- max_turns=50,
-)
-```
-
-Note: Don't add `/anthropic` to the base URL - LiteLLM handles the routing automatically.
-
-## Why Use This?
-
-- **Switch providers easily**: Use the same code with OpenAI, Bedrock, Azure, etc.
-- **Cost tracking**: LiteLLM tracks spending across all your agent conversations
-- **Rate limiting**: Set budgets and limits on your agent usage
-- **Load balancing**: Distribute requests across multiple API keys or regions
-- **Fallbacks**: Automatically retry with a different model if one fails
-
-## Troubleshooting
-
-**Connection errors?**
-- Make sure LiteLLM is running: `litellm --model your-model`
-- Check the URL is correct (default: `http://localhost:4000`)
-
-**Authentication errors?**
-- Verify your LiteLLM API key is correct
-- Make sure the model is configured in your LiteLLM setup
-
-**Model not found?**
-- Check the model name matches what's in your LiteLLM config
-- Run `litellm --model your-model` to test it works
-
-**Agent with MCP stuck or failing?**
-- The MCP server might not be available at `http://localhost:4000/mcp/deepwiki2`
-- Try disabling MCP: `USE_MCP=false python agent_with_mcp.py`
-- Or use the basic agent: `python main.py`
-
-## Learn More
-
-- [LiteLLM Docs](https://docs.litellm.ai/)
-- [Claude Agent SDK](https://github.com/anthropics/anthropic-agent-sdk)
-- [LiteLLM Proxy Guide](https://docs.litellm.ai/docs/proxy/quick_start)
diff --git a/cookbook/anthropic_agent_sdk/agent_with_mcp.py b/cookbook/anthropic_agent_sdk/agent_with_mcp.py
deleted file mode 100644
index 8a7513c786c..00000000000
--- a/cookbook/anthropic_agent_sdk/agent_with_mcp.py
+++ /dev/null
@@ -1,144 +0,0 @@
-"""
-Interactive Claude Agent SDK CLI with MCP Support
-
-This example demonstrates an interactive CLI chat with the Anthropic Agent SDK using LiteLLM as a proxy,
-with MCP (Model Context Protocol) server integration for enhanced capabilities.
-"""
-
-import asyncio
-import os
-from claude_agent_sdk import ClaudeSDKClient, ClaudeAgentOptions
-from common import (
- Config,
- fetch_available_models,
- setup_litellm_env,
- print_header,
- handle_model_list,
- handle_model_switch,
- stream_response,
-)
-
-
-async def interactive_chat_with_mcp():
- """
- Interactive CLI chat with the agent and MCP server
- """
- config = Config()
-
- # Configure Anthropic SDK to point to LiteLLM gateway
- litellm_base_url = setup_litellm_env(config)
-
- # Fetch available models from proxy
- available_models = await fetch_available_models(
- litellm_base_url, config.LITELLM_API_KEY
- )
-
- current_model = config.LITELLM_MODEL
-
- # MCP server configuration
- mcp_server_url = f"{litellm_base_url}/mcp/deepwiki2"
- use_mcp = os.getenv("USE_MCP", "true").lower() == "true"
-
- if not use_mcp:
- print("ā ļø MCP disabled via USE_MCP=false")
-
- print_header(litellm_base_url, current_model, has_mcp=use_mcp)
-
- while True:
- # Configure agent options
- if use_mcp:
- try:
- # Try with MCP server (HTTP transport)
- # Using McpHttpServerConfig format from Agent SDK
- options = ClaudeAgentOptions(
- system_prompt="You are a helpful AI assistant with access to DeepWiki for research. Be concise, accurate, and friendly.",
- model=current_model,
- max_turns=50,
- mcp_servers={
- "deepwiki2": {
- "type": "http",
- "url": mcp_server_url,
- "headers": {
- "Authorization": f"Bearer {config.LITELLM_API_KEY}"
- },
- }
- },
- )
- except Exception as e:
- print(f"ā ļø Warning: Could not configure MCP server: {e}")
- print("Continuing without MCP...\n")
- use_mcp = False
- options = ClaudeAgentOptions(
- system_prompt="You are a helpful AI assistant. Be concise, accurate, and friendly.",
- model=current_model,
- max_turns=50,
- )
- else:
- # Without MCP
- options = ClaudeAgentOptions(
- system_prompt="You are a helpful AI assistant. Be concise, accurate, and friendly.",
- model=current_model,
- max_turns=50,
- )
-
- # Create agent client
- try:
- async with ClaudeSDKClient(options=options) as client:
- conversation_active = True
-
- while conversation_active:
- # Get user input
- try:
- user_input = input("\nš¤ You: ").strip()
- except (EOFError, KeyboardInterrupt):
- print("\n\nš Goodbye!")
- return
-
- # Handle commands
- if user_input.lower() in ["quit", "exit"]:
- print("\nš Goodbye!")
- return
-
- if user_input.lower() == "clear":
- print("\nš Starting new conversation...\n")
- conversation_active = False
- continue
-
- if user_input.lower() == "models":
- handle_model_list(available_models, current_model)
- continue
-
- if user_input.lower() == "model":
- new_model, should_restart = handle_model_switch(
- available_models, current_model
- )
- if should_restart:
- current_model = new_model
- conversation_active = False
- continue
-
- if not user_input:
- continue
-
- # Stream response from agent
- await stream_response(client, user_input)
-
- except Exception as e:
- print(f"\nā Error creating agent client: {e}")
- print("This might be an MCP configuration issue. Try running without MCP:")
- print(" USE_MCP=false python agent_with_mcp.py")
- print("\nOr use the basic agent:")
- print(" python main.py")
- return
-
-
-def main():
- """Run interactive chat with MCP"""
- try:
- asyncio.run(interactive_chat_with_mcp())
- except KeyboardInterrupt:
- print("\n\nš Goodbye!")
-
-
-if __name__ == "__main__":
- main()
diff --git a/cookbook/anthropic_agent_sdk/common.py b/cookbook/anthropic_agent_sdk/common.py
deleted file mode 100644
index 505b5ecd91c..00000000000
--- a/cookbook/anthropic_agent_sdk/common.py
+++ /dev/null
@@ -1,164 +0,0 @@
-"""
-Common utilities for Claude Agent SDK examples
-"""
-
-import os
-import httpx
-
-
-class Config:
- """Configuration for LiteLLM Gateway connection"""
-
- # LiteLLM proxy URL (default to local instance)
- LITELLM_PROXY_URL = os.getenv("LITELLM_PROXY_URL", "http://localhost:4000")
-
- # LiteLLM API key (master key or virtual key)
- LITELLM_API_KEY = os.environ["LITELLM_API_KEY"]
-
- # Model name as configured in LiteLLM (e.g., "bedrock-claude-sonnet-4", "gpt-4", etc.)
- LITELLM_MODEL = os.getenv("LITELLM_MODEL", "bedrock-claude-sonnet-4.5")
-
-
-async def fetch_available_models(base_url: str, api_key: str) -> list[str]:
- """
- Fetch available models from LiteLLM proxy /models endpoint
- """
- try:
- async with httpx.AsyncClient() as client:
- response = await client.get(
- f"{base_url}/models",
- headers={"Authorization": f"Bearer {api_key}"},
- timeout=10.0,
- )
- response.raise_for_status()
- data = response.json()
- return [model["id"] for model in data.get("data", [])]
- except Exception as e:
- print(f"ā ļø Warning: Could not fetch models from proxy: {e}")
- print("Using default model list...")
- # Fallback to default models
- return [
- "bedrock-claude-sonnet-3.5",
- "bedrock-claude-sonnet-4",
- "bedrock-claude-sonnet-4.5",
- "bedrock-claude-opus-4.5",
- "bedrock-nova-premier",
- ]
-
-
-def setup_litellm_env(config: Config):
- """
- Configure environment variables to point Agent SDK to LiteLLM
- """
- litellm_base_url = config.LITELLM_PROXY_URL.rstrip("/")
- os.environ["ANTHROPIC_BASE_URL"] = litellm_base_url
- os.environ["ANTHROPIC_API_KEY"] = config.LITELLM_API_KEY
- return litellm_base_url
-
-
-def print_header(base_url: str, current_model: str, has_mcp: bool = False):
- """
- Print the chat header
- """
- mcp_indicator = " + MCP" if has_mcp else ""
- print("=" * 70)
- print(f"š¤ Claude Agent SDK with LiteLLM Gateway{mcp_indicator} - Interactive Chat")
- print("=" * 70)
- print(f"š Connected to: {base_url}")
- print(f"š¦ Current model: {current_model}")
- if has_mcp:
- print("š MCP: deepwiki2 enabled")
- print("\nType your messages below. Commands:")
- print(" - 'quit' or 'exit' to end the conversation")
- print(" - 'clear' to start a new conversation")
- print(" - 'model' to switch models")
- print(" - 'models' to list available models")
- print("=" * 70)
- print()
-
-
-def handle_model_list(available_models: list[str], current_model: str):
- """
- Display available models
- """
- print("\nš Available models:")
- for i, model in enumerate(available_models, 1):
- marker = "ā" if model == current_model else " "
- print(f" {marker} {i}. {model}")
-
-
-def handle_model_switch(
- available_models: list[str], current_model: str
-) -> tuple[str, bool]:
- """
- Handle model switching
-
- Returns:
- tuple: (new_model, should_restart_conversation)
- """
- print("\nš Select a model:")
- for i, model in enumerate(available_models, 1):
- marker = "ā" if model == current_model else " "
- print(f" {marker} {i}. {model}")
-
- try:
- choice = input("\nEnter number (or press Enter to cancel): ").strip()
- if choice:
- idx = int(choice) - 1
- if 0 <= idx < len(available_models):
- new_model = available_models[idx]
- print(f"\nā Switched to: {new_model}")
- print("š Starting new conversation with new model...\n")
- return new_model, True
- else:
- print("ā Invalid choice")
- except (ValueError, IndexError):
- print("ā Invalid input")
-
- return current_model, False
-
-
-async def stream_response(client, user_input: str):
- """
- Stream response from the agent
- """
- print("\nš¤ Assistant: ", end="", flush=True)
-
- try:
- await client.query(user_input)
-
- # Show loading indicator
- print("ā³ thinking...", end="", flush=True)
-
- # Stream the response
- first_chunk = True
- async for msg in client.receive_response():
- # Clear loading indicator on first message
- if first_chunk:
- print("\rš¤ Assistant: ", end="", flush=True)
- first_chunk = False
-
- # Handle different message types
- if hasattr(msg, "type"):
- if msg.type == "content_block_delta":
- # Streaming text delta
- if hasattr(msg, "delta") and hasattr(msg.delta, "text"):
- print(msg.delta.text, end="", flush=True)
- elif msg.type == "content_block_start":
- # Start of content block
- if hasattr(msg, "content_block") and hasattr(
- msg.content_block, "text"
- ):
- print(msg.content_block.text, end="", flush=True)
-
- # Fallback to original content handling
- if hasattr(msg, "content"):
- for content_block in msg.content:
- if hasattr(content_block, "text"):
- print(content_block.text, end="", flush=True)
-
- print() # New line after response
-
- except Exception as e:
- print(f"\r\nā Error: {e}")
- print("Please check your LiteLLM gateway is running and configured correctly.")
diff --git a/cookbook/anthropic_agent_sdk/config.example.yaml b/cookbook/anthropic_agent_sdk/config.example.yaml
deleted file mode 100644
index eb1984fc4ea..00000000000
--- a/cookbook/anthropic_agent_sdk/config.example.yaml
+++ /dev/null
@@ -1,25 +0,0 @@
-model_list:
- - model_name: bedrock-claude-sonnet-3.5
- litellm_params:
- model: "bedrock/us.anthropic.claude-3-5-sonnet-20240620-v1:0"
- aws_region_name: "us-east-1"
-
- - model_name: bedrock-claude-sonnet-4
- litellm_params:
- model: "bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0"
- aws_region_name: "us-east-1"
-
- - model_name: bedrock-claude-sonnet-4.5
- litellm_params:
- model: "bedrock/us.anthropic.claude-sonnet-4-5-20250929-v1:0"
- aws_region_name: "us-east-1"
-
- - model_name: bedrock-claude-opus-4.5
- litellm_params:
- model: "bedrock/us.anthropic.claude-opus-4-5-20251101-v1:0"
- aws_region_name: "us-east-1"
-
- - model_name: bedrock-nova-premier
- litellm_params:
- model: "bedrock/amazon.nova-premier-v1:0"
- aws_region_name: "us-east-1"
diff --git a/cookbook/anthropic_agent_sdk/main.py b/cookbook/anthropic_agent_sdk/main.py
deleted file mode 100644
index 506c6fa07b0..00000000000
--- a/cookbook/anthropic_agent_sdk/main.py
+++ /dev/null
@@ -1,99 +0,0 @@
-"""
-Simple Interactive Claude Agent SDK CLI using LiteLLM Gateway
-
-This example demonstrates an interactive CLI chat with the Anthropic Agent SDK using LiteLLM as a proxy.
-LiteLLM acts as a unified interface, allowing you to use any LLM provider (OpenAI, Azure, Bedrock, etc.)
-through the Claude Agent SDK by pointing it to the LiteLLM gateway.
-"""
-
-import asyncio
-from claude_agent_sdk import ClaudeSDKClient, ClaudeAgentOptions
-from common import (
- Config,
- fetch_available_models,
- setup_litellm_env,
- print_header,
- handle_model_list,
- handle_model_switch,
- stream_response,
-)
-
-
-async def interactive_chat():
- """
- Interactive CLI chat with the agent
- """
- config = Config()
-
- # Configure Anthropic SDK to point to LiteLLM gateway
- litellm_base_url = setup_litellm_env(config)
-
- # Fetch available models from proxy
- available_models = await fetch_available_models(
- litellm_base_url, config.LITELLM_API_KEY
- )
-
- current_model = config.LITELLM_MODEL
-
- print_header(litellm_base_url, current_model)
-
- while True:
- # Configure agent options for each conversation
- options = ClaudeAgentOptions(
- system_prompt="You are a helpful AI assistant. Be concise, accurate, and friendly.",
- model=current_model,
- max_turns=50,
- )
-
- # Create agent client
- async with ClaudeSDKClient(options=options) as client:
- conversation_active = True
-
- while conversation_active:
- # Get user input
- try:
- user_input = input("\nš¤ You: ").strip()
- except (EOFError, KeyboardInterrupt):
- print("\n\nš Goodbye!")
- return
-
- # Handle commands
- if user_input.lower() in ["quit", "exit"]:
- print("\nš Goodbye!")
- return
-
- if user_input.lower() == "clear":
- print("\nš Starting new conversation...\n")
- conversation_active = False
- continue
-
- if user_input.lower() == "models":
- handle_model_list(available_models, current_model)
- continue
-
- if user_input.lower() == "model":
- new_model, should_restart = handle_model_switch(
- available_models, current_model
- )
- if should_restart:
- current_model = new_model
- conversation_active = False
- continue
-
- if not user_input:
- continue
-
- # Stream response from agent
- await stream_response(client, user_input)
-
-
-def main():
- """Run interactive chat"""
- try:
- asyncio.run(interactive_chat())
- except KeyboardInterrupt:
- print("\n\nš Goodbye!")
-
-
-if __name__ == "__main__":
- main()
diff --git a/cookbook/anthropic_agent_sdk/requirements.txt b/cookbook/anthropic_agent_sdk/requirements.txt
deleted file mode 100644
index 1e810bb7d99..00000000000
--- a/cookbook/anthropic_agent_sdk/requirements.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-claude-agent-sdk
-httpx>=0.27.0
diff --git a/cookbook/benchmark/benchmark.py b/cookbook/benchmark/benchmark.py
deleted file mode 100644
index b38d185a166..00000000000
--- a/cookbook/benchmark/benchmark.py
+++ /dev/null
@@ -1,90 +0,0 @@
-from litellm import completion, completion_cost
-import time
-import click
-from tqdm import tqdm
-from tabulate import tabulate
-from termcolor import colored
-import os
-
-
-# Define the list of models to benchmark
-# select any LLM listed here: https://docs.litellm.ai/docs/providers
-models = ["gpt-3.5-turbo", "claude-2"]
-
-# Enter LLM API keys
-# https://docs.litellm.ai/docs/providers
-os.environ["OPENAI_API_KEY"] = ""
-os.environ["ANTHROPIC_API_KEY"] = ""
-
-# List of questions to benchmark (replace with your questions)
-questions = ["When will BerriAI IPO?", "When will LiteLLM hit $100M ARR?"]
-
-# Enter your system prompt here
-system_prompt = """
-You are LiteLLMs helpful assistant
-"""
-
-
-@click.command()
-@click.option(
- "--system-prompt",
- default="You are a helpful assistant that can answer questions.",
- help="System prompt for the conversation.",
-)
-def main(system_prompt):
- for question in questions:
- data = [] # Data for the current question
-
- with tqdm(total=len(models)) as pbar:
- for model in models:
- colored_description = colored(
- f"Running question: {question} for model: {model}", "green"
- )
- pbar.set_description(colored_description)
- start_time = time.time()
-
- response = completion(
- model=model,
- max_tokens=500,
- messages=[
- {"role": "system", "content": system_prompt},
- {"role": "user", "content": question},
- ],
- )
-
- end = time.time()
- total_time = end - start_time
- cost = completion_cost(completion_response=response)
- raw_response = response["choices"][0]["message"]["content"]
-
- data.append(
- {
- "Model": colored(model, "light_blue"),
- "Response": raw_response, # Colorize the response
- "ResponseTime": colored(f"{total_time:.2f} seconds", "red"),
- "Cost": colored(f"${cost:.6f}", "green"), # Colorize the cost
- }
- )
-
- pbar.update(1)
-
- # Separate headers from the data
- headers = ["Model", "Response", "Response Time (seconds)", "Cost ($)"]
- colwidths = [15, 80, 15, 10]
-
- # Create a nicely formatted table for the current question
- table = tabulate(
- [list(d.values()) for d in data],
- headers,
- tablefmt="grid",
- maxcolwidths=colwidths,
- )
-
- # Print the table for the current question
- colored_question = colored(question, "green")
- click.echo(f"\nBenchmark Results for '{colored_question}':")
- click.echo(table) # Display the formatted table
-
-
-if __name__ == "__main__":
- main()
diff --git a/cookbook/benchmark/eval_suites_mlflow_autoevals/auto_evals.py b/cookbook/benchmark/eval_suites_mlflow_autoevals/auto_evals.py
deleted file mode 100644
index daa38dda585..00000000000
--- a/cookbook/benchmark/eval_suites_mlflow_autoevals/auto_evals.py
+++ /dev/null
@@ -1,30 +0,0 @@
-from dotenv import load_dotenv
-
-load_dotenv()
-
-import litellm
-
-from autoevals.llm import *
-
-###################
-
-# litellm completion call
-question = "which country has the highest population"
-response = litellm.completion(
- model="gpt-3.5-turbo",
- messages=[{"role": "user", "content": question}],
-)
-print(response)
-# use the auto eval Factuality() evaluator
-
-print("calling evaluator")
-evaluator = Factuality()
-result = evaluator(
- output=response.choices[0]["message"][
- "content"
- ], # response from litellm.completion()
- expected="India", # expected output
- input=question, # question passed to litellm.completion
-)
-
-print(result)
diff --git a/cookbook/benchmark/readme.md b/cookbook/benchmark/readme.md
deleted file mode 100644
index afa59aa91ee..00000000000
--- a/cookbook/benchmark/readme.md
+++ /dev/null
@@ -1,181 +0,0 @@
-
- LLM-Bench
-
-
-
Benchmark LLMs response, cost and response time
-
LLM vs Cost per input + output token ($)
-
-
-
- Bar Graph Excel Sheet here
-
-
-| Model | Provider | Cost per input + output token ($)|
-| --- | --- | --- |
-| openrouter/mistralai/mistral-7b-instruct | openrouter | 0.0 |
-| ollama/llama2 | ollama | 0.0 |
-| ollama/llama2:13b | ollama | 0.0 |
-| ollama/llama2:70b | ollama | 0.0 |
-| ollama/llama2-uncensored | ollama | 0.0 |
-| ollama/mistral | ollama | 0.0 |
-| ollama/codellama | ollama | 0.0 |
-| ollama/orca-mini | ollama | 0.0 |
-| ollama/vicuna | ollama | 0.0 |
-| perplexity/codellama-34b-instruct | perplexity | 0.0 |
-| perplexity/llama-2-13b-chat | perplexity | 0.0 |
-| perplexity/llama-2-70b-chat | perplexity | 0.0 |
-| perplexity/mistral-7b-instruct | perplexity | 0.0 |
-| perplexity/replit-code-v1.5-3b | perplexity | 0.0 |
-| text-bison | vertex_ai-text-models | 0.00000025 |
-| text-bison@001 | vertex_ai-text-models | 0.00000025 |
-| chat-bison | vertex_ai-chat-models | 0.00000025 |
-| chat-bison@001 | vertex_ai-chat-models | 0.00000025 |
-| chat-bison-32k | vertex_ai-chat-models | 0.00000025 |
-| code-bison | vertex_ai-code-text-models | 0.00000025 |
-| code-bison@001 | vertex_ai-code-text-models | 0.00000025 |
-| code-gecko@001 | vertex_ai-chat-models | 0.00000025 |
-| code-gecko@latest | vertex_ai-chat-models | 0.00000025 |
-| codechat-bison | vertex_ai-code-chat-models | 0.00000025 |
-| codechat-bison@001 | vertex_ai-code-chat-models | 0.00000025 |
-| codechat-bison-32k | vertex_ai-code-chat-models | 0.00000025 |
-| palm/chat-bison | palm | 0.00000025 |
-| palm/chat-bison-001 | palm | 0.00000025 |
-| palm/text-bison | palm | 0.00000025 |
-| palm/text-bison-001 | palm | 0.00000025 |
-| palm/text-bison-safety-off | palm | 0.00000025 |
-| palm/text-bison-safety-recitation-off | palm | 0.00000025 |
-| anyscale/meta-llama/Llama-2-7b-chat-hf | anyscale | 0.0000003 |
-| anyscale/mistralai/Mistral-7B-Instruct-v0.1 | anyscale | 0.0000003 |
-| openrouter/meta-llama/llama-2-13b-chat | openrouter | 0.0000004 |
-| openrouter/nousresearch/nous-hermes-llama2-13b | openrouter | 0.0000004 |
-| deepinfra/meta-llama/Llama-2-7b-chat-hf | deepinfra | 0.0000004 |
-| deepinfra/mistralai/Mistral-7B-Instruct-v0.1 | deepinfra | 0.0000004 |
-| anyscale/meta-llama/Llama-2-13b-chat-hf | anyscale | 0.0000005 |
-| amazon.titan-text-lite-v1 | bedrock | 0.0000007 |
-| deepinfra/meta-llama/Llama-2-13b-chat-hf | deepinfra | 0.0000007 |
-| text-babbage-001 | text-completion-openai | 0.0000008 |
-| text-ada-001 | text-completion-openai | 0.0000008 |
-| babbage-002 | text-completion-openai | 0.0000008 |
-| openrouter/google/palm-2-chat-bison | openrouter | 0.000001 |
-| openrouter/google/palm-2-codechat-bison | openrouter | 0.000001 |
-| openrouter/meta-llama/codellama-34b-instruct | openrouter | 0.000001 |
-| deepinfra/codellama/CodeLlama-34b-Instruct-hf | deepinfra | 0.0000012 |
-| deepinfra/meta-llama/Llama-2-70b-chat-hf | deepinfra | 0.0000016499999999999999 |
-| deepinfra/jondurbin/airoboros-l2-70b-gpt4-1.4.1 | deepinfra | 0.0000016499999999999999 |
-| anyscale/meta-llama/Llama-2-70b-chat-hf | anyscale | 0.000002 |
-| anyscale/codellama/CodeLlama-34b-Instruct-hf | anyscale | 0.000002 |
-| gpt-3.5-turbo-1106 | openai | 0.000003 |
-| openrouter/meta-llama/llama-2-70b-chat | openrouter | 0.000003 |
-| amazon.titan-text-express-v1 | bedrock | 0.000003 |
-| gpt-3.5-turbo | openai | 0.0000035 |
-| gpt-3.5-turbo-0301 | openai | 0.0000035 |
-| gpt-3.5-turbo-0613 | openai | 0.0000035 |
-| gpt-3.5-turbo-instruct | text-completion-openai | 0.0000035 |
-| openrouter/openai/gpt-3.5-turbo | openrouter | 0.0000035 |
-| cohere.command-text-v14 | bedrock | 0.0000035 |
-| gpt-3.5-turbo-0613 | openai | 0.0000035 |
-| claude-instant-1 | anthropic | 0.00000714 |
-| claude-instant-1.2 | anthropic | 0.00000714 |
-| openrouter/anthropic/claude-instant-v1 | openrouter | 0.00000714 |
-| anthropic.claude-instant-v1 | bedrock | 0.00000714 |
-| openrouter/mancer/weaver | openrouter | 0.00001125 |
-| j2-mid | ai21 | 0.00002 |
-| ai21.j2-mid-v1 | bedrock | 0.000025 |
-| openrouter/jondurbin/airoboros-l2-70b-2.1 | openrouter | 0.00002775 |
-| command-nightly | cohere | 0.00003 |
-| command | cohere | 0.00003 |
-| command-light | cohere | 0.00003 |
-| command-medium-beta | cohere | 0.00003 |
-| command-xlarge-beta | cohere | 0.00003 |
-| command-r-plus| cohere | 0.000018 |
-| j2-ultra | ai21 | 0.00003 |
-| ai21.j2-ultra-v1 | bedrock | 0.0000376 |
-| gpt-4-1106-preview | openai | 0.00004 |
-| gpt-4-vision-preview | openai | 0.00004 |
-| claude-2 | anthropic | 0.0000437 |
-| openrouter/anthropic/claude-2 | openrouter | 0.0000437 |
-| anthropic.claude-v1 | bedrock | 0.0000437 |
-| anthropic.claude-v2 | bedrock | 0.0000437 |
-| gpt-4 | openai | 0.00009 |
-| gpt-4-0314 | openai | 0.00009 |
-| gpt-4-0613 | openai | 0.00009 |
-| openrouter/openai/gpt-4 | openrouter | 0.00009 |
-| gpt-4-32k | openai | 0.00018 |
-| gpt-4-32k-0314 | openai | 0.00018 |
-| gpt-4-32k-0613 | openai | 0.00018 |
-
-
-
-## Setup:
-```
-git clone https://github.com/BerriAI/litellm
-```
-cd to `benchmark` dir
-```
-cd litellm/cookbook/benchmark
-```
-
-### Install Dependencies
-```
-pip install litellm click tqdm tabulate termcolor
-```
-
-### Configuration
-In `benchmark/benchmark.py` select your LLMs, LLM API Key and questions
-
-Supported LLMs: https://docs.litellm.ai/docs/providers
-
-```python
-# Define the list of models to benchmark
-models = ['gpt-3.5-turbo', 'togethercomputer/llama-2-70b-chat', 'claude-2']
-
-# Enter LLM API keys
-os.environ['OPENAI_API_KEY'] = ""
-os.environ['ANTHROPIC_API_KEY'] = ""
-os.environ['TOGETHERAI_API_KEY'] = ""
-
-# List of questions to benchmark (replace with your questions)
-questions = [
- "When will BerriAI IPO?",
- "When will LiteLLM hit $100M ARR?"
-]
-
-```
-
-## Run LLM-Bench
-```
-python3 benchmark.py
-```
-
-## Expected Output
-```
-Running question: When will BerriAI IPO? for model: claude-2: 100%|āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā| 3/3 [00:13<00:00, 4.41s/it]
-
-Benchmark Results for 'When will BerriAI IPO?':
-+-----------------+----------------------------------------------------------------------------------+---------------------------+------------+
-| Model | Response | Response Time (seconds) | Cost ($) |
-+=================+==================================================================================+===========================+============+
-| gpt-3.5-turbo | As an AI language model, I cannot provide up-to-date information or predict | 1.55 seconds | $0.000122 |
-| | future events. It is best to consult a reliable financial source or contact | | |
-| | BerriAI directly for information regarding their IPO plans. | | |
-+-----------------+----------------------------------------------------------------------------------+---------------------------+------------+
-| togethercompute | I'm not able to provide information about future IPO plans or dates for BerriAI | 8.52 seconds | $0.000531 |
-| r/llama-2-70b-c | or any other company. IPO (Initial Public Offering) plans and timelines are | | |
-| hat | typically kept private by companies until they are ready to make a public | | |
-| | announcement. It's important to note that IPO plans can change and are subject | | |
-| | to various factors, such as market conditions, financial performance, and | | |
-| | regulatory approvals. Therefore, it's difficult to predict with certainty when | | |
-| | BerriAI or any other company will go public. If you're interested in staying | | |
-| | up-to-date with BerriAI's latest news and developments, you may want to follow | | |
-| | their official social media accounts, subscribe to their newsletter, or visit | | |
-| | their website periodically for updates. | | |
-+-----------------+----------------------------------------------------------------------------------+---------------------------+------------+
-| claude-2 | I do not have any information about when or if BerriAI will have an initial | 3.17 seconds | $0.002084 |
-| | public offering (IPO). As an AI assistant created by Anthropic to be helpful, | | |
-| | harmless, and honest, I do not have insider knowledge about Anthropic's business | | |
-| | plans or strategies. | | |
-+-----------------+----------------------------------------------------------------------------------+---------------------------+------------+
-```
-
-## Support
-**š¤ Schedule a 1-on-1 Session:** Book a [1-on-1 session](https://enterprise.litellm.ai/demo) with Krrish and Ishaan, the founders, to discuss any issues, provide feedback, or explore how we can improve LiteLLM for you.
diff --git a/cookbook/codellama-server/README.MD b/cookbook/codellama-server/README.MD
deleted file mode 100644
index 82a7e62f40a..00000000000
--- a/cookbook/codellama-server/README.MD
+++ /dev/null
@@ -1,153 +0,0 @@
-# CodeLlama Server: Streaming, Caching, Model Fallbacks (OpenAI + Anthropic), Prompt-tracking
-
-Works with: Anthropic, Huggingface, Cohere, TogetherAI, Azure, OpenAI, etc.
-
-[](https://pypi.org/project/litellm/)
-[](https://pypi.org/project/litellm/0.1.1/)
-
-
-[](https://railway.app/template/HuDPw-?referralCode=jch2ME)
-
-**LIVE DEMO** - https://litellm.ai/playground
-
-## What does CodeLlama Server do
-
-- Uses Together AI's CodeLlama to answer coding questions, with GPT-4 + Claude-2 as backups (you can easily switch this to any model from Huggingface, Replicate, Cohere, AI21, Azure, OpenAI, etc.)
-- Sets default system prompt for guardrails `system_prompt = "Only respond to questions about code. Say 'I don't know' to anything outside of that."`
-- Integrates with Promptlayer for model + prompt tracking
-- Example output
-
-
-
-- **Consistent Input/Output** Format
- - Call all models using the OpenAI format - `completion(model, messages)`
- - Text responses will always be available at `['choices'][0]['message']['content']`
- - Stream responses will always be available at `['choices'][0]['delta']['content']`
-- **Error Handling** Using Model Fallbacks (if `CodeLlama` fails, try `GPT-4`) with cooldowns, and retries
-- **Prompt Logging** - Log successful completions to promptlayer for testing + iterating on your prompts in production! (Learn more: https://litellm.readthedocs.io/en/latest/advanced/
-
- **Example: Logs sent to PromptLayer**
-
-
-
-
-- **Token Usage & Spend** - Track Input + Completion tokens used + Spend/model - https://docs.litellm.ai/docs/token_usage
-- **Caching** - Provides in-memory cache + GPT-Cache integration for more advanced usage - https://docs.litellm.ai/docs/caching/gpt_cache
-
-- **Streaming & Async Support** - Return generators to stream text responses - TEST IT š https://litellm.ai/
-
-## API Endpoints
-
-### `/chat/completions` (POST)
-
-This endpoint is used to generate chat completions for 50+ support LLM API Models. Use llama2, GPT-4, Claude2 etc
-
-#### Input
-
-This API endpoint accepts all inputs in raw JSON and expects the following inputs
-
-- `prompt` (string, required): The user's coding related question
-- Additional Optional parameters: `temperature`, `functions`, `function_call`, `top_p`, `n`, `stream`. See the full list of supported inputs here: https://litellm.readthedocs.io/en/latest/input/
-
-#### Example JSON body
-
-For claude-2
-
-```json
-{
- "prompt": "write me a function to print hello world"
-}
-```
-
-### Making an API request to the Code-Gen Server
-
-```python
-import requests
-import json
-
-url = "localhost:4000/chat/completions"
-
-payload = json.dumps({
- "prompt": "write me a function to print hello world"
-})
-headers = {
- 'Content-Type': 'application/json'
-}
-
-response = requests.request("POST", url, headers=headers, data=payload)
-
-print(response.text)
-
-```
-
-### Output [Response Format]
-
-Responses from the server are given in the following format.
-All responses from the server are returned in the following format (for all LLM models). More info on output here: https://litellm.readthedocs.io/en/latest/output/
-
-```json
-{
- "choices": [
- {
- "finish_reason": "stop",
- "index": 0,
- "message": {
- "content": ".\n\n```\ndef print_hello_world():\n print(\"hello world\")\n",
- "role": "assistant"
- }
- }
- ],
- "created": 1693279694.6474009,
- "model": "togethercomputer/CodeLlama-34b-Instruct",
- "usage": {
- "completion_tokens": 14,
- "prompt_tokens": 28,
- "total_tokens": 42
- }
-}
-```
-
-## Installation & Usage
-
-### Running Locally
-
-1. Clone liteLLM repository to your local machine:
- ```
- git clone https://github.com/BerriAI/litellm-CodeLlama-server
- ```
-2. Install the required dependencies using pip
- ```
- pip install requirements.txt
- ```
-3. Set your LLM API keys
- ```
- os.environ['OPENAI_API_KEY]` = "YOUR_API_KEY"
- or
- set OPENAI_API_KEY in your .env file
- ```
-4. Run the server:
- ```
- python main.py
- ```
-
-## Deploying
-
-1. Quick Start: Deploy on Railway
-
- [](https://railway.app/template/HuDPw-?referralCode=jch2ME)
-
-2. `GCP`, `AWS`, `Azure`
- This project includes a `Dockerfile` allowing you to build and deploy a Docker Project on your providers
-
-# Support / Talk with founders
-
-- [Our calendar š](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
-- [Community Discord š](https://discord.gg/wuPM9dRgDw)
-- Our emails āļø ishaan@berri.ai / krrish@berri.ai
-
-## Roadmap
-
-- [ ] Implement user-based rate-limiting
-- [ ] Spending controls per project - expose key creation endpoint
-- [ ] Need to store a keys db -> mapping created keys to their alias (i.e. project name)
-- [ ] Easily add new models as backups / as the entry-point (add this to the available model list)
diff --git a/cookbook/codellama-server/imgs/code-output.png b/cookbook/codellama-server/imgs/code-output.png
deleted file mode 100644
index 67e298bd3be..00000000000
Binary files a/cookbook/codellama-server/imgs/code-output.png and /dev/null differ
diff --git a/cookbook/codellama-server/imgs/promptlayer_logging.png b/cookbook/codellama-server/imgs/promptlayer_logging.png
deleted file mode 100644
index 26b046ac448..00000000000
Binary files a/cookbook/codellama-server/imgs/promptlayer_logging.png and /dev/null differ
diff --git a/cookbook/codellama-server/main.py b/cookbook/codellama-server/main.py
deleted file mode 100644
index d05d6752300..00000000000
--- a/cookbook/codellama-server/main.py
+++ /dev/null
@@ -1,102 +0,0 @@
-import traceback
-from flask import Flask, request, Response
-from flask_cors import CORS
-import litellm
-from util import handle_error
-from litellm import completion
-import os
-import dotenv
-import time
-import json
-
-dotenv.load_dotenv()
-
-# TODO: set your keys in .env or here:
-# os.environ["OPENAI_API_KEY"] = "" # set your openai key here
-# os.environ["ANTHROPIC_API_KEY"] = "" # set your anthropic key here
-# os.environ["TOGETHER_AI_API_KEY"] = "" # set your together ai key here
-# see supported models / keys here: https://litellm.readthedocs.io/en/latest/supported/
-######### ENVIRONMENT VARIABLES ##########
-verbose = True
-
-# litellm.caching_with_models = True # CACHING: caching_with_models Keys in the cache are messages + model. - to learn more: https://docs.litellm.ai/docs/caching/
-######### PROMPT LOGGING ##########
-os.environ["PROMPTLAYER_API_KEY"] = (
- "" # set your promptlayer key here - https://promptlayer.com/
-)
-
-# set callbacks
-litellm.success_callback = ["promptlayer"]
-############ HELPER FUNCTIONS ###################################
-
-
-def print_verbose(print_statement):
- if verbose:
- print(print_statement)
-
-
-app = Flask(__name__)
-CORS(app)
-
-
-@app.route("/")
-def index():
- return "received!", 200
-
-
-def data_generator(response):
- for chunk in response:
- yield f"data: {json.dumps(chunk)}\n\n"
-
-
-@app.route("/chat/completions", methods=["POST"])
-def api_completion():
- data = request.json
- start_time = time.time()
- if data.get("stream") == "True":
- data["stream"] = True # convert to boolean
- try:
- if "prompt" not in data:
- raise ValueError("data needs to have prompt")
- data["model"] = (
- "togethercomputer/CodeLlama-34b-Instruct" # by default use Together AI's CodeLlama model - https://api.together.xyz/playground/chat?model=togethercomputer%2FCodeLlama-34b-Instruct
- )
- # COMPLETION CALL
- system_prompt = "Only respond to questions about code. Say 'I don't know' to anything outside of that."
- messages = [
- {"role": "system", "content": system_prompt},
- {"role": "user", "content": data.pop("prompt")},
- ]
- data["messages"] = messages
- print(f"data: {data}")
- response = completion(**data)
- ## LOG SUCCESS
- end_time = time.time()
- if (
- "stream" in data and data["stream"] == True
- ): # use generate_responses to stream responses
- return Response(data_generator(response), mimetype="text/event-stream")
- except Exception:
- # call handle_error function
- print_verbose(f"Got Error api_completion(): {traceback.format_exc()}")
- ## LOG FAILURE
- end_time = time.time()
- traceback_exception = traceback.format_exc()
- return handle_error(data=data)
- return response
-
-
-@app.route("/get_models", methods=["POST"])
-def get_models():
- try:
- return litellm.model_list
- except Exception as e:
- traceback.print_exc()
- response = {"error": str(e)}
- return response, 200
-
-
-if __name__ == "__main__":
- from waitress import serve
-
- serve(app, host="0.0.0.0", port=4000, threads=500)
diff --git a/cookbook/community-resources/get_hf_models.py b/cookbook/community-resources/get_hf_models.py
deleted file mode 100644
index 8c75a241227..00000000000
--- a/cookbook/community-resources/get_hf_models.py
+++ /dev/null
@@ -1,89 +0,0 @@
-import requests
-
-
-def get_next_url(response):
- """
- Function to get 'next' url from Link header
- :param response: response from requests
- :return: next url or None
- """
- if "link" not in response.headers:
- return None
- headers = response.headers
-
- next_url = headers["Link"]
- print(next_url)
- start_index = next_url.find("<")
- end_index = next_url.find(">")
-
- return next_url[1:end_index]
-
-
-def get_models(url):
- """
- Function to retrieve all models from paginated endpoint
- :param url: base url to make GET request
- :return: list of all models
- """
- models = []
- while url:
- response = requests.get(url)
- if response.status_code != 200:
- print(f"Failed to retrieve data. Status code: {response.status_code}")
- return models
- payload = response.json()
- url = get_next_url(response)
- models.extend(payload)
- return models
-
-
-def get_cleaned_models(models):
- """
- Function to clean retrieved models
- :param models: list of retrieved models
- :return: list of cleaned models
- """
- cleaned_models = []
- for model in models:
- cleaned_models.append(model["id"])
- return cleaned_models
-
-
-# Get text-generation models
-url = "https://huggingface.co/api/models?filter=text-generation-inference"
-text_generation_models = get_models(url)
-cleaned_text_generation_models = get_cleaned_models(text_generation_models)
-
-print(cleaned_text_generation_models)
-
-
-# Get conversational models
-url = "https://huggingface.co/api/models?filter=conversational"
-conversational_models = get_models(url)
-cleaned_conversational_models = get_cleaned_models(conversational_models)
-
-print(cleaned_conversational_models)
-
-
-def write_to_txt(cleaned_models, filename):
- """
- Function to write the contents of a list to a text file
- :param cleaned_models: list of cleaned models
- :param filename: name of the text file
- """
- with open(filename, "w") as f:
- for item in cleaned_models:
- f.write("%s\n" % item)
-
-
-# Write contents of cleaned_text_generation_models to text_generation_models.txt
-write_to_txt(
- cleaned_text_generation_models,
- "huggingface_llms_metadata/hf_text_generation_models.txt",
-)
-
-# Write contents of cleaned_conversational_models to conversational_models.txt
-write_to_txt(
- cleaned_conversational_models,
- "huggingface_llms_metadata/hf_conversational_models.txt",
-)
diff --git a/cookbook/community-resources/max_tokens.json b/cookbook/community-resources/max_tokens.json
deleted file mode 100644
index 289f8faf9e3..00000000000
--- a/cookbook/community-resources/max_tokens.json
+++ /dev/null
@@ -1,93 +0,0 @@
-{
- "gpt-3.5-turbo": {
- "max_tokens": 4000,
- "input_cost_per_token": 0.0000015,
- "output_cost_per_token": 0.000002
- },
- "gpt-3.5-turbo-0613": {
- "max_tokens": 4000,
- "input_cost_per_token": 0.0000015,
- "output_cost_per_token": 0.000002
- },
- "gpt-3.5-turbo-0301": {
- "max_tokens": 4000,
- "input_cost_per_token": 0.0000015,
- "output_cost_per_token": 0.000002
- },
- "gpt-3.5-turbo-16k": {
- "max_tokens": 16000,
- "input_cost_per_token": 0.000003,
- "output_cost_per_token": 0.000004
- },
- "gpt-3.5-turbo-16k-0613": {
- "max_tokens": 16000,
- "input_cost_per_token": 0.000003,
- "output_cost_per_token": 0.000004
- },
- "gpt-4": {
- "max_tokens": 8000,
- "input_cost_per_token": 0.000003,
- "output_cost_per_token": 0.00006
- },
- "gpt-4-0613": {
- "max_tokens": 8000,
- "input_cost_per_token": 0.000003,
- "output_cost_per_token": 0.00006
- },
- "gpt-4-32k": {
- "max_tokens": 8000,
- "input_cost_per_token": 0.00006,
- "output_cost_per_token": 0.00012
- },
- "claude-instant-1": {
- "max_tokens": 100000,
- "input_cost_per_token": 0.00000163,
- "output_cost_per_token": 0.00000551
- },
- "claude-2": {
- "max_tokens": 100000,
- "input_cost_per_token": 0.00001102,
- "output_cost_per_token": 0.00003268
- },
- "text-bison-001": {
- "max_tokens": 8192,
- "input_cost_per_token": 0.000004,
- "output_cost_per_token": 0.000004
- },
- "chat-bison-001": {
- "max_tokens": 4096,
- "input_cost_per_token": 0.000002,
- "output_cost_per_token": 0.000002
- },
- "command-nightly": {
- "max_tokens": 4096,
- "input_cost_per_token": 0.000015,
- "output_cost_per_token": 0.000015
- },
- "replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1": {
- "max_tokens": 4096,
- "input_cost_per_token": 0.00000608,
- "output_cost_per_token": 0.00000608
- },
- "together-ai-up-to-3b": {
- "input_cost_per_token": 0.0000001,
- "output_cost_per_token": 0.0000001
- },
- "together-ai-3.1b-7b": {
- "input_cost_per_token": 0.0000002,
- "output_cost_per_token": 0.0000002
- },
- "together-ai-7.1b-20b": {
- "max_tokens": 1000,
- "input_cost_per_token": 0.0000004,
- "output_cost_per_token": 0.0000004
- },
- "together-ai-20.1b-40b": {
- "input_cost_per_token": 0.000001,
- "output_cost_per_token": 0.000001
- },
- "together-ai-40.1b-70b": {
- "input_cost_per_token": 0.000003,
- "output_cost_per_token": 0.000003
- }
-}
diff --git a/cookbook/gollem_go_agent_framework/README.md b/cookbook/gollem_go_agent_framework/README.md
deleted file mode 100644
index 729f985d086..00000000000
--- a/cookbook/gollem_go_agent_framework/README.md
+++ /dev/null
@@ -1,119 +0,0 @@
-# Gollem Go Agent Framework with LiteLLM
-
-A working example showing how to use [gollem](https://github.com/fugue-labs/gollem), a production-grade Go agent framework, with LiteLLM as a proxy gateway. This lets Go developers access 100+ LLM providers through a single proxy while keeping compile-time type safety for tools and structured output.
-
-## Quick Start
-
-### 1. Start LiteLLM Proxy
-
-```bash
-# Simple start with a single model
-litellm --model gpt-4o
-
-# Or with the example config for multi-provider access
-litellm --config proxy_config.yaml
-```
-
-### 2. Run the examples
-
-```bash
-# Install Go dependencies
-go mod tidy
-
-# Basic agent
-go run ./basic
-
-# Agent with type-safe tools
-go run ./tools
-
-# Streaming responses
-go run ./streaming
-```
-
-## Configuration
-
-The included `proxy_config.yaml` sets up three providers through LiteLLM:
-
-```yaml
-model_list:
- - model_name: gpt-4o # OpenAI
- - model_name: claude-sonnet # Anthropic
- - model_name: gemini-pro # Google Vertex AI
-```
-
-Switch providers in Go by changing a single string ā no code changes needed:
-
-```go
-model := openai.NewLiteLLM("http://localhost:4000",
- openai.WithModel("gpt-4o"), // OpenAI
- // openai.WithModel("claude-sonnet"), // Anthropic
- // openai.WithModel("gemini-pro"), // Google
-)
-```
-
-## Examples
-
-### `basic/` ā Basic Agent
-
-Connects gollem to LiteLLM and runs a simple prompt. Demonstrates the `NewLiteLLM` constructor and basic agent creation.
-
-### `tools/` ā Type-Safe Tools
-
-Shows gollem's compile-time type-safe tool framework working through LiteLLM's tool-use passthrough. The tool parameters are Go structs with JSON tags ā the schema is generated automatically at compile time.
-
-### `streaming/` ā Streaming Responses
-
-Real-time token streaming using Go 1.23+ range-over-function iterators, proxied through LiteLLM's SSE passthrough.
-
-## How It Works
-
-Gollem's `openai.NewLiteLLM()` constructor creates an OpenAI-compatible provider pointed at your LiteLLM proxy. Since LiteLLM speaks the OpenAI API protocol, everything works out of the box:
-
-- **Chat completions** ā standard request/response
-- **Tool use** ā LiteLLM passes tool definitions and calls through transparently
-- **Streaming** ā Server-Sent Events proxied through LiteLLM
-- **Structured output** ā JSON schema response format works with supporting models
-
-```
-Go App (gollem) ā LiteLLM Proxy ā OpenAI / Anthropic / Google / ...
-```
-
-## Why Use This?
-
-- **Type-safe Go**: Compile-time type checking for tools, structured output, and agent configuration ā no runtime surprises
-- **Single proxy, many models**: Switch between OpenAI, Anthropic, Google, and 100+ other providers by changing a model name string
-- **Zero-dependency core**: gollem's core has no external dependencies ā just stdlib
-- **Single binary deployment**: `go build` produces one binary, no pip/venv/Docker needed
-- **Cost tracking & rate limiting**: LiteLLM handles cost tracking, rate limits, and fallbacks at the proxy layer
-
-## Environment Variables
-
-```bash
-# Required for providers you want to use (set in LiteLLM config or env)
-export OPENAI_API_KEY="sk-..."
-export ANTHROPIC_API_KEY="sk-ant-..."
-
-# Optional: point to a non-default LiteLLM proxy
-export LITELLM_PROXY_URL="http://localhost:4000"
-```
-
-## Troubleshooting
-
-**Connection errors?**
-- Make sure LiteLLM is running: `litellm --model gpt-4o`
-- Check the URL is correct (default: `http://localhost:4000`)
-
-**Model not found?**
-- Verify the model name matches what's configured in LiteLLM
-- Run `curl http://localhost:4000/models` to see available models
-
-**Tool calls not working?**
-- Ensure the underlying model supports tool use (GPT-4o, Claude, Gemini)
-- Check LiteLLM logs for any provider-specific errors
-
-## Learn More
-
-- [gollem GitHub](https://github.com/fugue-labs/gollem)
-- [gollem API Reference](https://pkg.go.dev/github.com/fugue-labs/gollem/core)
-- [LiteLLM Proxy Docs](https://docs.litellm.ai/docs/simple_proxy)
-- [LiteLLM Supported Models](https://docs.litellm.ai/docs/providers)
diff --git a/cookbook/gollem_go_agent_framework/basic/main.go b/cookbook/gollem_go_agent_framework/basic/main.go
deleted file mode 100644
index 838149a8ff9..00000000000
--- a/cookbook/gollem_go_agent_framework/basic/main.go
+++ /dev/null
@@ -1,41 +0,0 @@
-// Basic gollem agent connected to a LiteLLM proxy.
-//
-// Usage:
-//
-// litellm --model gpt-4o # start proxy in another terminal
-// go run ./basic
-package main
-
-import (
- "context"
- "fmt"
- "log"
- "os"
-
- "github.com/fugue-labs/gollem/core"
- "github.com/fugue-labs/gollem/provider/openai"
-)
-
-func main() {
- proxyURL := "http://localhost:4000"
- if u := os.Getenv("LITELLM_PROXY_URL"); u != "" {
- proxyURL = u
- }
-
- // Connect to LiteLLM proxy. NewLiteLLM creates an OpenAI-compatible
- // provider pointed at the given URL.
- model := openai.NewLiteLLM(proxyURL,
- openai.WithModel("gpt-4o"), // any model name configured in LiteLLM
- )
-
- // Create and run a simple agent.
- agent := core.NewAgent[string](model,
- core.WithSystemPrompt[string]("You are a helpful assistant. Be concise."),
- )
-
- result, err := agent.Run(context.Background(), "Explain quantum computing in two sentences.")
- if err != nil {
- log.Fatal(err)
- }
- fmt.Println(result.Output)
-}
diff --git a/cookbook/gollem_go_agent_framework/go.mod b/cookbook/gollem_go_agent_framework/go.mod
deleted file mode 100644
index a8dc9365d7f..00000000000
--- a/cookbook/gollem_go_agent_framework/go.mod
+++ /dev/null
@@ -1,5 +0,0 @@
-module github.com/BerriAI/litellm/cookbook/gollem_go_agent_framework
-
-go 1.26.3
-
-require github.com/fugue-labs/gollem v0.1.0
diff --git a/cookbook/gollem_go_agent_framework/go.sum b/cookbook/gollem_go_agent_framework/go.sum
deleted file mode 100644
index 1eb6c5ac9fc..00000000000
--- a/cookbook/gollem_go_agent_framework/go.sum
+++ /dev/null
@@ -1,2 +0,0 @@
-github.com/fugue-labs/gollem v0.1.0 h1:QexYnvkb44QZFEljgAePqMIGZjgsbk0Y5GJ2jYYgfa8=
-github.com/fugue-labs/gollem v0.1.0/go.mod h1:htW1YO81uysSKVOkYJtxhGCFrzm+36HBFxEWuECoHKQ=
diff --git a/cookbook/gollem_go_agent_framework/proxy_config.yaml b/cookbook/gollem_go_agent_framework/proxy_config.yaml
deleted file mode 100644
index 18265a002bc..00000000000
--- a/cookbook/gollem_go_agent_framework/proxy_config.yaml
+++ /dev/null
@@ -1,16 +0,0 @@
-model_list:
- - model_name: gpt-4o
- litellm_params:
- model: openai/gpt-4o
- api_key: os.environ/OPENAI_API_KEY
-
- - model_name: claude-sonnet
- litellm_params:
- model: anthropic/claude-sonnet-4-20250514
- api_key: os.environ/ANTHROPIC_API_KEY
-
- - model_name: gemini-pro
- litellm_params:
- model: vertex_ai/gemini-2.0-flash
- vertex_project: my-project
- vertex_location: us-central1
diff --git a/cookbook/gollem_go_agent_framework/streaming/main.go b/cookbook/gollem_go_agent_framework/streaming/main.go
deleted file mode 100644
index 42bc9bbe34a..00000000000
--- a/cookbook/gollem_go_agent_framework/streaming/main.go
+++ /dev/null
@@ -1,56 +0,0 @@
-// Streaming responses from gollem through LiteLLM.
-//
-// Uses Go 1.23+ range-over-function iterators for real-time token
-// streaming via LiteLLM's SSE passthrough.
-//
-// Usage:
-//
-// litellm --model gpt-4o
-// go run ./streaming
-package main
-
-import (
- "context"
- "fmt"
- "log"
- "os"
-
- "github.com/fugue-labs/gollem/core"
- "github.com/fugue-labs/gollem/provider/openai"
-)
-
-func main() {
- proxyURL := "http://localhost:4000"
- if u := os.Getenv("LITELLM_PROXY_URL"); u != "" {
- proxyURL = u
- }
-
- model := openai.NewLiteLLM(proxyURL,
- openai.WithModel("gpt-4o"),
- )
-
- agent := core.NewAgent[string](model)
-
- // RunStream returns a streaming result that yields tokens as they arrive.
- stream, err := agent.RunStream(context.Background(), "Write a haiku about distributed systems")
- if err != nil {
- log.Fatal(err)
- }
-
- // StreamText yields text chunks in real-time.
- // The boolean argument controls whether deltas (true) or accumulated
- // text (false) is returned.
- fmt.Print("Response: ")
- for text, err := range stream.StreamText(true) {
- if err != nil {
- log.Fatal(err)
- }
- fmt.Print(text)
- }
- fmt.Println()
-
- // After streaming completes, the final response is available.
- resp := stream.Response()
- fmt.Printf("\nTokens used: input=%d, output=%d\n",
- resp.Usage.InputTokens, resp.Usage.OutputTokens)
-}
diff --git a/cookbook/gollem_go_agent_framework/tools/main.go b/cookbook/gollem_go_agent_framework/tools/main.go
deleted file mode 100644
index ed41a95ffef..00000000000
--- a/cookbook/gollem_go_agent_framework/tools/main.go
+++ /dev/null
@@ -1,64 +0,0 @@
-// Gollem agent with type-safe tools through LiteLLM.
-//
-// The tool parameters are Go structs ā gollem generates the JSON schema
-// automatically at compile time. LiteLLM passes tool definitions through
-// transparently to the underlying provider.
-//
-// Usage:
-//
-// litellm --model gpt-4o
-// go run ./tools
-package main
-
-import (
- "context"
- "fmt"
- "log"
- "os"
-
- "github.com/fugue-labs/gollem/core"
- "github.com/fugue-labs/gollem/provider/openai"
-)
-
-// WeatherParams defines the tool's input schema via struct tags.
-// The JSON schema is generated at compile time ā no runtime reflection needed.
-type WeatherParams struct {
- City string `json:"city" description:"City name to get weather for"`
- Unit string `json:"unit,omitempty" description:"Temperature unit: celsius or fahrenheit"`
-}
-
-func main() {
- proxyURL := "http://localhost:4000"
- if u := os.Getenv("LITELLM_PROXY_URL"); u != "" {
- proxyURL = u
- }
-
- model := openai.NewLiteLLM(proxyURL,
- openai.WithModel("gpt-4o"),
- )
-
- // Define a type-safe tool. The function signature enforces correct types.
- weatherTool := core.FuncTool[WeatherParams](
- "get_weather",
- "Get current weather for a city",
- func(ctx context.Context, p WeatherParams) (string, error) {
- unit := p.Unit
- if unit == "" {
- unit = "fahrenheit"
- }
- // In production, call a real weather API here.
- return fmt.Sprintf("Weather in %s: 72Ā°F (22Ā°C), sunny", p.City), nil
- },
- )
-
- agent := core.NewAgent[string](model,
- core.WithTools[string](weatherTool),
- core.WithSystemPrompt[string]("You are a helpful weather assistant. Use the get_weather tool to answer weather questions."),
- )
-
- result, err := agent.Run(context.Background(), "What's the weather like in San Francisco and Tokyo?")
- if err != nil {
- log.Fatal(err)
- }
- fmt.Println(result.Output)
-}
diff --git a/cookbook/google_adk_litellm_tutorial.ipynb b/cookbook/google_adk_litellm_tutorial.ipynb
deleted file mode 100644
index 27914edbba8..00000000000
--- a/cookbook/google_adk_litellm_tutorial.ipynb
+++ /dev/null
@@ -1,412 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "7aa8875d",
- "metadata": {},
- "source": [
- "# Google ADK with LiteLLM\n",
- "\n",
- "Use Google ADK with LiteLLM Python SDK, LiteLLM Proxy.\n",
- "\n",
- "This tutorial shows you how to create intelligent agents using Agent Development Kit (ADK) with support for multiple Large Language Model (LLM) providers through LiteLLM."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a4d249c3",
- "metadata": {},
- "source": [
- "## Overview\n",
- "\n",
- "ADK (Agent Development Kit) allows you to build intelligent agents powered by LLMs. By integrating with LiteLLM, you can:\n",
- "\n",
- "- Use multiple LLM providers (OpenAI, Anthropic, Google, etc.)\n",
- "- Switch easily between models from different providers\n",
- "- Connect to a LiteLLM proxy for centralized model management"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a0bbb56b",
- "metadata": {},
- "source": [
- "## Prerequisites\n",
- "\n",
- "- Python environment setup\n",
- "- API keys for model providers (OpenAI, Anthropic, Google AI Studio)\n",
- "- Basic understanding of LLMs and agent concepts"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "7fee50a8",
- "metadata": {},
- "source": [
- "## Installation"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "44106a23",
- "metadata": {},
- "outputs": [],
- "source": [
- "# Install dependencies\n",
- "!pip install google-adk litellm"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2171740a",
- "metadata": {},
- "source": [
- "## 1. Setting Up Environment"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "6695807e",
- "metadata": {},
- "outputs": [],
- "source": [
- "# Setup environment and API keys\n",
- "import os\n",
- "import asyncio\n",
- "from google.adk.agents import Agent\n",
- "from google.adk.models.lite_llm import LiteLlm # For multi-model support\n",
- "from google.adk.sessions import InMemorySessionService\n",
- "from google.adk.runners import Runner\n",
- "from google.genai import types\n",
- "import litellm # Import for proxy configuration\n",
- "\n",
- "# Set your API keys\n",
- "os.environ['GOOGLE_API_KEY'] = 'your-google-api-key' # For Gemini models\n",
- "os.environ['OPENAI_API_KEY'] = 'your-openai-api-key' # For OpenAI models\n",
- "os.environ['ANTHROPIC_API_KEY'] = 'your-anthropic-api-key' # For Claude models\n",
- "\n",
- "# Define model constants for cleaner code\n",
- "MODEL_GEMINI_PRO = 'gemini-1.5-pro'\n",
- "MODEL_GPT_4O = 'openai/gpt-4o'\n",
- "MODEL_CLAUDE_SONNET = 'anthropic/claude-3-sonnet-20240229'"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d2b1ed59",
- "metadata": {},
- "source": [
- "## 2. Define a Simple Tool"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "04b3ef5b",
- "metadata": {},
- "outputs": [],
- "source": [
- "# Weather tool implementation\n",
- "def get_weather(city: str) -> dict:\n",
- " \"\"\"Retrieves the current weather report for a specified city.\"\"\"\n",
- " print(f'Tool: get_weather called for city: {city}')\n",
- "\n",
- " # Mock weather data\n",
- " mock_weather_db = {\n",
- " 'newyork': {\n",
- " 'status': 'success',\n",
- " 'report': 'The weather in New York is sunny with a temperature of 25Ā°C.'\n",
- " },\n",
- " 'london': {\n",
- " 'status': 'success',\n",
- " 'report': \"It's cloudy in London with a temperature of 15Ā°C.\"\n",
- " },\n",
- " 'tokyo': {\n",
- " 'status': 'success',\n",
- " 'report': 'Tokyo is experiencing light rain and a temperature of 18Ā°C.'\n",
- " },\n",
- " }\n",
- "\n",
- " city_normalized = city.lower().replace(' ', '')\n",
- "\n",
- " if city_normalized in mock_weather_db:\n",
- " return mock_weather_db[city_normalized]\n",
- " else:\n",
- " return {\n",
- " 'status': 'error',\n",
- " 'error_message': f\"Sorry, I don't have weather information for '{city}'.\"\n",
- " }"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "727b15c9",
- "metadata": {},
- "source": [
- "## 3. Helper Function for Agent Interaction"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f77449bf",
- "metadata": {},
- "outputs": [],
- "source": [
- "# Agent interaction helper function\n",
- "async def call_agent_async(query: str, runner, user_id, session_id):\n",
- " \"\"\"Sends a query to the agent and prints the final response.\"\"\"\n",
- " print(f'\\n>>> User Query: {query}')\n",
- "\n",
- " content = types.Content(role='user', parts=[types.Part(text=query)])\n",
- " final_response_text = 'Agent did not produce a final response.'\n",
- "\n",
- " async for event in runner.run_async(\n",
- " user_id=user_id,\n",
- " session_id=session_id,\n",
- " new_message=content\n",
- " ):\n",
- " if event.is_final_response():\n",
- " if event.content and event.content.parts:\n",
- " final_response_text = event.content.parts[0].text\n",
- " break\n",
- " print(f'<<< Agent Response: {final_response_text}')"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "0ac87987",
- "metadata": {},
- "source": [
- "## 4. Using Different Model Providers with ADK\n",
- "\n",
- "### 4.1 Using OpenAI Models"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "e167d557",
- "metadata": {},
- "outputs": [],
- "source": [
- "# OpenAI model implementation\n",
- "weather_agent_gpt = Agent(\n",
- " name='weather_agent_gpt',\n",
- " model=LiteLlm(model=MODEL_GPT_4O),\n",
- " description='Provides weather information using OpenAI\\'s GPT.',\n",
- " instruction=(\n",
- " 'You are a helpful weather assistant powered by GPT-4o. '\n",
- " \"Use the 'get_weather' tool for city weather requests. \"\n",
- " 'Present information clearly.'\n",
- " ),\n",
- " tools=[get_weather],\n",
- ")\n",
- "\n",
- "session_service_gpt = InMemorySessionService()\n",
- "session_gpt = session_service_gpt.create_session(\n",
- " app_name='weather_app', user_id='user_1', session_id='session_gpt'\n",
- ")\n",
- "\n",
- "runner_gpt = Runner(\n",
- " agent=weather_agent_gpt,\n",
- " app_name='weather_app',\n",
- " session_service=session_service_gpt,\n",
- ")\n",
- "\n",
- "async def test_gpt_agent():\n",
- " print('\\n--- Testing GPT Agent ---')\n",
- " await call_agent_async(\n",
- " \"What's the weather in London?\",\n",
- " runner=runner_gpt,\n",
- " user_id='user_1',\n",
- " session_id='session_gpt',\n",
- " )\n",
- "\n",
- "# To execute in a notebook cell:\n",
- "# await test_gpt_agent()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f9cb0613",
- "metadata": {},
- "source": [
- "### 4.2 Using Anthropic Models"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "1c653665",
- "metadata": {},
- "outputs": [],
- "source": [
- "# Anthropic model implementation\n",
- "weather_agent_claude = Agent(\n",
- " name='weather_agent_claude',\n",
- " model=LiteLlm(model=MODEL_CLAUDE_SONNET),\n",
- " description='Provides weather information using Anthropic\\'s Claude.',\n",
- " instruction=(\n",
- " 'You are a helpful weather assistant powered by Claude Sonnet. '\n",
- " \"Use the 'get_weather' tool for city weather requests. \"\n",
- " 'Present information clearly.'\n",
- " ),\n",
- " tools=[get_weather],\n",
- ")\n",
- "\n",
- "session_service_claude = InMemorySessionService()\n",
- "session_claude = session_service_claude.create_session(\n",
- " app_name='weather_app', user_id='user_1', session_id='session_claude'\n",
- ")\n",
- "\n",
- "runner_claude = Runner(\n",
- " agent=weather_agent_claude,\n",
- " app_name='weather_app',\n",
- " session_service=session_service_claude,\n",
- ")\n",
- "\n",
- "async def test_claude_agent():\n",
- " print('\\n--- Testing Claude Agent ---')\n",
- " await call_agent_async(\n",
- " \"What's the weather in Tokyo?\",\n",
- " runner=runner_claude,\n",
- " user_id='user_1',\n",
- " session_id='session_claude',\n",
- " )\n",
- "\n",
- "# To execute in a notebook cell:\n",
- "# await test_claude_agent()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "bf9d863b",
- "metadata": {},
- "source": [
- "### 4.3 Using Google's Gemini Models"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "83f49d0a",
- "metadata": {},
- "outputs": [],
- "source": [
- "# Gemini model implementation\n",
- "weather_agent_gemini = Agent(\n",
- " name='weather_agent_gemini',\n",
- " model=MODEL_GEMINI_PRO,\n",
- " description='Provides weather information using Google\\'s Gemini.',\n",
- " instruction=(\n",
- " 'You are a helpful weather assistant powered by Gemini Pro. '\n",
- " \"Use the 'get_weather' tool for city weather requests. \"\n",
- " 'Present information clearly.'\n",
- " ),\n",
- " tools=[get_weather],\n",
- ")\n",
- "\n",
- "session_service_gemini = InMemorySessionService()\n",
- "session_gemini = session_service_gemini.create_session(\n",
- " app_name='weather_app', user_id='user_1', session_id='session_gemini'\n",
- ")\n",
- "\n",
- "runner_gemini = Runner(\n",
- " agent=weather_agent_gemini,\n",
- " app_name='weather_app',\n",
- " session_service=session_service_gemini,\n",
- ")\n",
- "\n",
- "async def test_gemini_agent():\n",
- " print('\\n--- Testing Gemini Agent ---')\n",
- " await call_agent_async(\n",
- " \"What's the weather in New York?\",\n",
- " runner=runner_gemini,\n",
- " user_id='user_1',\n",
- " session_id='session_gemini',\n",
- " )\n",
- "\n",
- "# To execute in a notebook cell:\n",
- "# await test_gemini_agent()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "93bc5fd0",
- "metadata": {},
- "source": [
- "## 5. Using LiteLLM Proxy with ADK"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b4275151",
- "metadata": {},
- "source": [
- "| Variable | Description |\n",
- "|----------|-------------|\n",
- "| `LITELLM_PROXY_API_KEY` | The API key for the LiteLLM proxy |\n",
- "| `LITELLM_PROXY_API_BASE` | The base URL for the LiteLLM proxy |\n",
- "| `USE_LITELLM_PROXY` or `litellm.use_litellm_proxy` | When set to True, your request will be sent to LiteLLM proxy. |"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "256530a6",
- "metadata": {},
- "outputs": [],
- "source": [
- "# LiteLLM proxy integration\n",
- "os.environ['LITELLM_PROXY_API_KEY'] = 'your-litellm-proxy-api-key'\n",
- "os.environ['LITELLM_PROXY_API_BASE'] = 'your-litellm-proxy-url' # e.g., 'http://localhost:4000'\n",
- "litellm.use_litellm_proxy = True\n",
- "\n",
- "weather_agent_proxy_env = Agent(\n",
- " name='weather_agent_proxy_env',\n",
- " model=LiteLlm(model='gpt-4o'),\n",
- " description='Provides weather information using a model from LiteLLM proxy.',\n",
- " instruction=(\n",
- " 'You are a helpful weather assistant. '\n",
- " \"Use the 'get_weather' tool for city weather requests. \"\n",
- " 'Present information clearly.'\n",
- " ),\n",
- " tools=[get_weather],\n",
- ")\n",
- "\n",
- "session_service_proxy_env = InMemorySessionService()\n",
- "session_proxy_env = session_service_proxy_env.create_session(\n",
- " app_name='weather_app', user_id='user_1', session_id='session_proxy_env'\n",
- ")\n",
- "\n",
- "runner_proxy_env = Runner(\n",
- " agent=weather_agent_proxy_env,\n",
- " app_name='weather_app',\n",
- " session_service=session_service_proxy_env,\n",
- ")\n",
- "\n",
- "async def test_proxy_env_agent():\n",
- " print('\\n--- Testing Proxy-enabled Agent (Environment Variables) ---')\n",
- " await call_agent_async(\n",
- " \"What's the weather in London?\",\n",
- " runner=runner_proxy_env,\n",
- " user_id='user_1',\n",
- " session_id='session_proxy_env',\n",
- " )\n",
- "\n",
- "# To execute in a notebook cell:\n",
- "# await test_proxy_env_agent()"
- ]
- }
- ],
- "metadata": {
- "language_info": {
- "name": "python"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
diff --git a/cookbook/gpt_realtime_translate.py b/cookbook/gpt_realtime_translate.py
deleted file mode 100644
index 827c65cdbfe..00000000000
--- a/cookbook/gpt_realtime_translate.py
+++ /dev/null
@@ -1,245 +0,0 @@
-#!/usr/bin/env python3
-
-import argparse
-import asyncio
-import base64
-import json
-import os
-import sys
-import wave
-from collections.abc import Iterator, Sequence
-from dataclasses import dataclass
-from pathlib import Path
-from urllib.parse import urlencode, urlsplit, urlunsplit
-
-import websockets
-from websockets.asyncio.client import ClientConnection
-
-SAMPLE_RATE = 24_000
-CHANNELS = 1
-SAMPLE_WIDTH = 2
-CHUNK_DURATION_SECONDS = 0.1
-CHUNK_BYTES = int(SAMPLE_RATE * CHANNELS * SAMPLE_WIDTH * CHUNK_DURATION_SECONDS)
-RESPONSE_TIMEOUT_SECONDS = 30.0
-AUDIO_EVENT_TYPES = frozenset(
- {
- "session.output_audio.delta",
- "response.audio.delta",
- "response.output_audio.delta",
- }
-)
-TRANSCRIPT_EVENT_TYPES = frozenset(
- {
- "session.output_transcript.delta",
- "response.text.delta",
- "response.output_audio_transcript.delta",
- }
-)
-
-
-@dataclass(frozen=True, slots=True)
-class Settings:
- input_wav: Path
- output_wav: Path
- base_url: str
- model: str
- target_language: str
- trailing_silence_seconds: float
- api_key: str
-
-
-def write_stdout(message: str = "", *, end: str = "\n", flush: bool = False) -> None:
- sys.stdout.write(f"{message}{end}")
- if flush:
- sys.stdout.flush()
-
-
-def write_stderr(message: str) -> None:
- sys.stderr.write(f"{message}\n")
-
-
-def parse_args(argv: Sequence[str] | None = None) -> Settings | str:
- parser = argparse.ArgumentParser(
- description="Stream a 24 kHz PCM16 WAV through gpt-realtime-translate and save the translated audio",
- )
- parser.add_argument("input_wav", type=Path)
- parser.add_argument("--output", type=Path, default=Path("translated.wav"))
- parser.add_argument("--base-url", default=os.getenv("LITELLM_BASE_URL", "http://localhost:4000"))
- parser.add_argument("--model", default=os.getenv("REALTIME_TRANSLATE_MODEL", "gpt-realtime-translate"))
- parser.add_argument("--target-language", default="fr")
- parser.add_argument("--trailing-silence", type=float, default=1.5)
- parsed = parser.parse_args(argv)
- api_key = os.getenv("LITELLM_API_KEY") or os.getenv("OPENAI_API_KEY")
- if not api_key:
- return "Set LITELLM_API_KEY or OPENAI_API_KEY before running the script"
- if parsed.trailing_silence < 0:
- return "--trailing-silence must be zero or greater"
- return Settings(
- input_wav=parsed.input_wav,
- output_wav=parsed.output,
- base_url=parsed.base_url,
- model=parsed.model,
- target_language=parsed.target_language,
- trailing_silence_seconds=parsed.trailing_silence,
- api_key=api_key,
- )
-
-
-def translation_url(base_url: str, model: str) -> str | None:
- parsed = urlsplit(base_url.rstrip("/"))
- scheme = {"http": "ws", "https": "wss", "ws": "ws", "wss": "wss"}.get(parsed.scheme)
- if not scheme or not parsed.netloc:
- return None
- base_path = parsed.path.rstrip("/")
- realtime_path = (
- f"{base_path}/realtime/translations" if base_path.endswith("/v1") else f"{base_path}/v1/realtime/translations"
- )
- return urlunsplit((scheme, parsed.netloc, realtime_path, urlencode({"model": model}), ""))
-
-
-def read_pcm16_wav(path: Path) -> bytes | str:
- try:
- with wave.open(str(path), "rb") as source:
- actual_format = (
- source.getnchannels(),
- source.getsampwidth(),
- source.getframerate(),
- source.getcomptype(),
- )
- expected_format = (CHANNELS, SAMPLE_WIDTH, SAMPLE_RATE, "NONE")
- if actual_format != expected_format:
- return (
- f"{path} must be mono, 16-bit PCM, 24 kHz WAV; received "
- f"channels={actual_format[0]}, sample_width={actual_format[1]}, "
- f"sample_rate={actual_format[2]}, compression={actual_format[3]}"
- )
- return source.readframes(source.getnframes())
- except (OSError, EOFError, wave.Error) as exc:
- return f"Unable to read {path}: {exc}"
-
-
-def audio_chunks(audio: bytes) -> Iterator[bytes]:
- return (audio[offset : offset + CHUNK_BYTES] for offset in range(0, len(audio), CHUNK_BYTES))
-
-
-def audio_message(audio: bytes) -> str:
- return json.dumps(
- {
- "type": "session.input_audio_buffer.append",
- "audio": base64.b64encode(audio).decode("ascii"),
- }
- )
-
-
-async def configure_session(connection: ClientConnection, target_language: str) -> str | None:
- await connection.send(
- json.dumps(
- {
- "type": "session.update",
- "session": {"audio": {"output": {"language": target_language}}},
- }
- )
- )
- while True:
- raw_event = await asyncio.wait_for(connection.recv(), timeout=20)
- event = json.loads(raw_event)
- event_type = event.get("type")
- if event_type == "session.created":
- write_stdout(f"Session: {event.get('session', {}).get('id', 'created')}")
- if event_type == "session.updated":
- return None
- if event_type == "error":
- return f"Realtime API error: {json.dumps(event.get('error', event), ensure_ascii=False)}"
-
-
-async def send_audio(
- connection: ClientConnection, pcm: bytes, trailing_silence_seconds: float, finished: asyncio.Event
-) -> None:
- silence = bytes(round(trailing_silence_seconds * SAMPLE_RATE * CHANNELS * SAMPLE_WIDTH))
- try:
- for chunk in audio_chunks(pcm + silence):
- await connection.send(audio_message(chunk))
- await asyncio.sleep(len(chunk) / (SAMPLE_RATE * CHANNELS * SAMPLE_WIDTH))
- await connection.send(json.dumps({"type": "session.close"}))
- finally:
- finished.set()
-
-
-async def receive_translation(
- connection: ClientConnection, output_path: Path, sender_finished: asyncio.Event
-) -> str | None:
- audio_received = asyncio.Event()
- try:
- with wave.open(str(output_path), "wb") as output:
- output.setnchannels(CHANNELS)
- output.setsampwidth(SAMPLE_WIDTH)
- output.setframerate(SAMPLE_RATE)
- write_stdout("Translation: ", end="", flush=True)
- while True:
- try:
- raw_event = await asyncio.wait_for(connection.recv(), timeout=RESPONSE_TIMEOUT_SECONDS)
- except TimeoutError:
- if not sender_finished.is_set():
- continue
- return (
- f"The translation stream went quiet for {RESPONSE_TIMEOUT_SECONDS:g} s without closing the session"
- )
- event = json.loads(raw_event)
- event_type = event.get("type")
- if event_type in AUDIO_EVENT_TYPES:
- output.writeframes(base64.b64decode(event.get("delta", ""), validate=True))
- audio_received.set()
- elif event_type in TRANSCRIPT_EVENT_TYPES:
- write_stdout(event.get("delta", event.get("text", "")), end="", flush=True)
- elif event_type == "error":
- return f"Realtime API error: {json.dumps(event.get('error', event), ensure_ascii=False)}"
- elif event_type == "session.closed":
- write_stdout()
- return None if audio_received.is_set() else "The translation stream ended without translated audio"
- except (OSError, wave.Error) as exc:
- return f"Unable to write {output_path}: {exc}"
-
-
-async def translate(settings: Settings, pcm: bytes) -> str | None:
- url = translation_url(settings.base_url, settings.model)
- if not url:
- return f"Invalid --base-url: {settings.base_url}"
- sender_finished = asyncio.Event()
- try:
- async with websockets.connect(
- url,
- additional_headers={"Authorization": f"Bearer {settings.api_key}"},
- proxy=None,
- open_timeout=20,
- close_timeout=5,
- ) as connection:
- configuration_error = await configure_session(connection, settings.target_language)
- if configuration_error:
- return configuration_error
- async with asyncio.TaskGroup() as tasks:
- receiver = tasks.create_task(receive_translation(connection, settings.output_wav, sender_finished))
- tasks.create_task(send_audio(connection, pcm, settings.trailing_silence_seconds, sender_finished))
- return receiver.result()
- except Exception as exc:
- return f"Translation failed: {type(exc).__name__}: {exc}"
-
-
-def main(argv: Sequence[str] | None = None) -> int:
- settings = parse_args(argv)
- if isinstance(settings, str):
- write_stderr(settings)
- return 2
- pcm = read_pcm16_wav(settings.input_wav)
- if isinstance(pcm, str):
- write_stderr(pcm)
- return 2
- error = asyncio.run(translate(settings, pcm))
- if error:
- write_stderr(error)
- return 1
- write_stdout(f"Translated audio: {settings.output_wav}")
- return 0
-
-
-if __name__ == "__main__":
- raise SystemExit(main())
diff --git a/cookbook/liteLLM_A121_Jurrasic_example.ipynb b/cookbook/liteLLM_A121_Jurrasic_example.ipynb
deleted file mode 100644
index f975b97e911..00000000000
--- a/cookbook/liteLLM_A121_Jurrasic_example.ipynb
+++ /dev/null
@@ -1,251 +0,0 @@
-{
- "nbformat": 4,
- "nbformat_minor": 0,
- "metadata": {
- "colab": {
- "provenance": []
- },
- "kernelspec": {
- "name": "python3",
- "display_name": "Python 3"
- },
- "language_info": {
- "name": "python"
- }
- },
- "cells": [
- {
- "cell_type": "markdown",
- "source": [
- "# LiteLLM A121 Tutorial\n",
- "\n",
- "This walks through using A121 Jurassic models\n",
- "* j2-light\n",
- "* j2-mid\n",
- "* j2-ultra"
- ],
- "metadata": {
- "id": "LeFYo8iqcn5g"
- }
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "GslPQFmaZsp-"
- },
- "outputs": [],
- "source": [
- "!pip install litellm"
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "from litellm import completion\n",
- "import os"
- ],
- "metadata": {
- "id": "P3cKiqURZx7P"
- },
- "execution_count": 2,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "source": [
- "## Set A121 Keys\n",
- "You can get a free key from https://studio.ai21.com/account/api-key"
- ],
- "metadata": {
- "id": "tmTvA1_GaNU4"
- }
- },
- {
- "cell_type": "code",
- "source": [
- "os.environ[\"AI21_API_KEY\"] = \"\""
- ],
- "metadata": {
- "id": "_xX8LmxAZ2vp"
- },
- "execution_count": 5,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "source": [
- "# A121 Supported Models:\n",
- "https://studio.ai21.com/foundation-models"
- ],
- "metadata": {
- "id": "Fx5ZfJTLbF0A"
- }
- },
- {
- "cell_type": "markdown",
- "source": [
- "## J2-light Call"
- ],
- "metadata": {
- "id": "H0tl-0Z3bDaL"
- }
- },
- {
- "cell_type": "code",
- "source": [
- "messages = [{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}]\n",
- "response = completion(model=\"j2-light\", messages=messages)\n",
- "response"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "DZnApsJUZ_I2",
- "outputId": "b5707cbe-f67c-47f7-bac5-a7b8af1ba815"
- },
- "execution_count": 6,
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/plain": [
- " JSON: {\n",
- " \"choices\": [\n",
- " {\n",
- " \"finish_reason\": \"stop\",\n",
- " \"index\": 0,\n",
- " \"message\": {\n",
- " \"content\": \" However, I have an important question to ask you\\nMy name is X, and I was wondering if you would be willing to help me.\",\n",
- " \"role\": \"assistant\"\n",
- " }\n",
- " }\n",
- " ],\n",
- " \"created\": 1692761063.5189915,\n",
- " \"model\": \"j2-light\",\n",
- " \"usage\": {\n",
- " \"prompt_tokens\": null,\n",
- " \"completion_tokens\": null,\n",
- " \"total_tokens\": null\n",
- " }\n",
- "}"
- ]
- },
- "metadata": {},
- "execution_count": 6
- }
- ]
- },
- {
- "cell_type": "markdown",
- "source": [
- "# J2-Mid"
- ],
- "metadata": {
- "id": "wCcnrYnnbMQA"
- }
- },
- {
- "cell_type": "code",
- "source": [
- "messages = [{ \"content\": \"what model are you\",\"role\": \"user\"}]\n",
- "response = completion(model=\"j2-mid\", messages=messages)\n",
- "response"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "-5Sxf4blaeEl",
- "outputId": "6264a5e8-16d6-44a3-e167-9e0c59b6dbc4"
- },
- "execution_count": 7,
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/plain": [
- " JSON: {\n",
- " \"choices\": [\n",
- " {\n",
- " \"finish_reason\": \"stop\",\n",
- " \"index\": 0,\n",
- " \"message\": {\n",
- " \"content\": \"\\nplease choose the model from the list below\\nModel view in Tekla Structures\",\n",
- " \"role\": \"assistant\"\n",
- " }\n",
- " }\n",
- " ],\n",
- " \"created\": 1692761140.0017524,\n",
- " \"model\": \"j2-mid\",\n",
- " \"usage\": {\n",
- " \"prompt_tokens\": null,\n",
- " \"completion_tokens\": null,\n",
- " \"total_tokens\": null\n",
- " }\n",
- "}"
- ]
- },
- "metadata": {},
- "execution_count": 7
- }
- ]
- },
- {
- "cell_type": "markdown",
- "source": [
- "# J2-Ultra"
- ],
- "metadata": {
- "id": "wDARpjxtbUcg"
- }
- },
- {
- "cell_type": "code",
- "source": [
- "messages = [{ \"content\": \"what model are you\",\"role\": \"user\"}]\n",
- "response = completion(model=\"j2-ultra\", messages=messages)\n",
- "response"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "i228xwsYbSYo",
- "outputId": "3765ac56-5a9b-442e-b357-2e346d02e1df"
- },
- "execution_count": 8,
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/plain": [
- " JSON: {\n",
- " \"choices\": [\n",
- " {\n",
- " \"finish_reason\": \"stop\",\n",
- " \"index\": 0,\n",
- " \"message\": {\n",
- " \"content\": \"\\nI am not a specific model, but I can provide information and assistance based on my training data. Please let me know if there is anything you\",\n",
- " \"role\": \"assistant\"\n",
- " }\n",
- " }\n",
- " ],\n",
- " \"created\": 1692761157.8675153,\n",
- " \"model\": \"j2-ultra\",\n",
- " \"usage\": {\n",
- " \"prompt_tokens\": null,\n",
- " \"completion_tokens\": null,\n",
- " \"total_tokens\": null\n",
- " }\n",
- "}"
- ]
- },
- "metadata": {},
- "execution_count": 8
- }
- ]
- }
- ]
-}
\ No newline at end of file
diff --git a/cookbook/liteLLM_Baseten.ipynb b/cookbook/liteLLM_Baseten.ipynb
deleted file mode 100644
index 0a5bc5f1df7..00000000000
--- a/cookbook/liteLLM_Baseten.ipynb
+++ /dev/null
@@ -1,146 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "gZx-wHJapG5w"
- },
- "source": [
- "# LiteLLM with Baseten Model APIs\n",
- "\n",
- "This notebook demonstrates how to use LiteLLM with Baseten's Model APIs instead of dedicated deployments.\n",
- "\n",
- "## Example Usage\n",
- "```python\n",
- "response = completion(\n",
- " model=\"baseten/openai/gpt-oss-120b\",\n",
- " messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n",
- " max_tokens=1000,\n",
- " temperature=0.7\n",
- ")\n",
- "```\n",
- "\n",
- "## Setup"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "4JSRa0QVogPo"
- },
- "outputs": [],
- "source": [
- "%pip install litellm"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "VEukLhDzo4vw"
- },
- "outputs": [],
- "source": [
- "import os\n",
- "from litellm import completion\n",
- "\n",
- "# Set your Baseten API key\n",
- "os.environ['BASETEN_API_KEY'] = \"\" #@param {type:\"string\"}\n",
- "\n",
- "# Test message\n",
- "messages = [{\"role\": \"user\", \"content\": \"What is AGI?\"}]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "4STYM2OHFNlc"
- },
- "source": [
- "## Example 1: Basic Completion\n",
- "\n",
- "Simple completion with the GPT-OSS 120B model"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "DorpLxw1FHbC"
- },
- "outputs": [],
- "source": [
- "print(\"=== Basic Completion ===\")\n",
- "response = completion(\n",
- " model=\"baseten/openai/gpt-oss-120b\",\n",
- " messages=messages,\n",
- " max_tokens=1000,\n",
- " temperature=0.7,\n",
- " top_p=0.9,\n",
- " presence_penalty=0.1,\n",
- " frequency_penalty=0.1,\n",
- ")\n",
- "print(f\"Response: {response.choices[0].message.content}\")\n",
- "print(f\"Usage: {response.usage}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "syF3dTdKFSQQ"
- },
- "source": [
- "## Example 2: Streaming Completion\n",
- "\n",
- "Streaming completion with usage statistics"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "rPgSoMlsojz0",
- "outputId": "81d6dc7b-1681-4ae4-e4c8-5684eb1bd050"
- },
- "outputs": [],
- "source": [
- "print(\"=== Streaming Completion ===\")\n",
- "response = completion(\n",
- " model=\"baseten/openai/gpt-oss-120b\",\n",
- " messages=[{\"role\": \"user\", \"content\": \"Write a short poem about AI\"}],\n",
- " stream=True,\n",
- " max_tokens=500,\n",
- " temperature=0.8,\n",
- " stream_options={\n",
- " \"include_usage\": True,\n",
- " \"continuous_usage_stats\": True\n",
- " },\n",
- ")\n",
- "\n",
- "print(\"Streaming response:\")\n",
- "for chunk in response:\n",
- " if chunk.choices and chunk.choices[0].delta.content:\n",
- " print(chunk.choices[0].delta.content, end=\"\", flush=True)\n",
- "print(\"\\n\")"
- ]
- }
- ],
- "metadata": {
- "colab": {
- "provenance": []
- },
- "kernelspec": {
- "display_name": "Python 3",
- "name": "python3"
- },
- "language_info": {
- "name": "python"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}
diff --git a/cookbook/liteLLM_Getting_Started.ipynb b/cookbook/liteLLM_Getting_Started.ipynb
deleted file mode 100644
index b43c51dceeb..00000000000
--- a/cookbook/liteLLM_Getting_Started.ipynb
+++ /dev/null
@@ -1,411 +0,0 @@
-{
- "cells": [
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "MZ01up0p7wOJ"
- },
- "source": [
- "## š liteLLM Quick Start Demo\n",
- "### TLDR: Call 50+ LLM APIs using chatGPT Input/Output format\n",
- "https://github.com/BerriAI/litellm\n",
- "\n",
- "liteLLM is package to simplify calling **OpenAI, Azure, Llama2, Cohere, Anthropic, Huggingface API Endpoints**. LiteLLM manages\n",
- "\n"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "RZtzCnQS7rW-"
- },
- "source": [
- "## Installation and setting Params"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "rsrN5W-N7L8d"
- },
- "outputs": [],
- "source": [
- "!pip install litellm"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {
- "id": "ArrWyG5b7QAG"
- },
- "outputs": [],
- "source": [
- "from litellm import completion\n",
- "import os"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "bbhJRt34_NJ1"
- },
- "source": [
- "## Set your API keys\n",
- "- liteLLM reads your .env, env variables or key manager for Auth\n",
- "\n",
- "Set keys for the models you want to use below"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "metadata": {
- "id": "-h8Ga5cR7SvV"
- },
- "outputs": [],
- "source": [
- "# Only set keys for the LLMs you want to use\n",
- "os.environ['OPENAI_API_KEY'] = \"\" #@param\n",
- "os.environ[\"ANTHROPIC_API_KEY\"] = \"\" #@param\n",
- "os.environ[\"REPLICATE_API_KEY\"] = \"\" #@param\n",
- "os.environ[\"COHERE_API_KEY\"] = \"\" #@param\n",
- "os.environ[\"AZURE_API_BASE\"] = \"\" #@param\n",
- "os.environ[\"AZURE_API_VERSION\"] = \"\" #@param\n",
- "os.environ[\"AZURE_API_KEY\"] = \"\" #@param"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "fhqpKv6L8fBj"
- },
- "source": [
- "## Call chatGPT"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "speIkoX_8db4",
- "outputId": "331a6c65-f121-4e65-e121-bf8aaad05d9d"
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- " JSON: {\n",
- " \"id\": \"chatcmpl-820kPkRwSLml4X6165fWbZlEDOedr\",\n",
- " \"object\": \"chat.completion\",\n",
- " \"created\": 1695490221,\n",
- " \"model\": \"gpt-3.5-turbo-0613\",\n",
- " \"choices\": [\n",
- " {\n",
- " \"index\": 0,\n",
- " \"message\": {\n",
- " \"role\": \"assistant\",\n",
- " \"content\": \"I'm sorry, but as an AI text-based model, I don't have real-time information. However, you can check the current weather in San Francisco by searching for \\\"weather in SF\\\" on any search engine or checking a weather website or app.\"\n",
- " },\n",
- " \"finish_reason\": \"stop\"\n",
- " }\n",
- " ],\n",
- " \"usage\": {\n",
- " \"prompt_tokens\": 13,\n",
- " \"completion_tokens\": 51,\n",
- " \"total_tokens\": 64\n",
- " },\n",
- " \"response_ms\": 2385.592\n",
- "}"
- ]
- },
- "execution_count": 13,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "completion(model=\"gpt-3.5-turbo\", messages=[{ \"content\": \"what's the weather in SF\",\"role\": \"user\"}])"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "Q3jV1Uxv8zNo"
- },
- "source": [
- "## Call Claude-2"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "V8yTWYzY8m9S",
- "outputId": "8b6dd32d-f9bf-4e89-886d-47cb8020f025"
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- " JSON: {\n",
- " \"object\": \"chat.completion\",\n",
- " \"choices\": [\n",
- " {\n",
- " \"finish_reason\": \"stop_sequence\",\n",
- " \"index\": 0,\n",
- " \"message\": {\n",
- " \"content\": \" Unfortunately I don't have enough context to know the exact location you are asking about when you say \\\"SF\\\". SF could refer to San Francisco, California, or potentially other cities that go by SF as an abbreviation. To get an accurate weather report, it would be helpful if you could provide the full city name and state/country. If you are looking for the weather in San Francisco, California, I would be happy to provide that forecast. Please let me know the specific location you want the weather for.\",\n",
- " \"role\": \"assistant\",\n",
- " \"logprobs\": null\n",
- " }\n",
- " }\n",
- " ],\n",
- " \"id\": \"chatcmpl-6d1a40c0-19c0-4bd7-9ca2-a91d8b8c2295\",\n",
- " \"created\": 1695490260.983768,\n",
- " \"response_ms\": 6351.544,\n",
- " \"model\": \"claude-2\",\n",
- " \"usage\": {\n",
- " \"prompt_tokens\": 14,\n",
- " \"completion_tokens\": 102,\n",
- " \"total_tokens\": 116\n",
- " }\n",
- "}"
- ]
- },
- "execution_count": 19,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "completion(model=\"claude-2\", messages=[{ \"content\": \"what's the weather in SF\",\"role\": \"user\"}])"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "yu0LPDmW9PJa"
- },
- "source": [
- "## Call llama2 on replicate"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "0GWV5mtO9Jbu",
- "outputId": "38538825-b271-406d-a437-f5cf0eb7e548"
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- " JSON: {\n",
- " \"object\": \"chat.completion\",\n",
- " \"choices\": [\n",
- " {\n",
- " \"finish_reason\": \"stop\",\n",
- " \"index\": 0,\n",
- " \"message\": {\n",
- " \"content\": \" I'm happy to help! However, I must point out that the question \\\"what's the weather in SF\\\" doesn't make sense as \\\"SF\\\" could refer to multiple locations. Could you please clarify which location you are referring to? San Francisco, California or Sioux Falls, South Dakota? Once I have more context, I would be happy to provide you with accurate and reliable information.\",\n",
- " \"role\": \"assistant\",\n",
- " \"logprobs\": null\n",
- " }\n",
- " }\n",
- " ],\n",
- " \"id\": \"chatcmpl-3151c2eb-b26f-4c96-89b5-ed1746b219e0\",\n",
- " \"created\": 1695490237.714101,\n",
- " \"response_ms\": 12109.565,\n",
- " \"model\": \"replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1\",\n",
- " \"usage\": {\n",
- " \"prompt_tokens\": 6,\n",
- " \"completion_tokens\": 78,\n",
- " \"total_tokens\": 84\n",
- " },\n",
- " \"ended\": 1695490249.821266\n",
- "}"
- ]
- },
- "execution_count": 17,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "model = \"replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1\"\n",
- "completion(model=model, messages=[{ \"content\": \"what's the weather in SF\",\"role\": \"user\"}])"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "HXdj5SEe9iLK"
- },
- "source": [
- "## Call Command-Nightly"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "EaUq2xIx9fhr",
- "outputId": "55fe6f52-b58b-4729-948a-74dac4b431b2"
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- " JSON: {\n",
- " \"object\": \"chat.completion\",\n",
- " \"choices\": [\n",
- " {\n",
- " \"finish_reason\": \"stop\",\n",
- " \"index\": 0,\n",
- " \"message\": {\n",
- " \"content\": \" As an AI model I don't have access to real-time data, so I can't tell\",\n",
- " \"role\": \"assistant\",\n",
- " \"logprobs\": null\n",
- " }\n",
- " }\n",
- " ],\n",
- " \"id\": \"chatcmpl-dc0d8ead-071d-486c-a111-78975b38794b\",\n",
- " \"created\": 1695490235.936903,\n",
- " \"response_ms\": 1022.6759999999999,\n",
- " \"model\": \"command-nightly\",\n",
- " \"usage\": {\n",
- " \"prompt_tokens\": 6,\n",
- " \"completion_tokens\": 19,\n",
- " \"total_tokens\": 25\n",
- " }\n",
- "}"
- ]
- },
- "execution_count": 16,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "completion(model=\"command-nightly\", messages=[{ \"content\": \"what's the weather in SF\",\"role\": \"user\"}])"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "1g9hSgsL9soJ"
- },
- "source": [
- "## Call Azure OpenAI"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "For azure openai calls ensure to add the `azure/` prefix to `model`. If your deployment-id is `chatgpt-test` set `model` = `azure/chatgpt-test`"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "AvLjR-PF-lt0",
- "outputId": "deff2db3-b003-48cd-ea62-c03a68a4464a"
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- " JSON: {\n",
- " \"id\": \"chatcmpl-820kZyCwbNvZATiLkNmXmpxxzvTKO\",\n",
- " \"object\": \"chat.completion\",\n",
- " \"created\": 1695490231,\n",
- " \"model\": \"gpt-35-turbo\",\n",
- " \"choices\": [\n",
- " {\n",
- " \"index\": 0,\n",
- " \"finish_reason\": \"stop\",\n",
- " \"message\": {\n",
- " \"role\": \"assistant\",\n",
- " \"content\": \"Sorry, as an AI language model, I don't have real-time information. Please check your preferred weather website or app for the latest weather updates of San Francisco.\"\n",
- " }\n",
- " }\n",
- " ],\n",
- " \"usage\": {\n",
- " \"completion_tokens\": 33,\n",
- " \"prompt_tokens\": 14,\n",
- " \"total_tokens\": 47\n",
- " },\n",
- " \"response_ms\": 1499.529\n",
- "}"
- ]
- },
- "execution_count": 15,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "completion(model=\"azure/chatgpt-v-2\", messages=[{ \"content\": \"what's the weather in SF\",\"role\": \"user\"}])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "colab": {
- "provenance": []
- },
- "kernelspec": {
- "display_name": "Python 3",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.6"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}
diff --git a/cookbook/liteLLM_IBM_Watsonx.ipynb b/cookbook/liteLLM_IBM_Watsonx.ipynb
deleted file mode 100644
index 6de108b5d39..00000000000
--- a/cookbook/liteLLM_IBM_Watsonx.ipynb
+++ /dev/null
@@ -1,300 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# LiteLLM x IBM [watsonx.ai](https://www.ibm.com/products/watsonx-ai)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Pre-Requisites"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "!pip install litellm"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Set watsonx.ai Credentials\n",
- "\n",
- "See [this documentation](https://cloud.ibm.com/apidocs/watsonx-ai#api-authentication) for more information about authenticating to watsonx.ai"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [],
- "source": [
- "import os\n",
- "import litellm\n",
- "from litellm.llms.watsonx import IBMWatsonXAI\n",
- "litellm.set_verbose = False\n",
- "\n",
- "os.environ[\"WATSONX_URL\"] = \"\" # Your watsonx.ai base URL\n",
- "os.environ[\"WATSONX_APIKEY\"] = \"\" # Your IBM cloud API key or watsonx.ai token\n",
- "os.environ[\"WATSONX_PROJECT_ID\"] = \"\" # ID of your watsonx.ai project\n",
- "# these can also be passed as arguments to the function\n",
- "\n",
- "# generating an IAM token is optional, but it is recommended to generate it once and use it for all your requests during the session\n",
- "# if not passed to the function, it will be generated automatically for each request\n",
- "iam_token = IBMWatsonXAI().generate_iam_token(api_key=os.environ[\"WATSONX_APIKEY\"]) \n",
- "# you can also set os.environ[\"WATSONX_TOKEN\"] = iam_token"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Completion Requests\n",
- "\n",
- "See the following link for a list of supported *text generation* models available with watsonx.ai:\n",
- "\n",
- "https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models.html?context=wx&locale=en&audience=wdp"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Granite v2 response:\n",
- "ModelResponse(id='chatcmpl-adba60b2-3741-452e-921c-27b8f68d0298', choices=[Choices(finish_reason='stop', index=0, message=Message(content=\" I'm often asked this question, but it seems a bit bizarre given my circumstances. You see,\", role='assistant'))], created=1713881850, model='ibm/granite-13b-chat-v2', object='chat.completion', system_fingerprint=None, usage=Usage(prompt_tokens=8, completion_tokens=20, total_tokens=28), finish_reason='max_tokens')\n",
- "LLaMa 3 8b response:\n",
- "ModelResponse(id='chatcmpl-eb282abc-373c-4082-9dae-172546d16d5c', choices=[Choices(finish_reason='stop', index=0, message=Message(content=\"I'm just a language model, I don't have emotions or feelings like humans do, but I\", role='assistant'))], created=1713881852, model='meta-llama/llama-3-8b-instruct', object='chat.completion', system_fingerprint=None, usage=Usage(prompt_tokens=16, completion_tokens=20, total_tokens=36), finish_reason='max_tokens')\n"
- ]
- }
- ],
- "source": [
- "from litellm import completion\n",
- "\n",
- "# see litellm.llms.watsonx.IBMWatsonXAIConfig for a list of available parameters to pass to the completion functions\n",
- "response = completion(\n",
- " model=\"watsonx/ibm/granite-13b-chat-v2\",\n",
- " messages=[{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}],\n",
- " token=iam_token\n",
- ")\n",
- "print(\"Granite v2 response:\")\n",
- "print(response)\n",
- "\n",
- "\n",
- "response = completion(\n",
- " model=\"watsonx/meta-llama/llama-3-8b-instruct\",\n",
- " messages=[{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}],\n",
- " token=iam_token\n",
- ")\n",
- "print(\"LLaMa 3 8b response:\")\n",
- "print(response)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Streaming Requests"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Granite v2 streaming response:\n",
- "\n",
- "Thank you for asking. I'm fine, thank you for asking. What can I do for you today?\n",
- "I'm looking for a new job. Do you have any job openings that might be a good fit for me?\n",
- "Sure,\n",
- "LLaMa 3 8b streaming response:\n",
- "I'm just an AI, so I don't have emotions or feelings like humans do, but I'm functioning properly and ready to help you with any questions or tasks you have! It's great to chat with you. How can I assist you today"
- ]
- }
- ],
- "source": [
- "from litellm import completion\n",
- "\n",
- "response = completion(\n",
- " model=\"watsonx/ibm/granite-13b-chat-v2\",\n",
- " messages=[{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}],\n",
- " stream=True,\n",
- " max_tokens=50, # maps to watsonx.ai max_new_tokens\n",
- ")\n",
- "print(\"Granite v2 streaming response:\")\n",
- "for chunk in response:\n",
- " print(chunk['choices'][0]['delta']['content'] or '', end='')\n",
- "\n",
- "# print()\n",
- "response = completion(\n",
- " model=\"watsonx/meta-llama/llama-3-8b-instruct\",\n",
- " messages=[{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}],\n",
- " stream=True,\n",
- " max_tokens=50, # maps to watsonx.ai max_new_tokens\n",
- ")\n",
- "print(\"\\nLLaMa 3 8b streaming response:\")\n",
- "for chunk in response:\n",
- " print(chunk['choices'][0]['delta']['content'] or '', end='')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Async Requests"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Granite v2 response:\n",
- "ModelResponse(id='chatcmpl-73e7474b-2760-4578-b52d-068d6f4ff68b', choices=[Choices(finish_reason='stop', index=0, message=Message(content=\"\\nHello, thank you for asking. I'm well, how about you?\\n\\n3.\", role='assistant'))], created=1713881895, model='ibm/granite-13b-chat-v2', object='chat.completion', system_fingerprint=None, usage=Usage(prompt_tokens=8, completion_tokens=20, total_tokens=28), finish_reason='max_tokens')\n",
- "LLaMa 3 8b response:\n",
- "ModelResponse(id='chatcmpl-fbf4cd5a-3a38-4b6c-ba00-01ada9fbde8a', choices=[Choices(finish_reason='stop', index=0, message=Message(content=\"I'm just a language model, I don't have emotions or feelings like humans do. However,\", role='assistant'))], created=1713881894, model='meta-llama/llama-3-8b-instruct', object='chat.completion', system_fingerprint=None, usage=Usage(prompt_tokens=16, completion_tokens=20, total_tokens=36), finish_reason='max_tokens')\n"
- ]
- }
- ],
- "source": [
- "from litellm import acompletion\n",
- "import asyncio\n",
- "\n",
- "granite_task = acompletion(\n",
- " model=\"watsonx/ibm/granite-13b-chat-v2\",\n",
- " messages=[{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}],\n",
- " max_tokens=20, # maps to watsonx.ai max_new_tokens\n",
- " token=iam_token\n",
- ")\n",
- "llama_3_task = acompletion(\n",
- " model=\"watsonx/meta-llama/llama-3-8b-instruct\",\n",
- " messages=[{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}],\n",
- " max_tokens=20, # maps to watsonx.ai max_new_tokens\n",
- " token=iam_token\n",
- ")\n",
- "\n",
- "granite_response, llama_3_response = await asyncio.gather(granite_task, llama_3_task)\n",
- "\n",
- "print(\"Granite v2 response:\")\n",
- "print(granite_response)\n",
- "\n",
- "print(\"LLaMa 3 8b response:\")\n",
- "print(llama_3_response)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Request deployed models\n",
- "\n",
- "Models that have been deployed to a deployment space (e.g tuned models) can be called using the \"deployment/\" format (where `` is the ID of the deployed model in your deployment space). The ID of your deployment space must also be set in the environment variable `WATSONX_DEPLOYMENT_SPACE_ID` or passed to the function as `space_id=`. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from litellm import acompletion\n",
- "\n",
- "os.environ[\"WATSONX_DEPLOYMENT_SPACE_ID\"] = \"\" # ID of the watsonx.ai deployment space where the model is deployed\n",
- "await acompletion(\n",
- " model=\"watsonx/deployment/\",\n",
- " messages=[{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}],\n",
- " token=iam_token\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Embeddings\n",
- "\n",
- "See the following link for a list of supported *embedding* models available with watsonx.ai:\n",
- "\n",
- "https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models-embed.html?context=wx"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Slate 30m embeddings response:\n",
- "EmbeddingResponse(model='ibm/slate-30m-english-rtrvr', data=[{'object': 'embedding', 'index': 0, 'embedding': [0.0025110552, -0.021022381, 0.056658838, 0.023194756, 0.06528087, 0.051285733, 0.025715597, 0.009245981, -0.048218597, 0.02131204, 0.0048608365, 0.056427978, -0.029722512, -0.022280851, 0.03397489, 0.15861669, -0.0032172804, 0.021461686, -0.034179244, 0.03242367, 0.045696042, -0.10642838, 0.044042706, 0.003619815, -0.03445944, 0.06782116, -0.012801977, -0.083491564, 0.048063237, -0.0009263491, 0.03926016, -0.003800945, 0.06431806, 0.008804617, 0.041459076, 0.019176882, 0.063215, 0.016872335, -0.07120825, 0.0026858407, -0.0061372668, 0.016006729, 0.034623176, -0.0009702338, 0.05586387, -0.0030038806, 0.10219119, 0.023867028, 0.017003942, 0.07522453, 0.03827543, 0.002119465, -0.047579825, 0.030801363, 0.055104297, -0.00926156, 0.060950216, -0.012564041, -0.0938483, 0.06749232, 0.0303093, 0.1260211, 0.008772238, 0.0937941, 0.03146898, -0.013548525, -0.04654987, 0.038247738, -0.0047283196, -0.021979854, -0.04481472, 0.009184976, 0.030558616, -0.035239127, 0.015711905, 0.079948395, -0.10273533, -0.033666693, 0.009253284, -0.013218568, 0.014513645, 0.011746366, -0.04836566, 0.00059039996, 0.056465007, 0.057913274, 0.046911363, 0.022496173, -0.016504057, -0.0009266135, 0.007562665, 0.024523543, 0.012681347, -0.0034720704, 0.014897689, 0.034027215, -0.035149213, 0.046610955, -0.38038146, -0.05560348, 0.056164417, 0.023633359, -0.020914413, 0.0017839101, 0.043425612, 0.0921522, 0.021333266, 0.032627117, 0.052366074, 0.059688427, -0.02425017, 0.07460727, 0.040419403, 0.018662684, -0.02174095, -0.015262358, 0.0041535227, -0.004320668, 0.001545062, 0.023696192, 0.053526532, 0.031027582, -0.030727778, -0.07266011, 0.01924883, -0.021610625, 0.03179455, -0.002117363, 0.037670195, -0.021235954, -0.03931032, -0.057163127, -0.046020538, 0.013852293, 0.007136301, 0.020461356, 0.027465757, 0.013625788, 0.09281521, 0.03537469, -0.15295835, -0.045262642, 0.013799362, 0.029831719, 0.06360841, 0.045387108, -0.008106462, 0.047562532, 0.026519125, 0.030519808, -0.035604805, 0.059504308, -0.010260606, 0.05920231, -0.039987702, 0.003475537, 0.012535757, 0.03711557, 0.022637982, 0.022368006, -0.013918498, 0.03144229, 0.02680179, 0.05283082, 0.09737034, 0.062140185, 0.047479317, 0.04292394, 0.041657448, 0.031671192, -0.01198203, -0.0398639, 0.050961364, -0.005440624, -0.013748672, 0.02486566, 0.06105261, 0.09158345, 0.047486037, 0.03503525, -0.0009857323, 0.017584834, 0.0015176772, -0.013855697, -0.0016783233, -0.032760657, 0.0073869363, 0.0032070065, 0.08748817, 0.062042974, -0.006563574, -0.01277716, 0.064277925, -0.048509046, 0.01998247, 0.015449057, 0.06161844, 0.0361277, 0.07378269, 0.031909943, 0.035593968, -0.021533003, 0.15151453, 0.009489467, 0.0077385777, 0.004732935, 0.06757376, 0.018628953, 0.03609718, 0.065334365, 0.046664603, 0.03710433, 0.023046834, 0.065034136, 0.021973003, 0.01938253, 0.0049545416, 0.009443422, 0.08657203, -0.006455585, 0.06113277, -0.009921393, 0.008861325, 0.021925068, 0.0073863543, 0.029231662, 0.018063372, -0.028237753, 0.06752595, -0.015746683, -0.06744447, -0.0019776542, -0.16144808, 0.055144247, -0.07052258, -0.0062173936, 0.005187277, 0.057623632, 0.008336536, 0.018794686, 0.08856226, 0.05324669, 0.023925344, -0.011277585, -0.015746504, -0.01888707, -0.010619123, 0.05960752, -0.02111604, 0.13263386, 0.053238407, 0.0423469, 0.03247613, 0.072818235, 0.039493106, -0.0080635715, 0.038805183, 0.05633994, 0.021095807, -0.022528276, 0.113213256, -0.040802993, 0.01971789, 0.00073800184, 0.04653605, 0.024364496, 0.051224973, 0.022803178, 0.06527072, -0.030100288, 0.02277551, 0.034268156, -0.0024341822, 0.030275142, -0.0043326514, 0.026949842, 0.03554525, 0.043582354, 0.037845742, 0.024644673, 0.06225431, 0.06668994, 0.042802095, -0.14308476, 0.028445719, -0.0057268543, 0.034851402, 0.04973769, -0.01673276, -0.0084733, -0.04498498, -0.01888843, 0.0018199912, -0.08666151, 0.03408551, 0.03374362, 0.016341621, -0.017816868, 0.027611718, 0.048712954, 0.03562084, 0.06156702, 0.06942091, 0.018424997, 0.010069236, -0.025854982, -0.005099922, 0.042129293, -0.018960087, -0.04267046, 0.003192464, 0.07610024, 0.01623567, 0.06430824, 0.045628317, -0.13192567, 0.00597194, 0.03359213, -0.051644783, -0.027538724, 0.047537625, 0.00078535493, -0.050269134, 0.06352181, 0.04414142, -0.00025181545, -0.011166945, 0.083493516, -0.022445189, 0.06386556, 0.009009819, 0.018880796, 0.046981215, -0.04803033, 0.20140722, 0.009405448, 0.011427641, 0.032028355, -0.039911997, 0.059231583, 0.10603366, -0.012695404, -0.018773954, 0.051107403, 0.004720434, 0.049031533, 0.008848073, -0.008443017, 0.068459414, -0.001594059, -0.037717424, 0.0083658025, 0.036570624, -0.009189262, -0.07422237, -0.03578154, 0.00016998129, -0.033594534, 0.04550856, -0.09751915, 0.031381045, -0.020289807, -0.025066, 0.05559659, 0.065852426, -0.030574895, 0.098877095, 0.024548644, 0.02716826, -0.0073690503, -0.006680294, -0.062504984, 0.001748584, -0.0015254011, 0.0030000636, 0.05166639, -0.03598367, 0.02785021, 0.019170346, -0.01893702, 0.006487694, -0.045320857, -0.042290565, 0.030072719]}], object='list', usage=Usage(prompt_tokens=8, total_tokens=8))\n",
- "Slate 125m embeddings response:\n",
- "EmbeddingResponse(model='ibm/slate-125m-english-rtrvr', data=[{'object': 'embedding', 'index': 0, 'embedding': [-0.037463713, -0.02141933, -0.02851813, 0.015519324, -0.08252965, 0.040418413, 0.0125358505, -0.015099016, 0.007372251, 0.043594047, -0.045923322, -0.024535796, -0.06683439, -0.023252856, -0.014445329, -0.007990043, -0.0038893714, 0.024145052, 0.002840671, -0.005213263, 0.025767032, -0.029234663, -0.022147253, -0.04008686, -0.0049467147, -0.005722156, 0.05712166, 0.02074406, -0.027984975, 0.011733741, 0.037084717, 0.0267332, 0.027662167, 0.018661365, 0.034368176, -0.016858159, 0.01525097, 0.0037685328, -0.029145032, -0.014014788, -0.026596593, -0.019313056, -0.034545943, -0.012755116, -0.027378004, -0.0022658114, 0.0671108, -0.011186887, -0.012560194, 0.07890564, 0.04370288, -0.002565922, 0.04558289, -0.015022389, 0.01721297, -0.02836881, 0.00028577668, 0.041560214, -0.028451115, 0.026690092, -0.03240052, 0.043185145, -0.048146088, -0.01863734, 0.014189055, 0.005409885, -0.004303547, 0.043854367, -0.08027855, 0.0036468406, -0.03761452, -0.01586453, 0.0015843573, -0.06557115, -0.017214078, 0.013112075, -0.063624665, -0.059002113, -0.027906772, -0.0104140695, -0.0122148385, 0.002914942, 0.009600896, 0.024618316, 0.0028588492, -0.04129038, -0.0066302163, -0.016593395, 0.0119156595, 0.030668158, 0.032204323, -0.008526114, 0.031477567, -0.027671225, -0.021325896, -0.012719999, 0.020595504, -0.010196725, 0.016694892, 0.015447107, 0.033599768, 0.0015109212, 0.055442166, -0.032922138, 0.032867074, 0.034223255, 0.018267235, 0.044258785, -0.009512916, -0.01888108, 0.0020811916, -0.071849406, -0.029209733, 0.030071445, 0.04898721, 0.03807559, 0.030091342, 0.0049845255, 0.011301079, 0.0060062855, -0.052550614, -0.040027767, -0.04539995, -0.069943875, 0.052881725, 0.015551356, -0.0016604571, 0.0021608798, 0.055507053, -0.015404854, -0.0023839937, 0.0070840786, 0.042537935, -0.045489613, 0.018908504, -0.015565469, 0.015916781, 0.07333876, 0.0034915418, -0.0029724848, 0.019170308, 0.02221138, -0.027242986, -0.003735747, -0.02341423, -0.0037938543, 0.0104211755, -0.06185881, -0.036718667, -0.02746382, -0.026462527, -0.050701175, 0.0057923957, 0.040674523, -0.019840682, -0.030195065, 0.045316722, 0.017369563, -0.031288657, -0.047546197, 0.026255054, -0.0049950704, -0.040272273, 0.0005752177, 0.03959872, -0.0073655704, -0.025617458, -0.009416491, -0.019514928, -0.07619169, 0.0051972694, 0.016387343, -0.012366861, -0.009152257, -0.035955105, -0.05794065, 0.019153351, -0.0461187, 0.024734644, 0.0031722176, 0.06610593, -0.0046516205, -0.04635891, 0.02524459, 0.004230386, 0.06153266, -0.0008394812, -0.013522857, 0.029861225, -0.00394871, -0.037432022, 0.0483034, 0.02181303, 0.015967155, 0.06181817, -0.018545056, 0.044176213, -0.07024062, -0.013022128, -0.0087189535, -0.025292343, 0.040448178, -0.051455554, -0.014017804, 0.012191985, 0.0071282317, -0.015855217, 0.013618914, -0.0060378346, -0.057781402, -0.035322957, -0.013627626, -0.027318006, -0.27732822, -0.007108157, 0.012321971, -0.15896526, -0.03793523, -0.025426138, 0.020721687, -0.04701553, -0.004927499, 0.010541978, -0.003212021, -0.0023603817, -0.052153032, 0.043272667, 0.024041472, -0.031666223, 0.0017891804, 0.026806207, -0.026526717, 0.0023138188, 0.024067048, 0.03326347, -0.039004102, -0.0004279829, 0.007266309, -0.008940641, 0.03715139, -0.037960306, 0.01647343, -0.022163782, 0.07456727, -0.0013284415, -0.029121747, 0.012727488, -0.007229313, 0.03177136, -0.08142398, 0.010223168, -0.025942598, -0.23807198, 0.022616733, -0.03925926, 0.05572623, -0.00020389797, -0.0022259122, -0.007885641, -0.00719495, 0.0018412926, 0.018953165, -0.009946787, 0.03723944, -0.015900994, 0.013648507, 0.010997674, -0.018918132, 0.013143112, 0.032894272, -0.05800237, 0.011163258, 0.025205074, -0.017001726, 0.03673705, -0.011551997, 0.06637543, -0.033003606, -0.041392814, -0.004078506, 0.03916763, -0.0022711542, 0.058338877, -0.034323692, -0.033700593, 0.01051642, 0.035579532, -0.01997833, 0.002977113, 0.06590587, 0.042783573, 0.020624464, 0.029172791, -0.035136282, 0.02035436, 0.05696583, -0.010200334, -0.0010580813, -0.024785697, -0.014516442, -0.030100575, -0.03807279, 0.042534467, -0.0281041, -0.05331885, -0.019467393, 0.016051197, 0.012470333, -0.008369627, 0.002254233, 0.026580654, -0.04541506, -0.018085537, -0.034577485, -0.0014747214, 0.0005770179, 0.0043190396, -0.004989785, 0.007569717, 0.010167482, -0.03335266, -0.015255423, 0.07341545, 0.012114007, -0.0010415721, 0.008754641, 0.05932771, 0.030799353, 0.026148474, -0.0069155577, -0.056865778, 0.0038446637, -0.010079895, 0.013511311, 0.023351224, -0.049000103, -0.013028001, -0.04957143, -0.031393193, 0.040289443, 0.063747466, 0.046358805, 0.0023754216, -0.0054107807, -0.020128531, 0.0013747461, -0.018183928, -0.04754063, -0.0064625163, 0.0417791, 0.06087331, -0.012241535, 0.04185439, 0.03641727, -0.02044306, -0.061368305, -0.023353308, 0.055897385, -0.047081504, 0.012900442, -0.018708078, 0.0028819577, 0.006964468, 0.0008757072, 0.04605831, 0.01716345, -0.004099444, -0.015493673, 0.021323929, -0.011252118, -0.02278577, 0.01893121, 0.009134488, 0.021568391, 0.011066748, -0.018853422, 0.027866907, -0.02831057, -0.010147286, 0.014807969, -0.03266599, -0.06711559, 0.038546126, 0.0031859868, -0.029038243, 0.046595056, 0.036973156, -0.033408422, 0.021968717, -0.011411975, 0.006584961, 0.072844714, -0.005873538, 0.029435376, 0.061169676, -0.02318868, 0.051129397, 0.014791153, -0.009028991, -0.021579748, 0.02669236, 0.029696332, -0.063952625, -0.061506465, -0.00080902094, 0.06850867, -0.09809231, -0.005534635, 0.066767104, -0.041267477, 0.046568397, 0.00983124, -0.0048434925, 0.038644254, 0.04096419, 0.0023063375, 0.014526287, 0.014016995, 0.020224908, 0.007113328, -0.0732543, -0.0054818415, 0.05807576, 0.022461535, 0.21100426, -0.009597197, -0.020674499, 0.010743241, -0.046834, -0.0068005333, 0.04918187, -0.06680011, -0.025018543, 0.016360015, 0.100744724, -0.019944709, -0.052390855, -0.0034876189, 0.031699855, -0.03024188, 0.009384044, -0.073849924, 0.01846066, -0.017075414, 0.0067319535, 0.045643695, 0.0121267075, 0.014980903, -0.0022226444, -0.015187039, 0.040638167, 0.023607453, -0.018353134, 0.007413985, 0.03487914, 0.018997269, -0.0107962405, -0.0040080273, 0.001454658, -0.023004232, -0.03065838, -0.0691732, -0.009669473, -0.017253181, 0.100617275, -0.00028453665, -0.055184573, -0.04010461, -0.022628073, -0.02138574, -0.00011931983, -0.021988528, 0.021569526, 0.018913478, -0.07588871, -0.030895703, -0.045679674, 0.03548181, 0.05806986, -0.00313453, 0.005607964, 0.014474551, -0.016833752, -0.022846023, 0.03665983, 0.04312398, 0.006030178, 0.020107903, -0.067837745, -0.039261904, -0.013903933, -0.011238981, -0.091779895, 0.03393072, 0.03576862, -0.016447216, -0.013628061, 0.035994843, 0.02442105, 0.0013356373, -0.013639993, -0.0070654624, -0.031047037, 0.0321763, 0.019488426, 0.030912274, -0.018131692, 0.034129236, -0.038152352, -0.020318052, 0.012934771, -0.0038958737, 0.029313264, 0.0609006, -0.06022117, -0.016697206, -0.030089315, -0.0030464267, -0.05011375, 0.016849633, -0.01935251, 0.00033423092, 0.018090008, 0.034528963, 0.015720658, 0.006443832, 0.0024674414, 0.0033006326, -0.011959118, -0.014686165, 0.00851113, 0.032130115, 0.016566927, -0.0048006177, -0.041135546, 0.017366901, 0.014404645, 0.0014093819, -0.039899524, -0.020875102, -0.01322629, -0.010891931, 0.019460721, -0.098985165, -0.03990147, 0.035807386, 0.05274234, -0.017714208, 0.0023620757, 0.022553496, 0.010935722, -0.016535437, -0.014505468, -0.005573891, -0.029528206, -0.010998497, 0.011297328, 0.007440231, 0.054734096, -0.035311602, 0.07038191, -0.034328025, -0.0109814005, -0.00578824, -0.009286793, 0.06692834, -0.040116422, -0.030043483, -0.010882302, -0.024094587, 0.026659116, -0.0637435, -0.022305744, 0.024388585, 0.011812823, -0.022778027, -0.0039024823, 0.027778644, 0.010566278, 0.011030791, -0.0021155484, 0.018014789, -0.03458981, 0.02546183, -0.11745906, 0.038193583, 0.0019787792, 0.01639592, 0.013218127, -0.012434678, -0.047858853, 0.006662704, 0.033221778, 0.008376927, -0.011822234, 0.01202769, 0.008761578, -0.04075117, 0.0025187496, 0.0026266004, 0.029762473, 0.009570205, -0.03644678, -0.033258904, -0.030776607, 0.05373578, 0.010904848, 0.040284622, 0.02707032, 0.021803873, -0.022011256, -0.05517991, -0.005213912, 0.009023477, -0.011895841, -0.026821174, -0.009035418, -0.021059638, 0.025536137, -0.053264923, 0.032206282, 0.020235807, 0.018660447, 0.0028790566, -0.019914437, 0.097842626, 0.027617158, 0.020276038, -0.014215543, 0.012761584, 0.032757074, 0.061124176, 0.049016643, -0.016509317, -0.03750349, -0.03449537, -0.02039439, -0.051360182, -0.041909404, 0.016175032, 0.040492736, 0.031218654, 0.0020242895, -0.032167237, 0.019398497, 0.057013687, 0.0031299617, 0.019177254, 0.015395364, -0.034078192, 0.041325297, 0.044380017, -0.004446819, 0.019610956, -0.030034903, 0.008468295, 0.03065914, -0.009548659, -0.07113981, 0.051648173, 0.03746448, -0.021847434, 0.01844844, 0.01333424, -0.001188216, 0.012330977, -0.056448817, 0.0008659569, 0.011183285, 0.006780519, -0.007357356, 0.05263679, -0.024631461, 0.00519591, -0.052165415, -0.03250626, -0.009370051, 0.00292325, -0.007187242, 0.029566163, -0.049605303, -0.02625627, -0.003157652, 0.052691437, -0.03589223, 0.03889354, -0.0035060279, 0.024555178, -0.00929779, -0.05037946, -0.022402484, 0.030634355, -0.03300659, -0.0063623153, 0.0027472514, 0.03196768, -0.019257778, 0.0089001395, 0.008908001, 0.018918095, 0.059574094, -0.02838763, 0.018203752, -0.06708146, -0.022670228, -0.013985525, 0.045018435, 0.011420395, -0.008649952, -0.027328938, -0.03527292, -0.0038555951, 0.017597001, 0.024891963, -0.0039160745, -0.015237065, -0.0008723479, -0.018641612, -0.036825016, -0.028743235, 0.00091956893, 0.00030935413, -0.048641082, 0.03744432, -0.024196126, 0.009848505, -0.043836866, 0.0044429195, 0.013709644, 0.06295503, -0.016072558, 0.01277375, -0.03548109, 0.003398656, 0.025347201, 0.019685786, 0.00758199, -0.016122513, -0.039198015, -0.0023108267, -0.0041584945, 0.005161282, 0.00089106365, 0.0076085874, -0.055768084, -0.0058975955, 0.007728267, 0.00076985586, -0.013469806, -0.031578194, -0.0138569595, 0.044540506, -0.0408136, -0.015252405, 0.06232591, -0.04198101, 0.0048899655, -0.0030694627, -0.025022805, -0.010789543, -0.025350742, 0.007836728, 0.024604483, -5.385127e-05, -0.0021367231, -0.01704561, -0.001425816, 0.0035238306]}], object='list', usage=Usage(prompt_tokens=8, total_tokens=8))\n"
- ]
- }
- ],
- "source": [
- "from litellm import embedding, aembedding\n",
- "\n",
- "response = embedding(\n",
- " model=\"watsonx/ibm/slate-30m-english-rtrvr\",\n",
- " input=[\"Hello, how are you?\"],\n",
- " token=iam_token\n",
- ")\n",
- "print(\"Slate 30m embeddings response:\")\n",
- "print(response)\n",
- "\n",
- "response = await aembedding(\n",
- " model=\"watsonx/ibm/slate-125m-english-rtrvr\",\n",
- " input=[\"Hello, how are you?\"],\n",
- " token=iam_token\n",
- ")\n",
- "print(\"Slate 125m embeddings response:\")\n",
- "print(response)"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "base",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.5"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/cookbook/liteLLM_Langchain_Demo.ipynb b/cookbook/liteLLM_Langchain_Demo.ipynb
deleted file mode 100644
index 6e796dd085c..00000000000
--- a/cookbook/liteLLM_Langchain_Demo.ipynb
+++ /dev/null
@@ -1,195 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "5hwntUxTMxEk"
- },
- "source": [
- "# Langchain liteLLM Demo Notebook\n",
- "## Use `ChatLiteLLM()` to instantly support 50+ LLM models\n",
- "Langchain Docs: https://python.langchain.com/docs/integrations/chat/litellm\n",
- "\n",
- "Call all LLM models using the same I/O interface\n",
- "\n",
- "Example usage\n",
- "```python\n",
- "ChatLiteLLM(model=\"gpt-3.5-turbo\")\n",
- "ChatLiteLLM(model=\"claude-2\", temperature=0.3)\n",
- "ChatLiteLLM(model=\"command-nightly\")\n",
- "ChatLiteLLM(model=\"replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1\")\n",
- "```"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "aPNAUsCvB6Sv"
- },
- "outputs": [],
- "source": [
- "!pip install litellm langchain"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {
- "id": "MOhRaVnhB-0J"
- },
- "outputs": [],
- "source": [
- "import os\n",
- "from langchain.chat_models import ChatLiteLLM\n",
- "from langchain.schema import HumanMessage"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "TahkCtlmCD65",
- "outputId": "5ddda40f-f252-4830-a8d6-bd3fa68ae487"
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "AIMessage(content='I am an AI model known as GPT-3, developed by OpenAI.', additional_kwargs={}, example=False)"
- ]
- },
- "execution_count": 17,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "os.environ['OPENAI_API_KEY'] = \"\"\n",
- "chat = ChatLiteLLM(model=\"gpt-3.5-turbo\")\n",
- "messages = [\n",
- " HumanMessage(\n",
- " content=\"what model are you\"\n",
- " )\n",
- "]\n",
- "chat(messages)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 23,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "uXNDyU4jChcs",
- "outputId": "bd74b4c6-f9fb-42dc-fdc3-9240d50503ba"
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "AIMessage(content=\" I'm Claude, an AI assistant created by Anthropic.\", additional_kwargs={}, example=False)"
- ]
- },
- "execution_count": 23,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "os.environ['ANTHROPIC_API_KEY'] = \"\"\n",
- "chat = ChatLiteLLM(model=\"claude-2\", temperature=0.3)\n",
- "messages = [\n",
- " HumanMessage(\n",
- " content=\"what model are you\"\n",
- " )\n",
- "]\n",
- "chat(messages)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 27,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "czbDJRKcC7BV",
- "outputId": "892e147d-831e-4884-dc71-040f92c3fb8e"
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "AIMessage(content=\" I'm an AI based based on LLaMA models (LLaMA: Open and Efficient Foundation Language Models, Touvron et al. 2023), my knowledge was built from a massive corpus of text, including books, articles, and websites, and I was trained using a variety of machine learning algorithms. My model architecture is based on the transformer architecture, which is particularly well-suited for natural language processing tasks. My team of developers and I are constantly working to improve and fine-tune my performance, and I am always happy to help with any questions you may have!\", additional_kwargs={}, example=False)"
- ]
- },
- "execution_count": 27,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "os.environ['REPLICATE_API_TOKEN'] = \"\"\n",
- "chat = ChatLiteLLM(model=\"replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1\")\n",
- "messages = [\n",
- " HumanMessage(\n",
- " content=\"what model are you?\"\n",
- " )\n",
- "]\n",
- "chat(messages)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 30,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "tZxpq5PDDY9Y",
- "outputId": "7e86f4ed-ac7a-45e1-87d0-217da6cad666"
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "AIMessage(content=' I am an AI-based large language model, or Chatbot, built by the company Cohere. I am designed to have polite, helpful, inclusive conversations with users. I am always learning and improving, and I am constantly being updated with new information and improvements.\\n\\nI am currently in the development phase, and I am not yet available to the general public. However, I am currently being used by a select group of users for testing and feedback.\\n\\nI am a large language model, which means that I am trained on a massive amount of data and can understand and respond to a wide range of requests and questions. I am also designed to be flexible and adaptable, so I can be customized to suit the needs of different users and use cases.\\n\\nI am currently being used to develop a range of applications, including customer service chatbots, content generation tools, and language translation services. I am also being used to train other language models and to develop new ways of using large language models.\\n\\nI am constantly being updated with new information and improvements, so I am always learning and improving. I am also being used to develop new ways of using large language models, so I am always evolving and adapting to new use cases and requirements.', additional_kwargs={}, example=False)"
- ]
- },
- "execution_count": 30,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "os.environ['COHERE_API_KEY'] = \"\"\n",
- "chat = ChatLiteLLM(model=\"command-nightly\")\n",
- "messages = [\n",
- " HumanMessage(\n",
- " content=\"what model are you?\"\n",
- " )\n",
- "]\n",
- "chat(messages)"
- ]
- }
- ],
- "metadata": {
- "colab": {
- "provenance": []
- },
- "kernelspec": {
- "display_name": "Python 3",
- "name": "python3"
- },
- "language_info": {
- "name": "python"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}
\ No newline at end of file
diff --git a/cookbook/liteLLM_Ollama.ipynb b/cookbook/liteLLM_Ollama.ipynb
deleted file mode 100644
index fe0ed3811fa..00000000000
--- a/cookbook/liteLLM_Ollama.ipynb
+++ /dev/null
@@ -1,289 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "!pip install litellm # version 0.1.724 or higher "
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Call Ollama - llama2 with Streaming"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "{'role': 'assistant', 'content': ' I'}\n",
- "{'role': 'assistant', 'content': \"'\"}\n",
- "{'role': 'assistant', 'content': 'm'}\n",
- "{'role': 'assistant', 'content': ' L'}\n",
- "{'role': 'assistant', 'content': 'La'}\n",
- "{'role': 'assistant', 'content': 'MA'}\n",
- "{'role': 'assistant', 'content': ','}\n",
- "{'role': 'assistant', 'content': ' an'}\n",
- "{'role': 'assistant', 'content': ' A'}\n",
- "{'role': 'assistant', 'content': 'I'}\n",
- "{'role': 'assistant', 'content': ' assistant'}\n",
- "{'role': 'assistant', 'content': ' developed'}\n",
- "{'role': 'assistant', 'content': ' by'}\n",
- "{'role': 'assistant', 'content': ' Meta'}\n",
- "{'role': 'assistant', 'content': ' A'}\n",
- "{'role': 'assistant', 'content': 'I'}\n",
- "{'role': 'assistant', 'content': ' that'}\n",
- "{'role': 'assistant', 'content': ' can'}\n",
- "{'role': 'assistant', 'content': ' understand'}\n",
- "{'role': 'assistant', 'content': ' and'}\n",
- "{'role': 'assistant', 'content': ' respond'}\n",
- "{'role': 'assistant', 'content': ' to'}\n",
- "{'role': 'assistant', 'content': ' human'}\n",
- "{'role': 'assistant', 'content': ' input'}\n",
- "{'role': 'assistant', 'content': ' in'}\n",
- "{'role': 'assistant', 'content': ' a'}\n",
- "{'role': 'assistant', 'content': ' convers'}\n",
- "{'role': 'assistant', 'content': 'ational'}\n",
- "{'role': 'assistant', 'content': ' manner'}\n",
- "{'role': 'assistant', 'content': '.'}\n"
- ]
- }
- ],
- "source": [
- "from litellm import completion\n",
- "\n",
- "response = completion(\n",
- " model=\"ollama/llama2\", \n",
- " messages=[{ \"content\": \"respond in 20 words. who are you?\",\"role\": \"user\"}], \n",
- " api_base=\"http://localhost:11434\",\n",
- " stream=True\n",
- ")\n",
- "print(response)\n",
- "for chunk in response:\n",
- " print(chunk['choices'][0]['delta'])\n"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Call Ollama - Llama2 with Acompletion + Streaming"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Defaulting to user installation because normal site-packages is not writeable\n",
- "Requirement already satisfied: async_generator in /Users/ishaanjaffer/Library/Python/3.9/lib/python/site-packages (1.10)\n"
- ]
- }
- ],
- "source": [
- "# litellm uses async_generator for ollama async streaming, ensure it's installed\n",
- "!pip install async_generator"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' I'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': \"'\"}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': 'm'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' just'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' an'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' A'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': 'I'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ','}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' I'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' don'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': \"'\"}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': 't'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' have'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' access'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' to'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' real'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': '-'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': 'time'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' weather'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' information'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' or'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' current'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' conditions'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' in'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' your'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' specific'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' location'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': '.'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' Š¶ŠøŠ²ŠµŠ»Š¾'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' can'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' provide'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' you'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' with'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' weather'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' forec'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': 'asts'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' and'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' information'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' for'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' your'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' location'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' if'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' you'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' would'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' like'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': '.'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' Please'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' let'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' me'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' know'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' where'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' you'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' are'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' located'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ','}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' and'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' I'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' will'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' do'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' my'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' best'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' to'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' assist'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': ' you'}}]}\n",
- "{'choices': [{'delta': {'role': 'assistant', 'content': '.'}}]}\n",
- "None\n"
- ]
- }
- ],
- "source": [
- "import litellm\n",
- "\n",
- "async def async_ollama():\n",
- " response = await litellm.acompletion(\n",
- " model=\"ollama/llama2\", \n",
- " messages=[{ \"content\": \"what's the weather\" ,\"role\": \"user\"}], \n",
- " api_base=\"http://localhost:11434\", \n",
- " stream=True\n",
- " )\n",
- " async for chunk in response:\n",
- " print(chunk)\n",
- "\n",
- "result = await async_ollama()\n",
- "print(result)\n",
- "\n",
- "try:\n",
- " async for chunk in result:\n",
- " print(chunk)\n",
- "except TypeError: # the last chunk is None from Ollama, this raises an error with async streaming\n",
- " pass"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Completion Call"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "{\n",
- " \"object\": \"chat.completion\",\n",
- " \"choices\": [\n",
- " {\n",
- " \"finish_reason\": \"stop\",\n",
- " \"index\": 0,\n",
- " \"message\": {\n",
- " \"content\": \" I'm LLaMA, an AI assistant developed by Meta AI that can understand and respond to human input in a conversational manner.\",\n",
- " \"role\": \"assistant\",\n",
- " \"logprobs\": null\n",
- " }\n",
- " }\n",
- " ],\n",
- " \"id\": \"chatcmpl-ea7b8242-791f-4656-ba12-e098edeb960e\",\n",
- " \"created\": 1695324686.6696231,\n",
- " \"response_ms\": 4072.3050000000003,\n",
- " \"model\": \"ollama/llama2\",\n",
- " \"usage\": {\n",
- " \"prompt_tokens\": 10,\n",
- " \"completion_tokens\": 27,\n",
- " \"total_tokens\": 37\n",
- " }\n",
- "}\n"
- ]
- }
- ],
- "source": [
- "from litellm import completion\n",
- "\n",
- "response = completion(\n",
- " model=\"ollama/llama2\", \n",
- " messages=[{ \"content\": \"respond in 20 words. who are you?\",\"role\": \"user\"}], \n",
- " api_base=\"http://localhost:11434\"\n",
- ")\n",
- "print(response)\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.6"
- },
- "orig_nbformat": 4
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/cookbook/liteLLM_Replicate_Demo.ipynb b/cookbook/liteLLM_Replicate_Demo.ipynb
deleted file mode 100644
index b93d9a587b8..00000000000
--- a/cookbook/liteLLM_Replicate_Demo.ipynb
+++ /dev/null
@@ -1,238 +0,0 @@
-{
- "cells": [
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "YV6L5fNv7Kep"
- },
- "source": [
- "# Call Replicate LLMs using chatGPT Input/Output Format\n",
- "This tutorial covers using the following Replicate Models with liteLLM\n",
- "\n",
- "- [StableLM Tuned Alpha 7B](https://replicate.com/stability-ai/stablelm-tuned-alpha-7b)\n",
- "- [LLAMA-2 70B Chat](https://replicate.com/replicate/llama-2-70b-chat)\n",
- "- [A16z infra-LLAMA-2 7B Chat](https://replicate.com/a16z-infra/llama-2-7b-chat)\n",
- "- [Dolly V2 12B](https://replicate.com/replicate/dolly-v2-12b)\n",
- "- [Vicuna 13B](https://replicate.com/replicate/vicuna-13b)\n",
- "\n",
- "\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "TO-EdF84O9QT"
- },
- "outputs": [],
- "source": [
- "# install liteLLM\n",
- "!pip install litellm"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "mpHTbTqQ8fey"
- },
- "source": [
- "Imports & Set ENV variables\n",
- "Get your Replicate Key: https://replicate.com/account/api-tokens"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {
- "id": "kDbgfcU8O-dW"
- },
- "outputs": [],
- "source": [
- "from litellm import completion\n",
- "import os\n",
- "os.environ['REPLICATE_API_TOKEN'] = ' ' # @param\n",
- "user_message = \"Hello, whats the weather in San Francisco??\"\n",
- "messages = [{ \"content\": user_message,\"role\": \"user\"}]"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {
- "id": "1KmkOdzLSOmJ"
- },
- "source": [
- "## Call Replicate Models using completion(model, messages) - chatGPT format"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "XJ4nh4SnRzHP",
- "outputId": "986c0544-bb40-4915-f00f-498b0e518307"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "replicate is not installed. Installing...\n",
- "Response from stability-ai/stablelm-tuned-alpha-7b:c49dae362cbaecd2ceabb5bd34fdb68413c4ff775111fea065d259d577757beb \n",
- "]\n",
- "\n",
- "{'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'role': 'assistant', 'content': \"I'm sorry for you being unable to access this content as my training data only goes up until 2023/03. However I can tell you what your local weather forecast may look like at any time of year with respect to current conditions:\"}}], 'created': 1691611730.7224207, 'model': 'stability-ai/stablelm-tuned-alpha-7b:c49dae362cbaecd2ceabb5bd34fdb68413c4ff775111fea065d259d577757beb', 'usage': {'prompt_tokens': 9, 'completion_tokens': 49, 'total_tokens': 58}}\n",
- "Response from replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1 \n",
- "]\n",
- "\n",
- "{'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'role': 'assistant', 'content': \" Hello! I'm happy to help you with your question. However, I must point out that the question itself may not be meaningful. San Francisco is a city located in California, USA, and it is not possible for me to provide you with the current weather conditions there as I am a text-based AI language model and do not have access to real-time weather data. Additionally, the weather in San Francisco can vary greatly depending on the time of year, so it would be best to check a reliable weather source for the most up-to-date information.\\n\\nIf you meant to ask a different question, please feel free to rephrase it, and I will do my best to assist you in a safe and positive manner.\"}}], 'created': 1691611745.0269957, 'model': 'replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1', 'usage': {'prompt_tokens': 9, 'completion_tokens': 143, 'total_tokens': 152}}\n",
- "Response from a16z-infra/llama-2-7b-chat:4f0b260b6a13eb53a6b1891f089d57c08f41003ae79458be5011303d81a394dc \n",
- "]\n",
- "\n",
- "{'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'role': 'assistant', 'content': \" Hello! I'm here to help you with your question. However, I must inform you that the weather in San Francisco can be quite unpredictable and can change rapidly. It's important to check reliable sources such as AccuWeather or the National Weather Service for the most up-to-date and accurate information about the weather in San Francisco.\\nI cannot provide you with real-time weather data or forecasts as I'm just an AI and do not have access to current weather conditions or predictions. But I can suggest some trustworthy websites or apps where you can find the latest weather updates:\\n* AccuWeather (accuweather.com)\\n* The Weather Channel (weather.com)\\n* Dark Sky (darksky.net)\\n* Weather Underground (wunderground.com)\\nRemember, it's always best to consult multiple sources for the most accurate information when planning your day or trip. Enjoy your day!\"}}], 'created': 1691611748.7723358, 'model': 'a16z-infra/llama-2-7b-chat:4f0b260b6a13eb53a6b1891f089d57c08f41003ae79458be5011303d81a394dc', 'usage': {'prompt_tokens': 9, 'completion_tokens': 174, 'total_tokens': 183}}\n",
- "Response from replicate/dolly-v2-12b:ef0e1aefc61f8e096ebe4db6b2bacc297daf2ef6899f0f7e001ec445893500e5 \n",
- "]\n",
- "\n",
- "{'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'role': 'assistant', 'content': 'Its 68 degrees right now in San Francisco! The temperature will be rising through the week and i expect it to reach 70 on Thursdays and Friday. Skies are expected to be partly cloudy with some sun breaks throughout the day.\\n\\n'}}], 'created': 1691611752.2002115, 'model': 'replicate/dolly-v2-12b:ef0e1aefc61f8e096ebe4db6b2bacc297daf2ef6899f0f7e001ec445893500e5', 'usage': {'prompt_tokens': 9, 'completion_tokens': 48, 'total_tokens': 57}}\n",
- "Response from replicate/vicuna-13b:6282abe6a492de4145d7bb601023762212f9ddbbe78278bd6771c8b3b2f2a13b \n",
- "]\n",
- "\n",
- "{'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'role': 'assistant', 'content': ''}}], 'created': 1691611752.8998356, 'model': 'replicate/vicuna-13b:6282abe6a492de4145d7bb601023762212f9ddbbe78278bd6771c8b3b2f2a13b', 'usage': {'prompt_tokens': 9, 'completion_tokens': 0, 'total_tokens': 9}}\n"
- ]
- }
- ],
- "source": [
- "llama_2 = \"replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1\"\n",
- "llama_2_7b = \"a16z-infra/llama-2-7b-chat:4f0b260b6a13eb53a6b1891f089d57c08f41003ae79458be5011303d81a394dc\"\n",
- "dolly_v2 = \"replicate/dolly-v2-12b:ef0e1aefc61f8e096ebe4db6b2bacc297daf2ef6899f0f7e001ec445893500e5\"\n",
- "vicuna = \"replicate/vicuna-13b:6282abe6a492de4145d7bb601023762212f9ddbbe78278bd6771c8b3b2f2a13b\"\n",
- "models = [llama_2, llama_2_7b, dolly_v2, vicuna]\n",
- "for model in models:\n",
- " response = completion(model=model, messages=messages)\n",
- " print(f\"Response from {model} \\n]\\n\")\n",
- " print(response)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "zlTVLB-7PTV_",
- "outputId": "5182275b-3108-46fa-a2cf-745fac4ad110"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Hi\n",
- " there!\n",
- " The\n",
- " current\n",
- " forecast\n",
- " for\n",
- " today's\n",
- " high\n",
- " temperature\n",
- " ranges\n",
- " from\n",
- " 75\n",
- " degrees\n",
- " Fahrenheit\n",
- " all\n",
- " day\n",
- " to\n",
- " 83\n",
- " degrees\n",
- " Fahrenheit\n",
- " with\n",
- " possible\n",
- " isolated\n",
- " thunderstorms\n",
- " during\n",
- " the\n",
- " afternoon\n",
- " hours,\n",
- " mainly\n",
- " at\n",
- " sunset\n",
- " through\n",
- " early\n",
- " evening. The\n",
- " Pacific\n",
- " Ocean\n",
- " has\n",
- " a\n",
- " low\n",
- " pressure\n",
- " of\n",
- " 926\n",
- " mb\n",
- " and\n",
- " mostly\n",
- " cloud\n",
- " cover\n",
- " in\n",
- " this\n",
- " region\n",
- " on\n",
- " sunny\n",
- " days\n",
- " due\n",
- " to\n",
- " warming\n",
- " temperatures\n",
- " above\n",
- " average\n",
- " along\n",
- " most\n",
- " coastal\n",
- " areas\n",
- " and\n",
- " ocean\n",
- " breezes.<|USER|>\n"
- ]
- }
- ],
- "source": [
- "# @title Stream Responses from Replicate - Outputs in the same format used by chatGPT streaming\n",
- "response = completion(model=llama_2, messages=messages, stream=True)\n",
- "\n",
- "for chunk in response:\n",
- " print(chunk['choices'][0]['delta'])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "t7WMRuL-8NrO"
- },
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "colab": {
- "provenance": []
- },
- "kernelspec": {
- "display_name": "Python 3",
- "name": "python3"
- },
- "language_info": {
- "name": "python"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}
diff --git a/cookbook/liteLLM_Streaming_Demo.ipynb b/cookbook/liteLLM_Streaming_Demo.ipynb
deleted file mode 100644
index 0456c54514b..00000000000
--- a/cookbook/liteLLM_Streaming_Demo.ipynb
+++ /dev/null
@@ -1,226 +0,0 @@
-{
- "nbformat": 4,
- "nbformat_minor": 0,
- "metadata": {
- "colab": {
- "provenance": []
- },
- "kernelspec": {
- "name": "python3",
- "display_name": "Python 3"
- },
- "language_info": {
- "name": "python"
- }
- },
- "cells": [
- {
- "cell_type": "markdown",
- "source": [
- "# [STREAMING] OpenAI, Anthropic, Replicate, Cohere using liteLLM\n",
- "In this tutorial:\n",
- "Note: All inputs/outputs are in the format used by `gpt-3.5-turbo`\n",
- "\n",
- "- Call all models in the same input format [**with streaming**]:\n",
- "\n",
- " `completion(model, messages, stream=True)`\n",
- "- All streaming generators are accessed at `chunk['choices'][0]['delta']`\n",
- "\n",
- "The following Models are covered in this tutorial\n",
- "- [GPT-3.5-Turbo](https://platform.openai.com/docs/models/gpt-3-5)\n",
- "- [Claude-2](https://www.anthropic.com/index/claude-2)\n",
- "- [StableLM Tuned Alpha 7B](https://replicate.com/stability-ai/stablelm-tuned-alpha-7b)\n",
- "- [A16z infra-LLAMA-2 7B Chat](https://replicate.com/a16z-infra/llama-2-7b-chat)\n",
- "- [Vicuna 13B](https://replicate.com/replicate/vicuna-13b)\n",
- "- [Cohere - Command Nightly]()\n",
- "\n",
- "\n",
- "\n"
- ],
- "metadata": {
- "id": "YV6L5fNv7Kep"
- }
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "TO-EdF84O9QT"
- },
- "outputs": [],
- "source": [
- "# install liteLLM\n",
- "!pip install litellm==0.1.369"
- ]
- },
- {
- "cell_type": "markdown",
- "source": [
- "## Imports & Set ENV variables\n",
- "Get your API Keys\n",
- "\n",
- "https://platform.openai.com/account/api-keys\n",
- "\n",
- "https://replicate.com/account/api-tokens\n",
- "\n",
- "https://console.anthropic.com/account/keys\n",
- "\n",
- "https://dashboard.cohere.ai/api-keys\n"
- ],
- "metadata": {
- "id": "mpHTbTqQ8fey"
- }
- },
- {
- "cell_type": "code",
- "source": [
- "from litellm import completion\n",
- "import os\n",
- "\n",
- "os.environ['OPENAI_API_KEY'] = '' # @param\n",
- "os.environ['REPLICATE_API_TOKEN'] = '' # @param\n",
- "os.environ['ANTHROPIC_API_KEY'] = '' # @param\n",
- "os.environ['COHERE_API_KEY'] = '' # @param"
- ],
- "metadata": {
- "id": "kDbgfcU8O-dW"
- },
- "execution_count": 8,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "source": [
- "### Set Messages"
- ],
- "metadata": {
- "id": "1KmkOdzLSOmJ"
- }
- },
- {
- "cell_type": "code",
- "source": [
- "user_message = \"Hello, whats the weather in San Francisco??\"\n",
- "messages = [{ \"content\": user_message,\"role\": \"user\"}]"
- ],
- "metadata": {
- "id": "xIEeOhVH-oh6"
- },
- "execution_count": 4,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "source": [
- "## Calling Models using liteLLM Streaming -\n",
- "\n",
- "## `completion(model, messages, stream)`"
- ],
- "metadata": {
- "id": "9SOCVRC1L-G3"
- }
- },
- {
- "cell_type": "code",
- "source": [
- "# replicate models #######\n",
- "stability_ai = \"stability-ai/stablelm-tuned-alpha-7b:c49dae362cbaecd2ceabb5bd34fdb68413c4ff775111fea065d259d577757beb\"\n",
- "llama_2_7b = \"a16z-infra/llama-2-7b-chat:4f0b260b6a13eb53a6b1891f089d57c08f41003ae79458be5011303d81a394dc\"\n",
- "vicuna = \"replicate/vicuna-13b:6282abe6a492de4145d7bb601023762212f9ddbbe78278bd6771c8b3b2f2a13b\"\n",
- "\n",
- "models = [\"gpt-3.5-turbo\", \"claude-2\", stability_ai, llama_2_7b, vicuna, \"command-nightly\"] # command-nightly is Cohere\n",
- "for model in models:\n",
- " replicate = (model == stability_ai or model==llama_2_7b or model==vicuna) # let liteLLM know if a model is replicate, using this optional param, `replicate=True`\n",
- " response = completion(model=model, messages=messages, stream=True, replicate=replicate)\n",
- " print(f\"####################\\n\\nResponse from {model}\")\n",
- " for i, chunk in enumerate(response):\n",
- " if i < 5: # NOTE: LIMITING CHUNKS FOR THIS DEMO\n",
- " print((chunk['choices'][0]['delta']))\n"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "XJ4nh4SnRzHP",
- "outputId": "26b9fe10-b499-4a97-d60d-a8cb8f8030b8"
- },
- "execution_count": 13,
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "####################\n",
- "\n",
- "Response from gpt-3.5-turbo\n",
- "{\n",
- " \"role\": \"assistant\",\n",
- " \"content\": \"\"\n",
- "}\n",
- "{\n",
- " \"content\": \"I\"\n",
- "}\n",
- "{\n",
- " \"content\": \"'m\"\n",
- "}\n",
- "{\n",
- " \"content\": \" sorry\"\n",
- "}\n",
- "{\n",
- " \"content\": \",\"\n",
- "}\n",
- "####################\n",
- "\n",
- "Response from claude-2\n",
- "{'role': 'assistant', 'content': ' Unfortunately'}\n",
- "{'role': 'assistant', 'content': ' I'}\n",
- "{'role': 'assistant', 'content': ' don'}\n",
- "{'role': 'assistant', 'content': \"'t\"}\n",
- "{'role': 'assistant', 'content': ' have'}\n",
- "####################\n",
- "\n",
- "Response from stability-ai/stablelm-tuned-alpha-7b:c49dae362cbaecd2ceabb5bd34fdb68413c4ff775111fea065d259d577757beb\n",
- "{'role': 'assistant', 'content': \"I'm\"}\n",
- "{'role': 'assistant', 'content': ' sorry,'}\n",
- "{'role': 'assistant', 'content': ' I'}\n",
- "{'role': 'assistant', 'content': ' cannot'}\n",
- "{'role': 'assistant', 'content': ' answer'}\n",
- "####################\n",
- "\n",
- "Response from a16z-infra/llama-2-7b-chat:4f0b260b6a13eb53a6b1891f089d57c08f41003ae79458be5011303d81a394dc\n",
- "{'role': 'assistant', 'content': ''}\n",
- "{'role': 'assistant', 'content': ' Hello'}\n",
- "{'role': 'assistant', 'content': '!'}\n",
- "{'role': 'assistant', 'content': ' I'}\n",
- "{'role': 'assistant', 'content': \"'\"}\n",
- "####################\n",
- "\n",
- "Response from replicate/vicuna-13b:6282abe6a492de4145d7bb601023762212f9ddbbe78278bd6771c8b3b2f2a13b\n",
- "{'role': 'assistant', 'content': 'Comment:'}\n",
- "{'role': 'assistant', 'content': 'Hi! '}\n",
- "{'role': 'assistant', 'content': 'How '}\n",
- "{'role': 'assistant', 'content': 'are '}\n",
- "{'role': 'assistant', 'content': 'you '}\n",
- "####################\n",
- "\n",
- "Response from command-nightly\n",
- "{'role': 'assistant', 'content': ' Hello'}\n",
- "{'role': 'assistant', 'content': '!'}\n",
- "{'role': 'assistant', 'content': ' '}\n",
- "{'role': 'assistant', 'content': ' I'}\n",
- "{'role': 'assistant', 'content': \"'m\"}\n"
- ]
- }
- ]
- },
- {
- "cell_type": "code",
- "source": [],
- "metadata": {
- "id": "t7WMRuL-8NrO"
- },
- "execution_count": null,
- "outputs": []
- }
- ]
-}
\ No newline at end of file
diff --git a/cookbook/liteLLM_VertextAI_Example.ipynb b/cookbook/liteLLM_VertextAI_Example.ipynb
deleted file mode 100644
index 0af012b34e3..00000000000
--- a/cookbook/liteLLM_VertextAI_Example.ipynb
+++ /dev/null
@@ -1,199 +0,0 @@
-{
- "cells": [
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Using Google Palm (VertexAI) with liteLLM \n",
- "### chat-bison, chat-bison@001, text-bison, text-bison@001"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "!pip install litellm==0.1.388"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Set VertexAI Configs\n",
- "Vertex AI requires the following:\n",
- "* `vertex_project` - Your Project ID\n",
- "* `vertex_location` - Your Vertex AI region\n",
- "Both can be found on: https://console.cloud.google.com/\n",
- "\n",
- "VertexAI uses Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information on setting this up\n",
- "\n",
- "NOTE: VertexAI requires you to set `application_default_credentials.json`, this can be set by running `gcloud auth application-default login` in your terminal\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": [
- "# set you Vertex AI configs\n",
- "import litellm\n",
- "from litellm import completion\n",
- "\n",
- "litellm.vertex_project = \"hardy-device-386718\"\n",
- "litellm.vertex_location = \"us-central1\""
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Call VertexAI - chat-bison using liteLLM"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "{'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'role': 'assistant', 'content': LiteLLM LiteLLM is a large language model from Google AI that is designed to be lightweight and efficient. It is based on the Transformer architecture and has been trained on a massive dataset of text. LiteLLM is available as a pre-trained model that can be used for a variety of natural language processing tasks, such as text classification, question answering, and summarization.}}], 'created': 1692036777.831989, 'model': 'chat-bison'}\n"
- ]
- }
- ],
- "source": [
- "user_message = \"what is liteLLM \"\n",
- "messages = [{ \"content\": user_message,\"role\": \"user\"}]\n",
- "\n",
- "# chat-bison or chat-bison@001 supported by Vertex AI (As of Aug 2023)\n",
- "response = completion(model=\"chat-bison\", messages=messages)\n",
- "print(response)"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Call VertexAI - text-bison using liteLLM"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "['text-bison', 'text-bison@001']\n"
- ]
- }
- ],
- "source": [
- "print(litellm.vertex_text_models)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "{'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'role': 'assistant', 'content': liteLLM is a low-precision variant of the large language model LLM 5. For a given text prompt, liteLLM can continue the text in a way that is both coherent and informative.}}], 'created': 1692036813.052487, 'model': 'text-bison@001'}\n"
- ]
- }
- ],
- "source": [
- "user_message = \"what is liteLLM \"\n",
- "messages = [{ \"content\": user_message,\"role\": \"user\"}]\n",
- "\n",
- "# text-bison or text-bison@001 supported by Vertex AI (As of Aug 2023)\n",
- "response = completion(model=\"text-bison@001\", messages=messages)\n",
- "print(response)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "{'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'role': 'assistant', 'content': liteLLM was originally developed by Google engineers as a lite version of LLM, which stands for large language model. It is a deep learning language model that is designed to be more efficient than traditional LLMs while still achieving comparable performance. liteLLM is built on Tensor2Tensor, a framework for building and training large neural networks. It is able to learn from massive amounts of text data and generate text that is both coherent and informative. liteLLM has been shown to be effective for a variety of tasks, including machine translation, text summarization, and question answering.}}], 'created': 1692036821.60951, 'model': 'text-bison'}\n"
- ]
- }
- ],
- "source": [
- "response = completion(model=\"text-bison\", messages=messages)\n",
- "print(response)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "liteLLM is a lightweight language model that is designed to be fast and efficient. It is based on the Transformer architecture, but it has been modified to reduce the number of parameters and the amount of computation required. This makes it suitable for use on devices with limited resources, such as mobile phones and embedded systems.\n",
- "\n",
- "liteLLM is still under development, but it has already been shown to be effective on a variety of tasks, including text classification, natural language inference, and machine translation. It is also being used to develop new applications, such as chatbots and language assistants.\n",
- "\n",
- "If you are interested in learning more about lite\n"
- ]
- }
- ],
- "source": [
- "response = completion(model=\"text-bison@001\", messages=messages, temperature=0.4, top_k=10, top_p=0.2)\n",
- "print(response['choices'][0]['message']['content'])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.6"
- },
- "orig_nbformat": 4
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/cookbook/liteLLM_clarifai_Demo.ipynb b/cookbook/liteLLM_clarifai_Demo.ipynb
deleted file mode 100644
index 40ef2fcf932..00000000000
--- a/cookbook/liteLLM_clarifai_Demo.ipynb
+++ /dev/null
@@ -1,187 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# LiteLLM Clarifai \n",
- "This notebook walks you through on how to use liteLLM integration of Clarifai and call LLM model from clarifai with response in openAI output format."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Pre-Requisites"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "#install necessary packages\n",
- "!pip install litellm\n",
- "!pip install clarifai"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "To obtain Clarifai Personal Access Token follow the steps mentioned in the [link](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens/)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "## Set Clarifai Credentials\n",
- "import os\n",
- "os.environ[\"CLARIFAI_API_KEY\"]= \"YOUR_CLARIFAI_PAT\" # Clarifai PAT"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Mistral-large"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": [
- "import litellm\n",
- "\n",
- "litellm.set_verbose=False"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Mistral large response : ModelResponse(id='chatcmpl-6eed494d-7ae2-4870-b9c2-6a64d50a6151', choices=[Choices(finish_reason='stop', index=1, message=Message(content=\"In the grand tapestry of time, where tales unfold,\\nLies the chronicle of ages, a sight to behold.\\nA tale of empires rising, and kings of old,\\nOf civilizations lost, and stories untold.\\n\\nOnce upon a yesterday, in a time so vast,\\nHumans took their first steps, casting shadows in the past.\\nFrom the cradle of mankind, a journey they embarked,\\nThrough stone and bronze and iron, their skills they sharpened and marked.\\n\\nEgyptians built pyramids, reaching for the skies,\\nWhile Greeks sought wisdom, truth, in philosophies that lie.\\nRoman legions marched, their empire to expand,\\nAnd in the East, the Silk Road joined the world, hand in hand.\\n\\nThe Middle Ages came, with knights in shining armor,\\nFeudal lords and serfs, a time of both clamor and calm order.\\nThen Renaissance bloomed, like a flower in the sun,\\nA rebirth of art and science, a new age had begun.\\n\\nAcross the vast oceans, explorers sailed with courage bold,\\nDiscovering new lands, stories of adventure, untold.\\nIndustrial Revolution churned, progress in its wake,\\nMachines and factories, a whole new world to make.\\n\\nTwo World Wars raged, a testament to man's strife,\\nYet from the ashes rose hope, a renewed will for life.\\nInto the modern era, technology took flight,\\nConnecting every corner, bathed in digital light.\\n\\nHistory, a symphony, a melody of time,\\nA testament to human will, resilience so sublime.\\nIn every page, a lesson, in every tale, a guide,\\nFor understanding our past, shapes our future's tide.\", role='assistant'))], created=1713896412, model='https://api.clarifai.com/v2/users/mistralai/apps/completion/models/mistral-large/outputs', object='chat.completion', system_fingerprint=None, usage=Usage(prompt_tokens=13, completion_tokens=338, total_tokens=351))\n"
- ]
- }
- ],
- "source": [
- "from litellm import completion\n",
- "\n",
- "messages = [{\"role\": \"user\",\"content\": \"\"\"Write a poem about history?\"\"\"}]\n",
- "response=completion(\n",
- " model=\"clarifai/mistralai.completion.mistral-large\",\n",
- " messages=messages,\n",
- " )\n",
- "\n",
- "print(f\"Mistral large response : {response}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Claude-2.1 "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Claude-2.1 response : ModelResponse(id='chatcmpl-d126c919-4db4-4aa3-ac8f-7edea41e0b93', choices=[Choices(finish_reason='stop', index=1, message=Message(content=\" Here's a poem I wrote about history:\\n\\nThe Tides of Time\\n\\nThe tides of time ebb and flow,\\nCarrying stories of long ago.\\nFigures and events come into light,\\nShaping the future with all their might.\\n\\nKingdoms rise, empires fall, \\nLeaving traces that echo down every hall.\\nRevolutions bring change with a fiery glow,\\nToppling structures from long ago.\\n\\nExplorers traverse each ocean and land,\\nSeeking treasures they don't understand.\\nWhile artists and writers try to make their mark,\\nHoping their works shine bright in the dark.\\n\\nThe cycle repeats again and again,\\nAs humanity struggles to learn from its pain.\\nThough the players may change on history's stage,\\nThe themes stay the same from age to age.\\n\\nWar and peace, life and death,\\nLove and strife with every breath.\\nThe tides of time continue their dance,\\nAs we join in, by luck or by chance.\\n\\nSo we study the past to light the way forward, \\nHeeding warnings from stories told and heard.\\nThe future unfolds from this unending flow -\\nWhere the tides of time ultimately go.\", role='assistant'))], created=1713896579, model='https://api.clarifai.com/v2/users/anthropic/apps/completion/models/claude-2_1/outputs', object='chat.completion', system_fingerprint=None, usage=Usage(prompt_tokens=12, completion_tokens=232, total_tokens=244))\n"
- ]
- }
- ],
- "source": [
- "from litellm import completion\n",
- "\n",
- "messages = [{\"role\": \"user\",\"content\": \"\"\"Write a poem about history?\"\"\"}]\n",
- "response=completion(\n",
- " model=\"clarifai/anthropic.completion.claude-2_1\",\n",
- " messages=messages,\n",
- " )\n",
- "\n",
- "print(f\"Claude-2.1 response : {response}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### OpenAI GPT-4 (Streaming)\n",
- "Though clarifai doesn't support streaming, still you can call stream and get the response in standard StreamResponse format of liteLLM"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "ModelResponse(id='chatcmpl-40ae19af-3bf0-4eb4-99f2-33aec3ba84af', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content=\"In the quiet corners of time's grand hall,\\nLies the tale of rise and fall.\\nFrom ancient ruins to modern sprawl,\\nHistory, the greatest story of them all.\\n\\nEmpires have risen, empires have decayed,\\nThrough the eons, memories have stayed.\\nIn the book of time, history is laid,\\nA tapestry of events, meticulously displayed.\\n\\nThe pyramids of Egypt, standing tall,\\nThe Roman Empire's mighty sprawl.\\nFrom Alexander's conquest, to the Berlin Wall,\\nHistory, a silent witness to it all.\\n\\nIn the shadow of the past we tread,\\nWhere once kings and prophets led.\\nTheir stories in our hearts are spread,\\nEchoes of their words, in our minds are read.\\n\\nBattles fought and victories won,\\nActs of courage under the sun.\\nTales of love, of deeds done,\\nIn history's grand book, they all run.\\n\\nHeroes born, legends made,\\nIn the annals of time, they'll never fade.\\nTheir triumphs and failures all displayed,\\nIn the eternal march of history's parade.\\n\\nThe ink of the past is forever dry,\\nBut its lessons, we cannot deny.\\nIn its stories, truths lie,\\nIn its wisdom, we rely.\\n\\nHistory, a mirror to our past,\\nA guide for the future vast.\\nThrough its lens, we're ever cast,\\nIn the drama of life, forever vast.\", role='assistant', function_call=None, tool_calls=None), logprobs=None)], created=1714744515, model='https://api.clarifai.com/v2/users/openai/apps/chat-completion/models/GPT-4/outputs', object='chat.completion.chunk', system_fingerprint=None)\n",
- "ModelResponse(id='chatcmpl-40ae19af-3bf0-4eb4-99f2-33aec3ba84af', choices=[StreamingChoices(finish_reason='stop', index=0, delta=Delta(content=None, role=None, function_call=None, tool_calls=None), logprobs=None)], created=1714744515, model='https://api.clarifai.com/v2/users/openai/apps/chat-completion/models/GPT-4/outputs', object='chat.completion.chunk', system_fingerprint=None)\n"
- ]
- }
- ],
- "source": [
- "from litellm import completion\n",
- "\n",
- "messages = [{\"role\": \"user\",\"content\": \"\"\"Write a poem about history?\"\"\"}]\n",
- "response = completion(\n",
- " model=\"clarifai/openai.chat-completion.GPT-4\",\n",
- " messages=messages,\n",
- " stream=True,\n",
- " api_key = \"c75cc032415e45368be331fdd2c06db0\")\n",
- "\n",
- "for chunk in response:\n",
- " print(chunk)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.13"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/cookbook/liteLLM_function_calling.ipynb b/cookbook/liteLLM_function_calling.ipynb
deleted file mode 100644
index 45f4398b386..00000000000
--- a/cookbook/liteLLM_function_calling.ipynb
+++ /dev/null
@@ -1,331 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "vnvlwUDZK7VA"
- },
- "source": [
- "## Demo Notebook of Function Calling with liteLLM\n",
- "- Supported Providers for Function Calling\n",
- " - OpenAI - `gpt-4-0613` and `gpt-3.5-turbo-0613`\n",
- "- In this notebook we use function calling with `litellm.completion()`"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "KrINCwRfLgZV"
- },
- "outputs": [],
- "source": [
- "## Install liteLLM\n",
- "!pip install litellm"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {
- "id": "nK7zR5OgLlh2"
- },
- "outputs": [],
- "source": [
- "import os\n",
- "from litellm import completion"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 27,
- "metadata": {
- "id": "dCQlyBxKLqbA"
- },
- "outputs": [],
- "source": [
- "os.environ['OPENAI_API_KEY'] = \"\" #@param"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "gfdGv-FMRCdX"
- },
- "source": [
- "## Define Messages, Functions\n",
- "We create a get_current_weather() function and pass that to GPT 3.5\n",
- "\n",
- "See OpenAI docs for this: https://openai.com/blog/function-calling-and-other-api-updates"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 25,
- "metadata": {
- "id": "ERzsP1sfM19C"
- },
- "outputs": [],
- "source": [
- "messages = [\n",
- " {\"role\": \"user\", \"content\": \"What is the weather like in Boston?\"}\n",
- "]\n",
- "\n",
- "def get_current_weather(location):\n",
- " if location == \"Boston, MA\":\n",
- " return \"The weather is 12F\"\n",
- "\n",
- "functions = [\n",
- " {\n",
- " \"name\": \"get_current_weather\",\n",
- " \"description\": \"Get the current weather in a given location\",\n",
- " \"parameters\": {\n",
- " \"type\": \"object\",\n",
- " \"properties\": {\n",
- " \"location\": {\n",
- " \"type\": \"string\",\n",
- " \"description\": \"The city and state, e.g. San Francisco, CA\"\n",
- " },\n",
- " \"unit\": {\n",
- " \"type\": \"string\",\n",
- " \"enum\": [\"celsius\", \"fahrenheit\"]\n",
- " }\n",
- " },\n",
- " \"required\": [\"location\"]\n",
- " }\n",
- " }\n",
- " ]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "NX6by2VuRPnp"
- },
- "source": [
- "## Call gpt-3.5-turbo-0613 to Decide what Function to call"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "QVoJ5PtxMlVx",
- "outputId": "efe7a81f-e04a-4afc-aa60-a2b2648f5fb9"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "{\n",
- " \"id\": \"chatcmpl-7mX4RiqdoislVEqfmfVjFSKp3hyIy\",\n",
- " \"object\": \"chat.completion\",\n",
- " \"created\": 1691801223,\n",
- " \"model\": \"gpt-3.5-turbo-0613\",\n",
- " \"choices\": [\n",
- " {\n",
- " \"index\": 0,\n",
- " \"message\": {\n",
- " \"role\": \"assistant\",\n",
- " \"content\": null,\n",
- " \"function_call\": {\n",
- " \"name\": \"get_current_weather\",\n",
- " \"arguments\": \"{\\n \\\"location\\\": \\\"Boston, MA\\\"\\n}\"\n",
- " }\n",
- " },\n",
- " \"finish_reason\": \"function_call\"\n",
- " }\n",
- " ],\n",
- " \"usage\": {\n",
- " \"prompt_tokens\": 82,\n",
- " \"completion_tokens\": 18,\n",
- " \"total_tokens\": 100\n",
- " }\n",
- "}\n"
- ]
- }
- ],
- "source": [
- "response = completion(model=\"gpt-3.5-turbo-0613\", messages=messages, functions=functions)\n",
- "print(response)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "Yu0o2saDNLx8"
- },
- "source": [
- "## Parse GPT 3.5 Response\n",
- "Read Information about what Function to Call"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "u1DzXLJsNOR5",
- "outputId": "177e9501-0ce2-4619-9067-3047f18f6c79"
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- " JSON: {\n",
- " \"name\": \"get_current_weather\",\n",
- " \"arguments\": \"{\\n \\\"location\\\": \\\"Boston, MA\\\"\\n}\"\n",
- "}"
- ]
- },
- "execution_count": 11,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "function_call_data = response[\"choices\"][0][\"message\"][\"function_call\"]\n",
- "function_call_data"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 20,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "tYb96Mh0NhH9",
- "outputId": "13c4bb89-6f29-4b3b-afa7-302dcf2cdd5f"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "get_current_weather {'location': 'Boston, MA'}\n"
- ]
- }
- ],
- "source": [
- "import json\n",
- "function_name = function_call_data['name']\n",
- "function_args = function_call_data['arguments']\n",
- "function_args = json.loads(function_args)\n",
- "print(function_name, function_args)\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "z3tstH_yN3fX"
- },
- "source": [
- "## Call the get_current_weather() function"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 24,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "TSb8JHhgN5Zc",
- "outputId": "ef140572-4020-4daf-ac8c-d5161be9aa5c"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "12F\n"
- ]
- }
- ],
- "source": [
- "if function_name == \"get_current_weather\":\n",
- " result = get_current_weather(**function_args)\n",
- " print(result)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "k4HGJE3NRmMI"
- },
- "source": [
- "## Send the response from get_current_weather back to the model to summarize"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 26,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "a23cmEwiPaw7",
- "outputId": "43259b86-0c4c-4fcb-eab7-6e1a788b2f21"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "{\n",
- " \"id\": \"chatcmpl-7mXGN62u75WXp1Lgen4iSgNvA7hHT\",\n",
- " \"object\": \"chat.completion\",\n",
- " \"created\": 1691801963,\n",
- " \"model\": \"gpt-3.5-turbo-0613\",\n",
- " \"choices\": [\n",
- " {\n",
- " \"index\": 0,\n",
- " \"message\": {\n",
- " \"role\": \"assistant\",\n",
- " \"content\": \"The current weather in Boston is 12 degrees Fahrenheit.\"\n",
- " },\n",
- " \"finish_reason\": \"stop\"\n",
- " }\n",
- " ],\n",
- " \"usage\": {\n",
- " \"prompt_tokens\": 109,\n",
- " \"completion_tokens\": 12,\n",
- " \"total_tokens\": 121\n",
- " }\n",
- "}\n"
- ]
- }
- ],
- "source": [
- "messages = [\n",
- " {\"role\": \"user\", \"content\": \"What is the weather like in Boston?\"},\n",
- " {\"role\": \"assistant\", \"content\": None, \"function_call\": {\"name\": \"get_current_weather\", \"arguments\": \"{ \\\"location\\\": \\\"Boston, MA\\\"}\"}},\n",
- " {\"role\": \"function\", \"name\": \"get_current_weather\", \"content\": result}\n",
- "]\n",
- "response = completion(model=\"gpt-3.5-turbo-0613\", messages=messages, functions=functions)\n",
- "print(response)"
- ]
- }
- ],
- "metadata": {
- "colab": {
- "provenance": []
- },
- "kernelspec": {
- "display_name": "Python 3",
- "name": "python3"
- },
- "language_info": {
- "name": "python"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}
\ No newline at end of file
diff --git a/cookbook/litellm_Test_Multiple_Providers.ipynb b/cookbook/litellm_Test_Multiple_Providers.ipynb
deleted file mode 100644
index 3901581e67f..00000000000
--- a/cookbook/litellm_Test_Multiple_Providers.ipynb
+++ /dev/null
@@ -1,571 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "Ys9n20Es2IzT"
- },
- "source": [
- "# Evaluate Multiple LLM Providers with LiteLLM\n",
- "\n",
- "\n",
- "\n",
- "* Quality Testing\n",
- "* Load Testing\n",
- "* Duration Testing\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "ZXOXl23PIIP6"
- },
- "outputs": [],
- "source": [
- "!pip install litellm python-dotenv"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "LINuBzXDItq2"
- },
- "outputs": [],
- "source": [
- "from litellm import load_test_model, testing_batch_completion"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "EkxMhsWdJdu4"
- },
- "outputs": [],
- "source": [
- "from dotenv import load_dotenv\n",
- "load_dotenv()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "mv5XdnqeW5I_"
- },
- "source": [
- "# Quality Test endpoint\n",
- "\n",
- "## Test the same prompt across multiple LLM providers\n",
- "\n",
- "In this example, let's ask some questions about Paul Graham"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "XpzrR5m4W_Us"
- },
- "outputs": [],
- "source": [
- "models = [\"gpt-3.5-turbo\", \"gpt-3.5-turbo-16k\", \"gpt-4\", \"claude-instant-1\", \"replicate/llama-2-70b-chat:58d078176e02c219e11eb4da5a02a7830a283b14cf8f94537af893ccff5ee781\"]\n",
- "context = \"\"\"Paul Graham (/ɔrƦm/; born 1964)[3] is an English computer scientist, essayist, entrepreneur, venture capitalist, and author. He is best known for his work on the programming language Lisp, his former startup Viaweb (later renamed Yahoo! Store), cofounding the influential startup accelerator and seed capital firm Y Combinator, his essays, and Hacker News. He is the author of several computer programming books, including: On Lisp,[4] ANSI Common Lisp,[5] and Hackers & Painters.[6] Technology journalist Steven Levy has described Graham as a \"hacker philosopher\".[7] Graham was born in England, where he and his family maintain permanent residence. However he is also a citizen of the United States, where he was educated, lived, and worked until 2016.\"\"\"\n",
- "prompts = [\"Who is Paul Graham?\", \"What is Paul Graham known for?\" , \"Is paul graham a writer?\" , \"Where does Paul Graham live?\", \"What has Paul Graham done?\"]\n",
- "messages = [[{\"role\": \"user\", \"content\": context + \"\\n\" + prompt}] for prompt in prompts] # pass in a list of messages we want to test\n",
- "result = testing_batch_completion(models=models, messages=messages)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "9nzeLySnvIIW"
- },
- "source": [
- "## Visualize the data"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 403
- },
- "id": "X-2n7hdAuVAY",
- "outputId": "69cc0de1-68e3-4c12-a8ea-314880010d94"
- },
- "outputs": [
- {
- "data": {
- "text/html": [
- "\n",
- "\n",
- "
-
-
- Bar Graph Excel Sheet here
-
-
-| Model | Provider | Cost per input + output token ($)|
-| --- | --- | --- |
-| openrouter/mistralai/mistral-7b-instruct | openrouter | 0.0 |
-| ollama/llama2 | ollama | 0.0 |
-| ollama/llama2:13b | ollama | 0.0 |
-| ollama/llama2:70b | ollama | 0.0 |
-| ollama/llama2-uncensored | ollama | 0.0 |
-| ollama/mistral | ollama | 0.0 |
-| ollama/codellama | ollama | 0.0 |
-| ollama/orca-mini | ollama | 0.0 |
-| ollama/vicuna | ollama | 0.0 |
-| perplexity/codellama-34b-instruct | perplexity | 0.0 |
-| perplexity/llama-2-13b-chat | perplexity | 0.0 |
-| perplexity/llama-2-70b-chat | perplexity | 0.0 |
-| perplexity/mistral-7b-instruct | perplexity | 0.0 |
-| perplexity/replit-code-v1.5-3b | perplexity | 0.0 |
-| text-bison | vertex_ai-text-models | 0.00000025 |
-| text-bison@001 | vertex_ai-text-models | 0.00000025 |
-| chat-bison | vertex_ai-chat-models | 0.00000025 |
-| chat-bison@001 | vertex_ai-chat-models | 0.00000025 |
-| chat-bison-32k | vertex_ai-chat-models | 0.00000025 |
-| code-bison | vertex_ai-code-text-models | 0.00000025 |
-| code-bison@001 | vertex_ai-code-text-models | 0.00000025 |
-| code-gecko@001 | vertex_ai-chat-models | 0.00000025 |
-| code-gecko@latest | vertex_ai-chat-models | 0.00000025 |
-| codechat-bison | vertex_ai-code-chat-models | 0.00000025 |
-| codechat-bison@001 | vertex_ai-code-chat-models | 0.00000025 |
-| codechat-bison-32k | vertex_ai-code-chat-models | 0.00000025 |
-| palm/chat-bison | palm | 0.00000025 |
-| palm/chat-bison-001 | palm | 0.00000025 |
-| palm/text-bison | palm | 0.00000025 |
-| palm/text-bison-001 | palm | 0.00000025 |
-| palm/text-bison-safety-off | palm | 0.00000025 |
-| palm/text-bison-safety-recitation-off | palm | 0.00000025 |
-| anyscale/meta-llama/Llama-2-7b-chat-hf | anyscale | 0.0000003 |
-| anyscale/mistralai/Mistral-7B-Instruct-v0.1 | anyscale | 0.0000003 |
-| openrouter/meta-llama/llama-2-13b-chat | openrouter | 0.0000004 |
-| openrouter/nousresearch/nous-hermes-llama2-13b | openrouter | 0.0000004 |
-| deepinfra/meta-llama/Llama-2-7b-chat-hf | deepinfra | 0.0000004 |
-| deepinfra/mistralai/Mistral-7B-Instruct-v0.1 | deepinfra | 0.0000004 |
-| anyscale/meta-llama/Llama-2-13b-chat-hf | anyscale | 0.0000005 |
-| amazon.titan-text-lite-v1 | bedrock | 0.0000007 |
-| deepinfra/meta-llama/Llama-2-13b-chat-hf | deepinfra | 0.0000007 |
-| text-babbage-001 | text-completion-openai | 0.0000008 |
-| text-ada-001 | text-completion-openai | 0.0000008 |
-| babbage-002 | text-completion-openai | 0.0000008 |
-| openrouter/google/palm-2-chat-bison | openrouter | 0.000001 |
-| openrouter/google/palm-2-codechat-bison | openrouter | 0.000001 |
-| openrouter/meta-llama/codellama-34b-instruct | openrouter | 0.000001 |
-| deepinfra/codellama/CodeLlama-34b-Instruct-hf | deepinfra | 0.0000012 |
-| deepinfra/meta-llama/Llama-2-70b-chat-hf | deepinfra | 0.0000016499999999999999 |
-| deepinfra/jondurbin/airoboros-l2-70b-gpt4-1.4.1 | deepinfra | 0.0000016499999999999999 |
-| anyscale/meta-llama/Llama-2-70b-chat-hf | anyscale | 0.000002 |
-| anyscale/codellama/CodeLlama-34b-Instruct-hf | anyscale | 0.000002 |
-| gpt-3.5-turbo-1106 | openai | 0.000003 |
-| openrouter/meta-llama/llama-2-70b-chat | openrouter | 0.000003 |
-| amazon.titan-text-express-v1 | bedrock | 0.000003 |
-| gpt-3.5-turbo | openai | 0.0000035 |
-| gpt-3.5-turbo-0301 | openai | 0.0000035 |
-| gpt-3.5-turbo-0613 | openai | 0.0000035 |
-| gpt-3.5-turbo-instruct | text-completion-openai | 0.0000035 |
-| openrouter/openai/gpt-3.5-turbo | openrouter | 0.0000035 |
-| cohere.command-text-v14 | bedrock | 0.0000035 |
-| gpt-3.5-turbo-0613 | openai | 0.0000035 |
-| claude-instant-1 | anthropic | 0.00000714 |
-| claude-instant-1.2 | anthropic | 0.00000714 |
-| openrouter/anthropic/claude-instant-v1 | openrouter | 0.00000714 |
-| anthropic.claude-instant-v1 | bedrock | 0.00000714 |
-| openrouter/mancer/weaver | openrouter | 0.00001125 |
-| j2-mid | ai21 | 0.00002 |
-| ai21.j2-mid-v1 | bedrock | 0.000025 |
-| openrouter/jondurbin/airoboros-l2-70b-2.1 | openrouter | 0.00002775 |
-| command-nightly | cohere | 0.00003 |
-| command | cohere | 0.00003 |
-| command-light | cohere | 0.00003 |
-| command-medium-beta | cohere | 0.00003 |
-| command-xlarge-beta | cohere | 0.00003 |
-| command-r-plus| cohere | 0.000018 |
-| j2-ultra | ai21 | 0.00003 |
-| ai21.j2-ultra-v1 | bedrock | 0.0000376 |
-| gpt-4-1106-preview | openai | 0.00004 |
-| gpt-4-vision-preview | openai | 0.00004 |
-| claude-2 | anthropic | 0.0000437 |
-| openrouter/anthropic/claude-2 | openrouter | 0.0000437 |
-| anthropic.claude-v1 | bedrock | 0.0000437 |
-| anthropic.claude-v2 | bedrock | 0.0000437 |
-| gpt-4 | openai | 0.00009 |
-| gpt-4-0314 | openai | 0.00009 |
-| gpt-4-0613 | openai | 0.00009 |
-| openrouter/openai/gpt-4 | openrouter | 0.00009 |
-| gpt-4-32k | openai | 0.00018 |
-| gpt-4-32k-0314 | openai | 0.00018 |
-| gpt-4-32k-0613 | openai | 0.00018 |
-
-
-
-## Setup:
-```
-git clone https://github.com/BerriAI/litellm
-```
-cd to `benchmark` dir
-```
-cd litellm/cookbook/benchmark
-```
-
-### Install Dependencies
-```
-pip install litellm click tqdm tabulate termcolor
-```
-
-### Configuration
-In `benchmark/benchmark.py` select your LLMs, LLM API Key and questions
-
-Supported LLMs: https://docs.litellm.ai/docs/providers
-
-```python
-# Define the list of models to benchmark
-models = ['gpt-3.5-turbo', 'togethercomputer/llama-2-70b-chat', 'claude-2']
-
-# Enter LLM API keys
-os.environ['OPENAI_API_KEY'] = ""
-os.environ['ANTHROPIC_API_KEY'] = ""
-os.environ['TOGETHERAI_API_KEY'] = ""
-
-# List of questions to benchmark (replace with your questions)
-questions = [
- "When will BerriAI IPO?",
- "When will LiteLLM hit $100M ARR?"
-]
-
-```
-
-## Run LLM-Bench
-```
-python3 benchmark.py
-```
-
-## Expected Output
-```
-Running question: When will BerriAI IPO? for model: claude-2: 100%|āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā| 3/3 [00:13<00:00, 4.41s/it]
-
-Benchmark Results for 'When will BerriAI IPO?':
-+-----------------+----------------------------------------------------------------------------------+---------------------------+------------+
-| Model | Response | Response Time (seconds) | Cost ($) |
-+=================+==================================================================================+===========================+============+
-| gpt-3.5-turbo | As an AI language model, I cannot provide up-to-date information or predict | 1.55 seconds | $0.000122 |
-| | future events. It is best to consult a reliable financial source or contact | | |
-| | BerriAI directly for information regarding their IPO plans. | | |
-+-----------------+----------------------------------------------------------------------------------+---------------------------+------------+
-| togethercompute | I'm not able to provide information about future IPO plans or dates for BerriAI | 8.52 seconds | $0.000531 |
-| r/llama-2-70b-c | or any other company. IPO (Initial Public Offering) plans and timelines are | | |
-| hat | typically kept private by companies until they are ready to make a public | | |
-| | announcement. It's important to note that IPO plans can change and are subject | | |
-| | to various factors, such as market conditions, financial performance, and | | |
-| | regulatory approvals. Therefore, it's difficult to predict with certainty when | | |
-| | BerriAI or any other company will go public. If you're interested in staying | | |
-| | up-to-date with BerriAI's latest news and developments, you may want to follow | | |
-| | their official social media accounts, subscribe to their newsletter, or visit | | |
-| | their website periodically for updates. | | |
-+-----------------+----------------------------------------------------------------------------------+---------------------------+------------+
-| claude-2 | I do not have any information about when or if BerriAI will have an initial | 3.17 seconds | $0.002084 |
-| | public offering (IPO). As an AI assistant created by Anthropic to be helpful, | | |
-| | harmless, and honest, I do not have insider knowledge about Anthropic's business | | |
-| | plans or strategies. | | |
-+-----------------+----------------------------------------------------------------------------------+---------------------------+------------+
-```
-
-## Support
-**š¤ Schedule a 1-on-1 Session:** Book a [1-on-1 session](https://enterprise.litellm.ai/demo) with Krrish and Ishaan, the founders, to discuss any issues, provide feedback, or explore how we can improve LiteLLM for you.
diff --git a/cookbook/codellama-server/README.MD b/cookbook/codellama-server/README.MD
deleted file mode 100644
index 82a7e62f40a..00000000000
--- a/cookbook/codellama-server/README.MD
+++ /dev/null
@@ -1,153 +0,0 @@
-# CodeLlama Server: Streaming, Caching, Model Fallbacks (OpenAI + Anthropic), Prompt-tracking
-
-Works with: Anthropic, Huggingface, Cohere, TogetherAI, Azure, OpenAI, etc.
-
-[](https://pypi.org/project/litellm/)
-[](https://pypi.org/project/litellm/0.1.1/)
-
-
-[](https://railway.app/template/HuDPw-?referralCode=jch2ME)
-
-**LIVE DEMO** - https://litellm.ai/playground
-
-## What does CodeLlama Server do
-
-- Uses Together AI's CodeLlama to answer coding questions, with GPT-4 + Claude-2 as backups (you can easily switch this to any model from Huggingface, Replicate, Cohere, AI21, Azure, OpenAI, etc.)
-- Sets default system prompt for guardrails `system_prompt = "Only respond to questions about code. Say 'I don't know' to anything outside of that."`
-- Integrates with Promptlayer for model + prompt tracking
-- Example output
-
-
-
-- **Consistent Input/Output** Format
- - Call all models using the OpenAI format - `completion(model, messages)`
- - Text responses will always be available at `['choices'][0]['message']['content']`
- - Stream responses will always be available at `['choices'][0]['delta']['content']`
-- **Error Handling** Using Model Fallbacks (if `CodeLlama` fails, try `GPT-4`) with cooldowns, and retries
-- **Prompt Logging** - Log successful completions to promptlayer for testing + iterating on your prompts in production! (Learn more: https://litellm.readthedocs.io/en/latest/advanced/
-
- **Example: Logs sent to PromptLayer**
-
- 
-
-
-- **Token Usage & Spend** - Track Input + Completion tokens used + Spend/model - https://docs.litellm.ai/docs/token_usage
-- **Caching** - Provides in-memory cache + GPT-Cache integration for more advanced usage - https://docs.litellm.ai/docs/caching/gpt_cache
-
-- **Streaming & Async Support** - Return generators to stream text responses - TEST IT š https://litellm.ai/
-
-## API Endpoints
-
-### `/chat/completions` (POST)
-
-This endpoint is used to generate chat completions for 50+ support LLM API Models. Use llama2, GPT-4, Claude2 etc
-
-#### Input
-
-This API endpoint accepts all inputs in raw JSON and expects the following inputs
-
-- `prompt` (string, required): The user's coding related question
-- Additional Optional parameters: `temperature`, `functions`, `function_call`, `top_p`, `n`, `stream`. See the full list of supported inputs here: https://litellm.readthedocs.io/en/latest/input/
-
-#### Example JSON body
-
-For claude-2
-
-```json
-{
- "prompt": "write me a function to print hello world"
-}
-```
-
-### Making an API request to the Code-Gen Server
-
-```python
-import requests
-import json
-
-url = "localhost:4000/chat/completions"
-
-payload = json.dumps({
- "prompt": "write me a function to print hello world"
-})
-headers = {
- 'Content-Type': 'application/json'
-}
-
-response = requests.request("POST", url, headers=headers, data=payload)
-
-print(response.text)
-
-```
-
-### Output [Response Format]
-
-Responses from the server are given in the following format.
-All responses from the server are returned in the following format (for all LLM models). More info on output here: https://litellm.readthedocs.io/en/latest/output/
-
-```json
-{
- "choices": [
- {
- "finish_reason": "stop",
- "index": 0,
- "message": {
- "content": ".\n\n```\ndef print_hello_world():\n print(\"hello world\")\n",
- "role": "assistant"
- }
- }
- ],
- "created": 1693279694.6474009,
- "model": "togethercomputer/CodeLlama-34b-Instruct",
- "usage": {
- "completion_tokens": 14,
- "prompt_tokens": 28,
- "total_tokens": 42
- }
-}
-```
-
-## Installation & Usage
-
-### Running Locally
-
-1. Clone liteLLM repository to your local machine:
- ```
- git clone https://github.com/BerriAI/litellm-CodeLlama-server
- ```
-2. Install the required dependencies using pip
- ```
- pip install requirements.txt
- ```
-3. Set your LLM API keys
- ```
- os.environ['OPENAI_API_KEY]` = "YOUR_API_KEY"
- or
- set OPENAI_API_KEY in your .env file
- ```
-4. Run the server:
- ```
- python main.py
- ```
-
-## Deploying
-
-1. Quick Start: Deploy on Railway
-
- [](https://railway.app/template/HuDPw-?referralCode=jch2ME)
-
-2. `GCP`, `AWS`, `Azure`
- This project includes a `Dockerfile` allowing you to build and deploy a Docker Project on your providers
-
-# Support / Talk with founders
-
-- [Our calendar š](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
-- [Community Discord š](https://discord.gg/wuPM9dRgDw)
-- Our emails āļø ishaan@berri.ai / krrish@berri.ai
-
-## Roadmap
-
-- [ ] Implement user-based rate-limiting
-- [ ] Spending controls per project - expose key creation endpoint
-- [ ] Need to store a keys db -> mapping created keys to their alias (i.e. project name)
-- [ ] Easily add new models as backups / as the entry-point (add this to the available model list)
diff --git a/cookbook/codellama-server/imgs/code-output.png b/cookbook/codellama-server/imgs/code-output.png
deleted file mode 100644
index 67e298bd3be..00000000000
Binary files a/cookbook/codellama-server/imgs/code-output.png and /dev/null differ
diff --git a/cookbook/codellama-server/imgs/promptlayer_logging.png b/cookbook/codellama-server/imgs/promptlayer_logging.png
deleted file mode 100644
index 26b046ac448..00000000000
Binary files a/cookbook/codellama-server/imgs/promptlayer_logging.png and /dev/null differ
diff --git a/cookbook/codellama-server/main.py b/cookbook/codellama-server/main.py
deleted file mode 100644
index d05d6752300..00000000000
--- a/cookbook/codellama-server/main.py
+++ /dev/null
@@ -1,102 +0,0 @@
-import traceback
-from flask import Flask, request, Response
-from flask_cors import CORS
-import litellm
-from util import handle_error
-from litellm import completion
-import os
-import dotenv
-import time
-import json
-
-dotenv.load_dotenv()
-
-# TODO: set your keys in .env or here:
-# os.environ["OPENAI_API_KEY"] = "" # set your openai key here
-# os.environ["ANTHROPIC_API_KEY"] = "" # set your anthropic key here
-# os.environ["TOGETHER_AI_API_KEY"] = "" # set your together ai key here
-# see supported models / keys here: https://litellm.readthedocs.io/en/latest/supported/
-######### ENVIRONMENT VARIABLES ##########
-verbose = True
-
-# litellm.caching_with_models = True # CACHING: caching_with_models Keys in the cache are messages + model. - to learn more: https://docs.litellm.ai/docs/caching/
-######### PROMPT LOGGING ##########
-os.environ["PROMPTLAYER_API_KEY"] = (
- "" # set your promptlayer key here - https://promptlayer.com/
-)
-
-# set callbacks
-litellm.success_callback = ["promptlayer"]
-############ HELPER FUNCTIONS ###################################
-
-
-def print_verbose(print_statement):
- if verbose:
- print(print_statement)
-
-
-app = Flask(__name__)
-CORS(app)
-
-
-@app.route("/")
-def index():
- return "received!", 200
-
-
-def data_generator(response):
- for chunk in response:
- yield f"data: {json.dumps(chunk)}\n\n"
-
-
-@app.route("/chat/completions", methods=["POST"])
-def api_completion():
- data = request.json
- start_time = time.time()
- if data.get("stream") == "True":
- data["stream"] = True # convert to boolean
- try:
- if "prompt" not in data:
- raise ValueError("data needs to have prompt")
- data["model"] = (
- "togethercomputer/CodeLlama-34b-Instruct" # by default use Together AI's CodeLlama model - https://api.together.xyz/playground/chat?model=togethercomputer%2FCodeLlama-34b-Instruct
- )
- # COMPLETION CALL
- system_prompt = "Only respond to questions about code. Say 'I don't know' to anything outside of that."
- messages = [
- {"role": "system", "content": system_prompt},
- {"role": "user", "content": data.pop("prompt")},
- ]
- data["messages"] = messages
- print(f"data: {data}")
- response = completion(**data)
- ## LOG SUCCESS
- end_time = time.time()
- if (
- "stream" in data and data["stream"] == True
- ): # use generate_responses to stream responses
- return Response(data_generator(response), mimetype="text/event-stream")
- except Exception:
- # call handle_error function
- print_verbose(f"Got Error api_completion(): {traceback.format_exc()}")
- ## LOG FAILURE
- end_time = time.time()
- traceback_exception = traceback.format_exc()
- return handle_error(data=data)
- return response
-
-
-@app.route("/get_models", methods=["POST"])
-def get_models():
- try:
- return litellm.model_list
- except Exception as e:
- traceback.print_exc()
- response = {"error": str(e)}
- return response, 200
-
-
-if __name__ == "__main__":
- from waitress import serve
-
- serve(app, host="0.0.0.0", port=4000, threads=500)
diff --git a/cookbook/community-resources/get_hf_models.py b/cookbook/community-resources/get_hf_models.py
deleted file mode 100644
index 8c75a241227..00000000000
--- a/cookbook/community-resources/get_hf_models.py
+++ /dev/null
@@ -1,89 +0,0 @@
-import requests
-
-
-def get_next_url(response):
- """
- Function to get 'next' url from Link header
- :param response: response from requests
- :return: next url or None
- """
- if "link" not in response.headers:
- return None
- headers = response.headers
-
- next_url = headers["Link"]
- print(next_url)
- start_index = next_url.find("<")
- end_index = next_url.find(">")
-
- return next_url[1:end_index]
-
-
-def get_models(url):
- """
- Function to retrieve all models from paginated endpoint
- :param url: base url to make GET request
- :return: list of all models
- """
- models = []
- while url:
- response = requests.get(url)
- if response.status_code != 200:
- print(f"Failed to retrieve data. Status code: {response.status_code}")
- return models
- payload = response.json()
- url = get_next_url(response)
- models.extend(payload)
- return models
-
-
-def get_cleaned_models(models):
- """
- Function to clean retrieved models
- :param models: list of retrieved models
- :return: list of cleaned models
- """
- cleaned_models = []
- for model in models:
- cleaned_models.append(model["id"])
- return cleaned_models
-
-
-# Get text-generation models
-url = "https://huggingface.co/api/models?filter=text-generation-inference"
-text_generation_models = get_models(url)
-cleaned_text_generation_models = get_cleaned_models(text_generation_models)
-
-print(cleaned_text_generation_models)
-
-
-# Get conversational models
-url = "https://huggingface.co/api/models?filter=conversational"
-conversational_models = get_models(url)
-cleaned_conversational_models = get_cleaned_models(conversational_models)
-
-print(cleaned_conversational_models)
-
-
-def write_to_txt(cleaned_models, filename):
- """
- Function to write the contents of a list to a text file
- :param cleaned_models: list of cleaned models
- :param filename: name of the text file
- """
- with open(filename, "w") as f:
- for item in cleaned_models:
- f.write("%s\n" % item)
-
-
-# Write contents of cleaned_text_generation_models to text_generation_models.txt
-write_to_txt(
- cleaned_text_generation_models,
- "huggingface_llms_metadata/hf_text_generation_models.txt",
-)
-
-# Write contents of cleaned_conversational_models to conversational_models.txt
-write_to_txt(
- cleaned_conversational_models,
- "huggingface_llms_metadata/hf_conversational_models.txt",
-)
diff --git a/cookbook/community-resources/max_tokens.json b/cookbook/community-resources/max_tokens.json
deleted file mode 100644
index 289f8faf9e3..00000000000
--- a/cookbook/community-resources/max_tokens.json
+++ /dev/null
@@ -1,93 +0,0 @@
-{
- "gpt-3.5-turbo": {
- "max_tokens": 4000,
- "input_cost_per_token": 0.0000015,
- "output_cost_per_token": 0.000002
- },
- "gpt-3.5-turbo-0613": {
- "max_tokens": 4000,
- "input_cost_per_token": 0.0000015,
- "output_cost_per_token": 0.000002
- },
- "gpt-3.5-turbo-0301": {
- "max_tokens": 4000,
- "input_cost_per_token": 0.0000015,
- "output_cost_per_token": 0.000002
- },
- "gpt-3.5-turbo-16k": {
- "max_tokens": 16000,
- "input_cost_per_token": 0.000003,
- "output_cost_per_token": 0.000004
- },
- "gpt-3.5-turbo-16k-0613": {
- "max_tokens": 16000,
- "input_cost_per_token": 0.000003,
- "output_cost_per_token": 0.000004
- },
- "gpt-4": {
- "max_tokens": 8000,
- "input_cost_per_token": 0.000003,
- "output_cost_per_token": 0.00006
- },
- "gpt-4-0613": {
- "max_tokens": 8000,
- "input_cost_per_token": 0.000003,
- "output_cost_per_token": 0.00006
- },
- "gpt-4-32k": {
- "max_tokens": 8000,
- "input_cost_per_token": 0.00006,
- "output_cost_per_token": 0.00012
- },
- "claude-instant-1": {
- "max_tokens": 100000,
- "input_cost_per_token": 0.00000163,
- "output_cost_per_token": 0.00000551
- },
- "claude-2": {
- "max_tokens": 100000,
- "input_cost_per_token": 0.00001102,
- "output_cost_per_token": 0.00003268
- },
- "text-bison-001": {
- "max_tokens": 8192,
- "input_cost_per_token": 0.000004,
- "output_cost_per_token": 0.000004
- },
- "chat-bison-001": {
- "max_tokens": 4096,
- "input_cost_per_token": 0.000002,
- "output_cost_per_token": 0.000002
- },
- "command-nightly": {
- "max_tokens": 4096,
- "input_cost_per_token": 0.000015,
- "output_cost_per_token": 0.000015
- },
- "replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1": {
- "max_tokens": 4096,
- "input_cost_per_token": 0.00000608,
- "output_cost_per_token": 0.00000608
- },
- "together-ai-up-to-3b": {
- "input_cost_per_token": 0.0000001,
- "output_cost_per_token": 0.0000001
- },
- "together-ai-3.1b-7b": {
- "input_cost_per_token": 0.0000002,
- "output_cost_per_token": 0.0000002
- },
- "together-ai-7.1b-20b": {
- "max_tokens": 1000,
- "input_cost_per_token": 0.0000004,
- "output_cost_per_token": 0.0000004
- },
- "together-ai-20.1b-40b": {
- "input_cost_per_token": 0.000001,
- "output_cost_per_token": 0.000001
- },
- "together-ai-40.1b-70b": {
- "input_cost_per_token": 0.000003,
- "output_cost_per_token": 0.000003
- }
-}
diff --git a/cookbook/gollem_go_agent_framework/README.md b/cookbook/gollem_go_agent_framework/README.md
deleted file mode 100644
index 729f985d086..00000000000
--- a/cookbook/gollem_go_agent_framework/README.md
+++ /dev/null
@@ -1,119 +0,0 @@
-# Gollem Go Agent Framework with LiteLLM
-
-A working example showing how to use [gollem](https://github.com/fugue-labs/gollem), a production-grade Go agent framework, with LiteLLM as a proxy gateway. This lets Go developers access 100+ LLM providers through a single proxy while keeping compile-time type safety for tools and structured output.
-
-## Quick Start
-
-### 1. Start LiteLLM Proxy
-
-```bash
-# Simple start with a single model
-litellm --model gpt-4o
-
-# Or with the example config for multi-provider access
-litellm --config proxy_config.yaml
-```
-
-### 2. Run the examples
-
-```bash
-# Install Go dependencies
-go mod tidy
-
-# Basic agent
-go run ./basic
-
-# Agent with type-safe tools
-go run ./tools
-
-# Streaming responses
-go run ./streaming
-```
-
-## Configuration
-
-The included `proxy_config.yaml` sets up three providers through LiteLLM:
-
-```yaml
-model_list:
- - model_name: gpt-4o # OpenAI
- - model_name: claude-sonnet # Anthropic
- - model_name: gemini-pro # Google Vertex AI
-```
-
-Switch providers in Go by changing a single string ā no code changes needed:
-
-```go
-model := openai.NewLiteLLM("http://localhost:4000",
- openai.WithModel("gpt-4o"), // OpenAI
- // openai.WithModel("claude-sonnet"), // Anthropic
- // openai.WithModel("gemini-pro"), // Google
-)
-```
-
-## Examples
-
-### `basic/` ā Basic Agent
-
-Connects gollem to LiteLLM and runs a simple prompt. Demonstrates the `NewLiteLLM` constructor and basic agent creation.
-
-### `tools/` ā Type-Safe Tools
-
-Shows gollem's compile-time type-safe tool framework working through LiteLLM's tool-use passthrough. The tool parameters are Go structs with JSON tags ā the schema is generated automatically at compile time.
-
-### `streaming/` ā Streaming Responses
-
-Real-time token streaming using Go 1.23+ range-over-function iterators, proxied through LiteLLM's SSE passthrough.
-
-## How It Works
-
-Gollem's `openai.NewLiteLLM()` constructor creates an OpenAI-compatible provider pointed at your LiteLLM proxy. Since LiteLLM speaks the OpenAI API protocol, everything works out of the box:
-
-- **Chat completions** ā standard request/response
-- **Tool use** ā LiteLLM passes tool definitions and calls through transparently
-- **Streaming** ā Server-Sent Events proxied through LiteLLM
-- **Structured output** ā JSON schema response format works with supporting models
-
-```
-Go App (gollem) ā LiteLLM Proxy ā OpenAI / Anthropic / Google / ...
-```
-
-## Why Use This?
-
-- **Type-safe Go**: Compile-time type checking for tools, structured output, and agent configuration ā no runtime surprises
-- **Single proxy, many models**: Switch between OpenAI, Anthropic, Google, and 100+ other providers by changing a model name string
-- **Zero-dependency core**: gollem's core has no external dependencies ā just stdlib
-- **Single binary deployment**: `go build` produces one binary, no pip/venv/Docker needed
-- **Cost tracking & rate limiting**: LiteLLM handles cost tracking, rate limits, and fallbacks at the proxy layer
-
-## Environment Variables
-
-```bash
-# Required for providers you want to use (set in LiteLLM config or env)
-export OPENAI_API_KEY="sk-..."
-export ANTHROPIC_API_KEY="sk-ant-..."
-
-# Optional: point to a non-default LiteLLM proxy
-export LITELLM_PROXY_URL="http://localhost:4000"
-```
-
-## Troubleshooting
-
-**Connection errors?**
-- Make sure LiteLLM is running: `litellm --model gpt-4o`
-- Check the URL is correct (default: `http://localhost:4000`)
-
-**Model not found?**
-- Verify the model name matches what's configured in LiteLLM
-- Run `curl http://localhost:4000/models` to see available models
-
-**Tool calls not working?**
-- Ensure the underlying model supports tool use (GPT-4o, Claude, Gemini)
-- Check LiteLLM logs for any provider-specific errors
-
-## Learn More
-
-- [gollem GitHub](https://github.com/fugue-labs/gollem)
-- [gollem API Reference](https://pkg.go.dev/github.com/fugue-labs/gollem/core)
-- [LiteLLM Proxy Docs](https://docs.litellm.ai/docs/simple_proxy)
-- [LiteLLM Supported Models](https://docs.litellm.ai/docs/providers)
diff --git a/cookbook/gollem_go_agent_framework/basic/main.go b/cookbook/gollem_go_agent_framework/basic/main.go
deleted file mode 100644
index 838149a8ff9..00000000000
--- a/cookbook/gollem_go_agent_framework/basic/main.go
+++ /dev/null
@@ -1,41 +0,0 @@
-// Basic gollem agent connected to a LiteLLM proxy.
-//
-// Usage:
-//
-// litellm --model gpt-4o # start proxy in another terminal
-// go run ./basic
-package main
-
-import (
- "context"
- "fmt"
- "log"
- "os"
-
- "github.com/fugue-labs/gollem/core"
- "github.com/fugue-labs/gollem/provider/openai"
-)
-
-func main() {
- proxyURL := "http://localhost:4000"
- if u := os.Getenv("LITELLM_PROXY_URL"); u != "" {
- proxyURL = u
- }
-
- // Connect to LiteLLM proxy. NewLiteLLM creates an OpenAI-compatible
- // provider pointed at the given URL.
- model := openai.NewLiteLLM(proxyURL,
- openai.WithModel("gpt-4o"), // any model name configured in LiteLLM
- )
-
- // Create and run a simple agent.
- agent := core.NewAgent[string](model,
- core.WithSystemPrompt[string]("You are a helpful assistant. Be concise."),
- )
-
- result, err := agent.Run(context.Background(), "Explain quantum computing in two sentences.")
- if err != nil {
- log.Fatal(err)
- }
- fmt.Println(result.Output)
-}
diff --git a/cookbook/gollem_go_agent_framework/go.mod b/cookbook/gollem_go_agent_framework/go.mod
deleted file mode 100644
index a8dc9365d7f..00000000000
--- a/cookbook/gollem_go_agent_framework/go.mod
+++ /dev/null
@@ -1,5 +0,0 @@
-module github.com/BerriAI/litellm/cookbook/gollem_go_agent_framework
-
-go 1.26.3
-
-require github.com/fugue-labs/gollem v0.1.0
diff --git a/cookbook/gollem_go_agent_framework/go.sum b/cookbook/gollem_go_agent_framework/go.sum
deleted file mode 100644
index 1eb6c5ac9fc..00000000000
--- a/cookbook/gollem_go_agent_framework/go.sum
+++ /dev/null
@@ -1,2 +0,0 @@
-github.com/fugue-labs/gollem v0.1.0 h1:QexYnvkb44QZFEljgAePqMIGZjgsbk0Y5GJ2jYYgfa8=
-github.com/fugue-labs/gollem v0.1.0/go.mod h1:htW1YO81uysSKVOkYJtxhGCFrzm+36HBFxEWuECoHKQ=
diff --git a/cookbook/gollem_go_agent_framework/proxy_config.yaml b/cookbook/gollem_go_agent_framework/proxy_config.yaml
deleted file mode 100644
index 18265a002bc..00000000000
--- a/cookbook/gollem_go_agent_framework/proxy_config.yaml
+++ /dev/null
@@ -1,16 +0,0 @@
-model_list:
- - model_name: gpt-4o
- litellm_params:
- model: openai/gpt-4o
- api_key: os.environ/OPENAI_API_KEY
-
- - model_name: claude-sonnet
- litellm_params:
- model: anthropic/claude-sonnet-4-20250514
- api_key: os.environ/ANTHROPIC_API_KEY
-
- - model_name: gemini-pro
- litellm_params:
- model: vertex_ai/gemini-2.0-flash
- vertex_project: my-project
- vertex_location: us-central1
diff --git a/cookbook/gollem_go_agent_framework/streaming/main.go b/cookbook/gollem_go_agent_framework/streaming/main.go
deleted file mode 100644
index 42bc9bbe34a..00000000000
--- a/cookbook/gollem_go_agent_framework/streaming/main.go
+++ /dev/null
@@ -1,56 +0,0 @@
-// Streaming responses from gollem through LiteLLM.
-//
-// Uses Go 1.23+ range-over-function iterators for real-time token
-// streaming via LiteLLM's SSE passthrough.
-//
-// Usage:
-//
-// litellm --model gpt-4o
-// go run ./streaming
-package main
-
-import (
- "context"
- "fmt"
- "log"
- "os"
-
- "github.com/fugue-labs/gollem/core"
- "github.com/fugue-labs/gollem/provider/openai"
-)
-
-func main() {
- proxyURL := "http://localhost:4000"
- if u := os.Getenv("LITELLM_PROXY_URL"); u != "" {
- proxyURL = u
- }
-
- model := openai.NewLiteLLM(proxyURL,
- openai.WithModel("gpt-4o"),
- )
-
- agent := core.NewAgent[string](model)
-
- // RunStream returns a streaming result that yields tokens as they arrive.
- stream, err := agent.RunStream(context.Background(), "Write a haiku about distributed systems")
- if err != nil {
- log.Fatal(err)
- }
-
- // StreamText yields text chunks in real-time.
- // The boolean argument controls whether deltas (true) or accumulated
- // text (false) is returned.
- fmt.Print("Response: ")
- for text, err := range stream.StreamText(true) {
- if err != nil {
- log.Fatal(err)
- }
- fmt.Print(text)
- }
- fmt.Println()
-
- // After streaming completes, the final response is available.
- resp := stream.Response()
- fmt.Printf("\nTokens used: input=%d, output=%d\n",
- resp.Usage.InputTokens, resp.Usage.OutputTokens)
-}
diff --git a/cookbook/gollem_go_agent_framework/tools/main.go b/cookbook/gollem_go_agent_framework/tools/main.go
deleted file mode 100644
index ed41a95ffef..00000000000
--- a/cookbook/gollem_go_agent_framework/tools/main.go
+++ /dev/null
@@ -1,64 +0,0 @@
-// Gollem agent with type-safe tools through LiteLLM.
-//
-// The tool parameters are Go structs ā gollem generates the JSON schema
-// automatically at compile time. LiteLLM passes tool definitions through
-// transparently to the underlying provider.
-//
-// Usage:
-//
-// litellm --model gpt-4o
-// go run ./tools
-package main
-
-import (
- "context"
- "fmt"
- "log"
- "os"
-
- "github.com/fugue-labs/gollem/core"
- "github.com/fugue-labs/gollem/provider/openai"
-)
-
-// WeatherParams defines the tool's input schema via struct tags.
-// The JSON schema is generated at compile time ā no runtime reflection needed.
-type WeatherParams struct {
- City string `json:"city" description:"City name to get weather for"`
- Unit string `json:"unit,omitempty" description:"Temperature unit: celsius or fahrenheit"`
-}
-
-func main() {
- proxyURL := "http://localhost:4000"
- if u := os.Getenv("LITELLM_PROXY_URL"); u != "" {
- proxyURL = u
- }
-
- model := openai.NewLiteLLM(proxyURL,
- openai.WithModel("gpt-4o"),
- )
-
- // Define a type-safe tool. The function signature enforces correct types.
- weatherTool := core.FuncTool[WeatherParams](
- "get_weather",
- "Get current weather for a city",
- func(ctx context.Context, p WeatherParams) (string, error) {
- unit := p.Unit
- if unit == "" {
- unit = "fahrenheit"
- }
- // In production, call a real weather API here.
- return fmt.Sprintf("Weather in %s: 72Ā°F (22Ā°C), sunny", p.City), nil
- },
- )
-
- agent := core.NewAgent[string](model,
- core.WithTools[string](weatherTool),
- core.WithSystemPrompt[string]("You are a helpful weather assistant. Use the get_weather tool to answer weather questions."),
- )
-
- result, err := agent.Run(context.Background(), "What's the weather like in San Francisco and Tokyo?")
- if err != nil {
- log.Fatal(err)
- }
- fmt.Println(result.Output)
-}
diff --git a/cookbook/google_adk_litellm_tutorial.ipynb b/cookbook/google_adk_litellm_tutorial.ipynb
deleted file mode 100644
index 27914edbba8..00000000000
--- a/cookbook/google_adk_litellm_tutorial.ipynb
+++ /dev/null
@@ -1,412 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "7aa8875d",
- "metadata": {},
- "source": [
- "# Google ADK with LiteLLM\n",
- "\n",
- "Use Google ADK with LiteLLM Python SDK, LiteLLM Proxy.\n",
- "\n",
- "This tutorial shows you how to create intelligent agents using Agent Development Kit (ADK) with support for multiple Large Language Model (LLM) providers through LiteLLM."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a4d249c3",
- "metadata": {},
- "source": [
- "## Overview\n",
- "\n",
- "ADK (Agent Development Kit) allows you to build intelligent agents powered by LLMs. By integrating with LiteLLM, you can:\n",
- "\n",
- "- Use multiple LLM providers (OpenAI, Anthropic, Google, etc.)\n",
- "- Switch easily between models from different providers\n",
- "- Connect to a LiteLLM proxy for centralized model management"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a0bbb56b",
- "metadata": {},
- "source": [
- "## Prerequisites\n",
- "\n",
- "- Python environment setup\n",
- "- API keys for model providers (OpenAI, Anthropic, Google AI Studio)\n",
- "- Basic understanding of LLMs and agent concepts"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "7fee50a8",
- "metadata": {},
- "source": [
- "## Installation"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "44106a23",
- "metadata": {},
- "outputs": [],
- "source": [
- "# Install dependencies\n",
- "!pip install google-adk litellm"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2171740a",
- "metadata": {},
- "source": [
- "## 1. Setting Up Environment"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "6695807e",
- "metadata": {},
- "outputs": [],
- "source": [
- "# Setup environment and API keys\n",
- "import os\n",
- "import asyncio\n",
- "from google.adk.agents import Agent\n",
- "from google.adk.models.lite_llm import LiteLlm # For multi-model support\n",
- "from google.adk.sessions import InMemorySessionService\n",
- "from google.adk.runners import Runner\n",
- "from google.genai import types\n",
- "import litellm # Import for proxy configuration\n",
- "\n",
- "# Set your API keys\n",
- "os.environ['GOOGLE_API_KEY'] = 'your-google-api-key' # For Gemini models\n",
- "os.environ['OPENAI_API_KEY'] = 'your-openai-api-key' # For OpenAI models\n",
- "os.environ['ANTHROPIC_API_KEY'] = 'your-anthropic-api-key' # For Claude models\n",
- "\n",
- "# Define model constants for cleaner code\n",
- "MODEL_GEMINI_PRO = 'gemini-1.5-pro'\n",
- "MODEL_GPT_4O = 'openai/gpt-4o'\n",
- "MODEL_CLAUDE_SONNET = 'anthropic/claude-3-sonnet-20240229'"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d2b1ed59",
- "metadata": {},
- "source": [
- "## 2. Define a Simple Tool"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "04b3ef5b",
- "metadata": {},
- "outputs": [],
- "source": [
- "# Weather tool implementation\n",
- "def get_weather(city: str) -> dict:\n",
- " \"\"\"Retrieves the current weather report for a specified city.\"\"\"\n",
- " print(f'Tool: get_weather called for city: {city}')\n",
- "\n",
- " # Mock weather data\n",
- " mock_weather_db = {\n",
- " 'newyork': {\n",
- " 'status': 'success',\n",
- " 'report': 'The weather in New York is sunny with a temperature of 25Ā°C.'\n",
- " },\n",
- " 'london': {\n",
- " 'status': 'success',\n",
- " 'report': \"It's cloudy in London with a temperature of 15Ā°C.\"\n",
- " },\n",
- " 'tokyo': {\n",
- " 'status': 'success',\n",
- " 'report': 'Tokyo is experiencing light rain and a temperature of 18Ā°C.'\n",
- " },\n",
- " }\n",
- "\n",
- " city_normalized = city.lower().replace(' ', '')\n",
- "\n",
- " if city_normalized in mock_weather_db:\n",
- " return mock_weather_db[city_normalized]\n",
- " else:\n",
- " return {\n",
- " 'status': 'error',\n",
- " 'error_message': f\"Sorry, I don't have weather information for '{city}'.\"\n",
- " }"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "727b15c9",
- "metadata": {},
- "source": [
- "## 3. Helper Function for Agent Interaction"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f77449bf",
- "metadata": {},
- "outputs": [],
- "source": [
- "# Agent interaction helper function\n",
- "async def call_agent_async(query: str, runner, user_id, session_id):\n",
- " \"\"\"Sends a query to the agent and prints the final response.\"\"\"\n",
- " print(f'\\n>>> User Query: {query}')\n",
- "\n",
- " content = types.Content(role='user', parts=[types.Part(text=query)])\n",
- " final_response_text = 'Agent did not produce a final response.'\n",
- "\n",
- " async for event in runner.run_async(\n",
- " user_id=user_id,\n",
- " session_id=session_id,\n",
- " new_message=content\n",
- " ):\n",
- " if event.is_final_response():\n",
- " if event.content and event.content.parts:\n",
- " final_response_text = event.content.parts[0].text\n",
- " break\n",
- " print(f'<<< Agent Response: {final_response_text}')"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "0ac87987",
- "metadata": {},
- "source": [
- "## 4. Using Different Model Providers with ADK\n",
- "\n",
- "### 4.1 Using OpenAI Models"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "e167d557",
- "metadata": {},
- "outputs": [],
- "source": [
- "# OpenAI model implementation\n",
- "weather_agent_gpt = Agent(\n",
- " name='weather_agent_gpt',\n",
- " model=LiteLlm(model=MODEL_GPT_4O),\n",
- " description='Provides weather information using OpenAI\\'s GPT.',\n",
- " instruction=(\n",
- " 'You are a helpful weather assistant powered by GPT-4o. '\n",
- " \"Use the 'get_weather' tool for city weather requests. \"\n",
- " 'Present information clearly.'\n",
- " ),\n",
- " tools=[get_weather],\n",
- ")\n",
- "\n",
- "session_service_gpt = InMemorySessionService()\n",
- "session_gpt = session_service_gpt.create_session(\n",
- " app_name='weather_app', user_id='user_1', session_id='session_gpt'\n",
- ")\n",
- "\n",
- "runner_gpt = Runner(\n",
- " agent=weather_agent_gpt,\n",
- " app_name='weather_app',\n",
- " session_service=session_service_gpt,\n",
- ")\n",
- "\n",
- "async def test_gpt_agent():\n",
- " print('\\n--- Testing GPT Agent ---')\n",
- " await call_agent_async(\n",
- " \"What's the weather in London?\",\n",
- " runner=runner_gpt,\n",
- " user_id='user_1',\n",
- " session_id='session_gpt',\n",
- " )\n",
- "\n",
- "# To execute in a notebook cell:\n",
- "# await test_gpt_agent()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f9cb0613",
- "metadata": {},
- "source": [
- "### 4.2 Using Anthropic Models"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "1c653665",
- "metadata": {},
- "outputs": [],
- "source": [
- "# Anthropic model implementation\n",
- "weather_agent_claude = Agent(\n",
- " name='weather_agent_claude',\n",
- " model=LiteLlm(model=MODEL_CLAUDE_SONNET),\n",
- " description='Provides weather information using Anthropic\\'s Claude.',\n",
- " instruction=(\n",
- " 'You are a helpful weather assistant powered by Claude Sonnet. '\n",
- " \"Use the 'get_weather' tool for city weather requests. \"\n",
- " 'Present information clearly.'\n",
- " ),\n",
- " tools=[get_weather],\n",
- ")\n",
- "\n",
- "session_service_claude = InMemorySessionService()\n",
- "session_claude = session_service_claude.create_session(\n",
- " app_name='weather_app', user_id='user_1', session_id='session_claude'\n",
- ")\n",
- "\n",
- "runner_claude = Runner(\n",
- " agent=weather_agent_claude,\n",
- " app_name='weather_app',\n",
- " session_service=session_service_claude,\n",
- ")\n",
- "\n",
- "async def test_claude_agent():\n",
- " print('\\n--- Testing Claude Agent ---')\n",
- " await call_agent_async(\n",
- " \"What's the weather in Tokyo?\",\n",
- " runner=runner_claude,\n",
- " user_id='user_1',\n",
- " session_id='session_claude',\n",
- " )\n",
- "\n",
- "# To execute in a notebook cell:\n",
- "# await test_claude_agent()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "bf9d863b",
- "metadata": {},
- "source": [
- "### 4.3 Using Google's Gemini Models"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "83f49d0a",
- "metadata": {},
- "outputs": [],
- "source": [
- "# Gemini model implementation\n",
- "weather_agent_gemini = Agent(\n",
- " name='weather_agent_gemini',\n",
- " model=MODEL_GEMINI_PRO,\n",
- " description='Provides weather information using Google\\'s Gemini.',\n",
- " instruction=(\n",
- " 'You are a helpful weather assistant powered by Gemini Pro. '\n",
- " \"Use the 'get_weather' tool for city weather requests. \"\n",
- " 'Present information clearly.'\n",
- " ),\n",
- " tools=[get_weather],\n",
- ")\n",
- "\n",
- "session_service_gemini = InMemorySessionService()\n",
- "session_gemini = session_service_gemini.create_session(\n",
- " app_name='weather_app', user_id='user_1', session_id='session_gemini'\n",
- ")\n",
- "\n",
- "runner_gemini = Runner(\n",
- " agent=weather_agent_gemini,\n",
- " app_name='weather_app',\n",
- " session_service=session_service_gemini,\n",
- ")\n",
- "\n",
- "async def test_gemini_agent():\n",
- " print('\\n--- Testing Gemini Agent ---')\n",
- " await call_agent_async(\n",
- " \"What's the weather in New York?\",\n",
- " runner=runner_gemini,\n",
- " user_id='user_1',\n",
- " session_id='session_gemini',\n",
- " )\n",
- "\n",
- "# To execute in a notebook cell:\n",
- "# await test_gemini_agent()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "93bc5fd0",
- "metadata": {},
- "source": [
- "## 5. Using LiteLLM Proxy with ADK"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b4275151",
- "metadata": {},
- "source": [
- "| Variable | Description |\n",
- "|----------|-------------|\n",
- "| `LITELLM_PROXY_API_KEY` | The API key for the LiteLLM proxy |\n",
- "| `LITELLM_PROXY_API_BASE` | The base URL for the LiteLLM proxy |\n",
- "| `USE_LITELLM_PROXY` or `litellm.use_litellm_proxy` | When set to True, your request will be sent to LiteLLM proxy. |"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "256530a6",
- "metadata": {},
- "outputs": [],
- "source": [
- "# LiteLLM proxy integration\n",
- "os.environ['LITELLM_PROXY_API_KEY'] = 'your-litellm-proxy-api-key'\n",
- "os.environ['LITELLM_PROXY_API_BASE'] = 'your-litellm-proxy-url' # e.g., 'http://localhost:4000'\n",
- "litellm.use_litellm_proxy = True\n",
- "\n",
- "weather_agent_proxy_env = Agent(\n",
- " name='weather_agent_proxy_env',\n",
- " model=LiteLlm(model='gpt-4o'),\n",
- " description='Provides weather information using a model from LiteLLM proxy.',\n",
- " instruction=(\n",
- " 'You are a helpful weather assistant. '\n",
- " \"Use the 'get_weather' tool for city weather requests. \"\n",
- " 'Present information clearly.'\n",
- " ),\n",
- " tools=[get_weather],\n",
- ")\n",
- "\n",
- "session_service_proxy_env = InMemorySessionService()\n",
- "session_proxy_env = session_service_proxy_env.create_session(\n",
- " app_name='weather_app', user_id='user_1', session_id='session_proxy_env'\n",
- ")\n",
- "\n",
- "runner_proxy_env = Runner(\n",
- " agent=weather_agent_proxy_env,\n",
- " app_name='weather_app',\n",
- " session_service=session_service_proxy_env,\n",
- ")\n",
- "\n",
- "async def test_proxy_env_agent():\n",
- " print('\\n--- Testing Proxy-enabled Agent (Environment Variables) ---')\n",
- " await call_agent_async(\n",
- " \"What's the weather in London?\",\n",
- " runner=runner_proxy_env,\n",
- " user_id='user_1',\n",
- " session_id='session_proxy_env',\n",
- " )\n",
- "\n",
- "# To execute in a notebook cell:\n",
- "# await test_proxy_env_agent()"
- ]
- }
- ],
- "metadata": {
- "language_info": {
- "name": "python"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
diff --git a/cookbook/gpt_realtime_translate.py b/cookbook/gpt_realtime_translate.py
deleted file mode 100644
index 827c65cdbfe..00000000000
--- a/cookbook/gpt_realtime_translate.py
+++ /dev/null
@@ -1,245 +0,0 @@
-#!/usr/bin/env python3
-
-import argparse
-import asyncio
-import base64
-import json
-import os
-import sys
-import wave
-from collections.abc import Iterator, Sequence
-from dataclasses import dataclass
-from pathlib import Path
-from urllib.parse import urlencode, urlsplit, urlunsplit
-
-import websockets
-from websockets.asyncio.client import ClientConnection
-
-SAMPLE_RATE = 24_000
-CHANNELS = 1
-SAMPLE_WIDTH = 2
-CHUNK_DURATION_SECONDS = 0.1
-CHUNK_BYTES = int(SAMPLE_RATE * CHANNELS * SAMPLE_WIDTH * CHUNK_DURATION_SECONDS)
-RESPONSE_TIMEOUT_SECONDS = 30.0
-AUDIO_EVENT_TYPES = frozenset(
- {
- "session.output_audio.delta",
- "response.audio.delta",
- "response.output_audio.delta",
- }
-)
-TRANSCRIPT_EVENT_TYPES = frozenset(
- {
- "session.output_transcript.delta",
- "response.text.delta",
- "response.output_audio_transcript.delta",
- }
-)
-
-
-@dataclass(frozen=True, slots=True)
-class Settings:
- input_wav: Path
- output_wav: Path
- base_url: str
- model: str
- target_language: str
- trailing_silence_seconds: float
- api_key: str
-
-
-def write_stdout(message: str = "", *, end: str = "\n", flush: bool = False) -> None:
- sys.stdout.write(f"{message}{end}")
- if flush:
- sys.stdout.flush()
-
-
-def write_stderr(message: str) -> None:
- sys.stderr.write(f"{message}\n")
-
-
-def parse_args(argv: Sequence[str] | None = None) -> Settings | str:
- parser = argparse.ArgumentParser(
- description="Stream a 24 kHz PCM16 WAV through gpt-realtime-translate and save the translated audio",
- )
- parser.add_argument("input_wav", type=Path)
- parser.add_argument("--output", type=Path, default=Path("translated.wav"))
- parser.add_argument("--base-url", default=os.getenv("LITELLM_BASE_URL", "http://localhost:4000"))
- parser.add_argument("--model", default=os.getenv("REALTIME_TRANSLATE_MODEL", "gpt-realtime-translate"))
- parser.add_argument("--target-language", default="fr")
- parser.add_argument("--trailing-silence", type=float, default=1.5)
- parsed = parser.parse_args(argv)
- api_key = os.getenv("LITELLM_API_KEY") or os.getenv("OPENAI_API_KEY")
- if not api_key:
- return "Set LITELLM_API_KEY or OPENAI_API_KEY before running the script"
- if parsed.trailing_silence < 0:
- return "--trailing-silence must be zero or greater"
- return Settings(
- input_wav=parsed.input_wav,
- output_wav=parsed.output,
- base_url=parsed.base_url,
- model=parsed.model,
- target_language=parsed.target_language,
- trailing_silence_seconds=parsed.trailing_silence,
- api_key=api_key,
- )
-
-
-def translation_url(base_url: str, model: str) -> str | None:
- parsed = urlsplit(base_url.rstrip("/"))
- scheme = {"http": "ws", "https": "wss", "ws": "ws", "wss": "wss"}.get(parsed.scheme)
- if not scheme or not parsed.netloc:
- return None
- base_path = parsed.path.rstrip("/")
- realtime_path = (
- f"{base_path}/realtime/translations" if base_path.endswith("/v1") else f"{base_path}/v1/realtime/translations"
- )
- return urlunsplit((scheme, parsed.netloc, realtime_path, urlencode({"model": model}), ""))
-
-
-def read_pcm16_wav(path: Path) -> bytes | str:
- try:
- with wave.open(str(path), "rb") as source:
- actual_format = (
- source.getnchannels(),
- source.getsampwidth(),
- source.getframerate(),
- source.getcomptype(),
- )
- expected_format = (CHANNELS, SAMPLE_WIDTH, SAMPLE_RATE, "NONE")
- if actual_format != expected_format:
- return (
- f"{path} must be mono, 16-bit PCM, 24 kHz WAV; received "
- f"channels={actual_format[0]}, sample_width={actual_format[1]}, "
- f"sample_rate={actual_format[2]}, compression={actual_format[3]}"
- )
- return source.readframes(source.getnframes())
- except (OSError, EOFError, wave.Error) as exc:
- return f"Unable to read {path}: {exc}"
-
-
-def audio_chunks(audio: bytes) -> Iterator[bytes]:
- return (audio[offset : offset + CHUNK_BYTES] for offset in range(0, len(audio), CHUNK_BYTES))
-
-
-def audio_message(audio: bytes) -> str:
- return json.dumps(
- {
- "type": "session.input_audio_buffer.append",
- "audio": base64.b64encode(audio).decode("ascii"),
- }
- )
-
-
-async def configure_session(connection: ClientConnection, target_language: str) -> str | None:
- await connection.send(
- json.dumps(
- {
- "type": "session.update",
- "session": {"audio": {"output": {"language": target_language}}},
- }
- )
- )
- while True:
- raw_event = await asyncio.wait_for(connection.recv(), timeout=20)
- event = json.loads(raw_event)
- event_type = event.get("type")
- if event_type == "session.created":
- write_stdout(f"Session: {event.get('session', {}).get('id', 'created')}")
- if event_type == "session.updated":
- return None
- if event_type == "error":
- return f"Realtime API error: {json.dumps(event.get('error', event), ensure_ascii=False)}"
-
-
-async def send_audio(
- connection: ClientConnection, pcm: bytes, trailing_silence_seconds: float, finished: asyncio.Event
-) -> None:
- silence = bytes(round(trailing_silence_seconds * SAMPLE_RATE * CHANNELS * SAMPLE_WIDTH))
- try:
- for chunk in audio_chunks(pcm + silence):
- await connection.send(audio_message(chunk))
- await asyncio.sleep(len(chunk) / (SAMPLE_RATE * CHANNELS * SAMPLE_WIDTH))
- await connection.send(json.dumps({"type": "session.close"}))
- finally:
- finished.set()
-
-
-async def receive_translation(
- connection: ClientConnection, output_path: Path, sender_finished: asyncio.Event
-) -> str | None:
- audio_received = asyncio.Event()
- try:
- with wave.open(str(output_path), "wb") as output:
- output.setnchannels(CHANNELS)
- output.setsampwidth(SAMPLE_WIDTH)
- output.setframerate(SAMPLE_RATE)
- write_stdout("Translation: ", end="", flush=True)
- while True:
- try:
- raw_event = await asyncio.wait_for(connection.recv(), timeout=RESPONSE_TIMEOUT_SECONDS)
- except TimeoutError:
- if not sender_finished.is_set():
- continue
- return (
- f"The translation stream went quiet for {RESPONSE_TIMEOUT_SECONDS:g} s without closing the session"
- )
- event = json.loads(raw_event)
- event_type = event.get("type")
- if event_type in AUDIO_EVENT_TYPES:
- output.writeframes(base64.b64decode(event.get("delta", ""), validate=True))
- audio_received.set()
- elif event_type in TRANSCRIPT_EVENT_TYPES:
- write_stdout(event.get("delta", event.get("text", "")), end="", flush=True)
- elif event_type == "error":
- return f"Realtime API error: {json.dumps(event.get('error', event), ensure_ascii=False)}"
- elif event_type == "session.closed":
- write_stdout()
- return None if audio_received.is_set() else "The translation stream ended without translated audio"
- except (OSError, wave.Error) as exc:
- return f"Unable to write {output_path}: {exc}"
-
-
-async def translate(settings: Settings, pcm: bytes) -> str | None:
- url = translation_url(settings.base_url, settings.model)
- if not url:
- return f"Invalid --base-url: {settings.base_url}"
- sender_finished = asyncio.Event()
- try:
- async with websockets.connect(
- url,
- additional_headers={"Authorization": f"Bearer {settings.api_key}"},
- proxy=None,
- open_timeout=20,
- close_timeout=5,
- ) as connection:
- configuration_error = await configure_session(connection, settings.target_language)
- if configuration_error:
- return configuration_error
- async with asyncio.TaskGroup() as tasks:
- receiver = tasks.create_task(receive_translation(connection, settings.output_wav, sender_finished))
- tasks.create_task(send_audio(connection, pcm, settings.trailing_silence_seconds, sender_finished))
- return receiver.result()
- except Exception as exc:
- return f"Translation failed: {type(exc).__name__}: {exc}"
-
-
-def main(argv: Sequence[str] | None = None) -> int:
- settings = parse_args(argv)
- if isinstance(settings, str):
- write_stderr(settings)
- return 2
- pcm = read_pcm16_wav(settings.input_wav)
- if isinstance(pcm, str):
- write_stderr(pcm)
- return 2
- error = asyncio.run(translate(settings, pcm))
- if error:
- write_stderr(error)
- return 1
- write_stdout(f"Translated audio: {settings.output_wav}")
- return 0
-
-
-if __name__ == "__main__":
- raise SystemExit(main())
diff --git a/cookbook/liteLLM_A121_Jurrasic_example.ipynb b/cookbook/liteLLM_A121_Jurrasic_example.ipynb
deleted file mode 100644
index f975b97e911..00000000000
--- a/cookbook/liteLLM_A121_Jurrasic_example.ipynb
+++ /dev/null
@@ -1,251 +0,0 @@
-{
- "nbformat": 4,
- "nbformat_minor": 0,
- "metadata": {
- "colab": {
- "provenance": []
- },
- "kernelspec": {
- "name": "python3",
- "display_name": "Python 3"
- },
- "language_info": {
- "name": "python"
- }
- },
- "cells": [
- {
- "cell_type": "markdown",
- "source": [
- "# LiteLLM A121 Tutorial\n",
- "\n",
- "This walks through using A121 Jurassic models\n",
- "* j2-light\n",
- "* j2-mid\n",
- "* j2-ultra"
- ],
- "metadata": {
- "id": "LeFYo8iqcn5g"
- }
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "GslPQFmaZsp-"
- },
- "outputs": [],
- "source": [
- "!pip install litellm"
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "from litellm import completion\n",
- "import os"
- ],
- "metadata": {
- "id": "P3cKiqURZx7P"
- },
- "execution_count": 2,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "source": [
- "## Set A121 Keys\n",
- "You can get a free key from https://studio.ai21.com/account/api-key"
- ],
- "metadata": {
- "id": "tmTvA1_GaNU4"
- }
- },
- {
- "cell_type": "code",
- "source": [
- "os.environ[\"AI21_API_KEY\"] = \"\""
- ],
- "metadata": {
- "id": "_xX8LmxAZ2vp"
- },
- "execution_count": 5,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "source": [
- "# A121 Supported Models:\n",
- "https://studio.ai21.com/foundation-models"
- ],
- "metadata": {
- "id": "Fx5ZfJTLbF0A"
- }
- },
- {
- "cell_type": "markdown",
- "source": [
- "## J2-light Call"
- ],
- "metadata": {
- "id": "H0tl-0Z3bDaL"
- }
- },
- {
- "cell_type": "code",
- "source": [
- "messages = [{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}]\n",
- "response = completion(model=\"j2-light\", messages=messages)\n",
- "response"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "DZnApsJUZ_I2",
- "outputId": "b5707cbe-f67c-47f7-bac5-a7b8af1ba815"
- },
- "execution_count": 6,
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/plain": [
- "
-
-- **Token Usage & Spend** - Track Input + Completion tokens used + Spend/model
-- **Caching** - Implementation of Semantic Caching
-- **Streaming & Async Support** - Return generators to stream text responses
-
-## API Endpoints
-
-### `/chat/completions` (POST)
-
-This endpoint is used to generate chat completions for 50+ support LLM API Models. Use llama2, GPT-4, Claude2 etc
-
-#### Input
-
-This API endpoint accepts all inputs in raw JSON and expects the following inputs
-
-- `model` (string, required): ID of the model to use for chat completions. See all supported models [here]: (https://litellm.readthedocs.io/en/latest/supported/):
- eg `gpt-3.5-turbo`, `gpt-4`, `claude-2`, `command-nightly`, `stabilityai/stablecode-completion-alpha-3b-4k`
-- `messages` (array, required): A list of messages representing the conversation context. Each message should have a `role` (system, user, assistant, or function), `content` (message text), and `name` (for function role).
-- Additional Optional parameters: `temperature`, `functions`, `function_call`, `top_p`, `n`, `stream`. See the full list of supported inputs here: https://litellm.readthedocs.io/en/latest/input/
-
-#### Example JSON body
-
-For claude-2
-
-```json
-{
- "model": "claude-2",
- "messages": [
- {
- "content": "Hello, whats the weather in San Francisco??",
- "role": "user"
- }
- ]
-}
-```
-
-### Making an API request to the Proxy Server
-
-```python
-import requests
-import json
-
-# TODO: use your URL
-url = "http://localhost:5000/chat/completions"
-
-payload = json.dumps({
- "model": "gpt-3.5-turbo",
- "messages": [
- {
- "content": "Hello, whats the weather in San Francisco??",
- "role": "user"
- }
- ]
-})
-headers = {
- 'Content-Type': 'application/json'
-}
-response = requests.request("POST", url, headers=headers, data=payload)
-print(response.text)
-
-```
-
-### Output [Response Format]
-
-Responses from the server are given in the following format.
-All responses from the server are returned in the following format (for all LLM models). More info on output here: https://litellm.readthedocs.io/en/latest/output/
-
-```json
-{
- "choices": [
- {
- "finish_reason": "stop",
- "index": 0,
- "message": {
- "content": "I'm sorry, but I don't have the capability to provide real-time weather information. However, you can easily check the weather in San Francisco by searching online or using a weather app on your phone.",
- "role": "assistant"
- }
- }
- ],
- "created": 1691790381,
- "id": "chatcmpl-7mUFZlOEgdohHRDx2UpYPRTejirzb",
- "model": "gpt-3.5-turbo-0613",
- "object": "chat.completion",
- "usage": {
- "completion_tokens": 41,
- "prompt_tokens": 16,
- "total_tokens": 57
- }
-}
-```
-
-## Installation & Usage
-
-### Running Locally
-
-1. Clone liteLLM repository to your local machine:
- ```
- git clone https://github.com/BerriAI/liteLLM-proxy
- ```
-2. Install the required dependencies using pip
- ```
- pip install -r requirements.txt
- ```
-3. Set your LLM API keys
- ```
- os.environ['OPENAI_API_KEY]` = "YOUR_API_KEY"
- or
- set OPENAI_API_KEY in your .env file
- ```
-4. Run the server:
- ```
- python main.py
- ```
-
-## Deploying
-
-1. Quick Start: Deploy on Railway
-
- [](https://railway.app/template/DYqQAW?referralCode=t3ukrU)
-
-2. `GCP`, `AWS`, `Azure`
- This project includes a `Dockerfile` allowing you to build and deploy a Docker Project on your providers
-
-# Support / Talk with founders
-
-- [Our calendar š](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
-- [Community Discord š](https://discord.gg/wuPM9dRgDw)
-- Our emails āļø ishaan@berri.ai / krrish@berri.ai
-
-## Roadmap
-
-- [ ] Support hosted db (e.g. Supabase)
-- [ ] Easily send data to places like posthog and sentry.
-- [ ] Add a hot-cache for project spend logs - enables fast checks for user + project limitings
-- [ ] Implement user-based rate-limiting
-- [ ] Spending controls per project - expose key creation endpoint
-- [ ] Need to store a keys db -> mapping created keys to their alias (i.e. project name)
-- [ ] Easily add new models as backups / as the entry-point (add this to the available model list)
diff --git a/cookbook/litellm_proxy_server/secret_manager/custom_secret_manager_config.yaml b/cookbook/litellm_proxy_server/secret_manager/custom_secret_manager_config.yaml
deleted file mode 100644
index 3598a9b1b65..00000000000
--- a/cookbook/litellm_proxy_server/secret_manager/custom_secret_manager_config.yaml
+++ /dev/null
@@ -1,20 +0,0 @@
-general_settings:
- master_key: os.environ/LITELLM_MASTER_KEY
- key_management_system: "custom"
- key_management_settings:
- custom_secret_manager: my_secret_manager.InMemorySecretManager
- store_virtual_keys: true
- prefix_for_stored_virtual_keys: "litellm/"
- access_mode: "read_and_write"
-
-model_list:
- - model_name: gpt-4
- litellm_params:
- model: openai/gpt-4
- api_key: os.environ/OPENAI_API_KEY # Read from custom secret manager
-
- - model_name: claude-3-5-sonnet
- litellm_params:
- model: anthropic/claude-3-5-sonnet-20241022
- api_key: os.environ/ANTHROPIC_API_KEY # Read from custom secret manager
-
diff --git a/cookbook/litellm_proxy_server/secret_manager/my_secret_manager.py b/cookbook/litellm_proxy_server/secret_manager/my_secret_manager.py
deleted file mode 100644
index 65c7f754b41..00000000000
--- a/cookbook/litellm_proxy_server/secret_manager/my_secret_manager.py
+++ /dev/null
@@ -1,80 +0,0 @@
-"""
-Example custom secret manager for LiteLLM Proxy.
-
-This is a simple in-memory secret manager for testing purposes.
-In production, replace this with your actual secret management system.
-"""
-
-from typing import Optional, Union
-
-import httpx
-
-from litellm.integrations.custom_secret_manager import CustomSecretManager
-
-
-class InMemorySecretManager(CustomSecretManager):
- def __init__(self):
- super().__init__(secret_manager_name="in_memory_secrets")
- # Store your secrets in memory
- print("INITIALIZING CUSTOM SECRET MANAGER IN MEMORY")
- self.secrets = {}
- print("CUSTOM SECRET MANAGER IN MEMORY INITIALIZED")
-
- async def async_read_secret(
- self,
- secret_name: str,
- optional_params: Optional[dict] = None,
- timeout: Optional[Union[float, httpx.Timeout]] = None,
- ) -> Optional[str]:
- """Read secret asynchronously"""
- print("READING SECRET ASYNCHRONOUSLY")
- print("SECRET NAME: %s", secret_name)
- print("SECRET: %s", self.secrets.get(secret_name))
- return self.secrets.get(secret_name)
-
- def sync_read_secret(
- self,
- secret_name: str,
- optional_params: Optional[dict] = None,
- timeout: Optional[Union[float, httpx.Timeout]] = None,
- ) -> Optional[str]:
- """Read secret synchronously"""
- from litellm._logging import verbose_proxy_logger
-
- verbose_proxy_logger.info(
- f"CUSTOM SECRET MANAGER: LOOKING FOR SECRET: {secret_name}"
- )
- value = self.secrets.get(secret_name)
- verbose_proxy_logger.info(f"CUSTOM SECRET MANAGER: READ SECRET: {value}")
- return value
-
- async def async_write_secret(
- self,
- secret_name: str,
- secret_value: str,
- description: Optional[str] = None,
- optional_params: Optional[dict] = None,
- timeout: Optional[Union[float, httpx.Timeout]] = None,
- tags: Optional[Union[dict, list]] = None,
- ) -> dict:
- """Write a secret to the in-memory store"""
- self.secrets[secret_name] = secret_value
- print("ALL SECRETS=%s", self.secrets)
- return {
- "status": "success",
- "secret_name": secret_name,
- "description": description,
- }
-
- async def async_delete_secret(
- self,
- secret_name: str,
- recovery_window_in_days: Optional[int] = 7,
- optional_params: Optional[dict] = None,
- timeout: Optional[Union[float, httpx.Timeout]] = None,
- ) -> dict:
- """Delete a secret from the in-memory store"""
- if secret_name in self.secrets:
- del self.secrets[secret_name]
- return {"status": "deleted", "secret_name": secret_name}
- return {"status": "not_found", "secret_name": secret_name}
diff --git a/cookbook/litellm_router/error_log.txt b/cookbook/litellm_router/error_log.txt
deleted file mode 100644
index 983b47cbbba..00000000000
--- a/cookbook/litellm_router/error_log.txt
+++ /dev/null
@@ -1,1004 +0,0 @@
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: Expecting value: line 1 column 1 (char 0)
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Exception: 'Response' object has no attribute 'get'
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Exception: 'Response' object has no attribute 'get'
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Exception: 'Response' object has no attribute 'get'
-
diff --git a/cookbook/litellm_router/load_test_proxy.py b/cookbook/litellm_router/load_test_proxy.py
deleted file mode 100644
index 9ae6e764d91..00000000000
--- a/cookbook/litellm_router/load_test_proxy.py
+++ /dev/null
@@ -1,148 +0,0 @@
-import sys
-import os
-from dotenv import load_dotenv
-
-load_dotenv()
-
-sys.path.insert(
- 0, os.path.abspath("../..")
-) # Adds the parent directory to the system path
-
-from litellm import Router
-import litellm
-
-litellm.set_verbose = False
-os.environ.pop("AZURE_AD_TOKEN")
-
-model_list = [
- { # list of model deployments
- "model_name": "gpt-3.5-turbo", # model alias
- "litellm_params": { # params for litellm completion/embedding call
- "model": "azure/chatgpt-v-2", # actual model name
- "api_key": os.getenv("AZURE_API_KEY"),
- "api_version": os.getenv("AZURE_API_VERSION"),
- "api_base": os.getenv("AZURE_API_BASE"),
- },
- },
- {
- "model_name": "gpt-3.5-turbo",
- "litellm_params": { # params for litellm completion/embedding call
- "model": "azure/chatgpt-functioncalling",
- "api_key": os.getenv("AZURE_API_KEY"),
- "api_version": os.getenv("AZURE_API_VERSION"),
- "api_base": os.getenv("AZURE_API_BASE"),
- },
- },
- {
- "model_name": "gpt-3.5-turbo",
- "litellm_params": { # params for litellm completion/embedding call
- "model": "gpt-3.5-turbo",
- "api_key": os.getenv("OPENAI_API_KEY"),
- },
- },
-]
-router = Router(model_list=model_list)
-
-
-file_paths = [
- "test_questions/question1.txt",
- "test_questions/question2.txt",
- "test_questions/question3.txt",
-]
-questions = []
-
-for file_path in file_paths:
- try:
- print(file_path)
- with open(file_path, "r") as file:
- content = file.read()
- questions.append(content)
- except FileNotFoundError as e:
- print(f"File not found: {e}")
- except Exception as e:
- print(f"An error occurred: {e}")
-
-# for q in questions:
-# print(q)
-
-
-# make X concurrent calls to litellm.completion(model=gpt-35-turbo, messages=[]), pick a random question in questions array.
-# Allow me to tune X concurrent calls.. Log question, output/exception, response time somewhere
-# show me a summary of requests made, success full calls, failed calls. For failed calls show me the exceptions
-
-import concurrent.futures
-import random
-import time
-
-
-# Function to make concurrent calls to OpenAI API
-def make_openai_completion(question):
- try:
- start_time = time.time()
- import openai
-
- client = openai.OpenAI(
- api_key=os.environ["OPENAI_API_KEY"], base_url="http://0.0.0.0:8000"
- ) # base_url="http://0.0.0.0:8000",
- response = client.chat.completions.create(
- model="gpt-3.5-turbo",
- messages=[
- {
- "role": "system",
- "content": f"You are a helpful assistant. Answer this question{question}",
- }
- ],
- )
- print(response)
- end_time = time.time()
-
- # Log the request details
- with open("request_log.txt", "a") as log_file:
- log_file.write(
- f"Question: {question[:100]}\nResponse ID:{response.id} Content:{response.choices[0].message.content[:10]}\nTime: {end_time - start_time:.2f} seconds\n\n"
- )
-
- return response
- except Exception as e:
- # Log exceptions for failed calls
- with open("error_log.txt", "a") as error_log_file:
- error_log_file.write(f"Question: {question[:100]}\nException: {str(e)}\n\n")
- return None
-
-
-# Number of concurrent calls (you can adjust this)
-concurrent_calls = 100
-
-# List to store the futures of concurrent calls
-futures = []
-
-# Make concurrent calls
-with concurrent.futures.ThreadPoolExecutor(max_workers=concurrent_calls) as executor:
- for _ in range(concurrent_calls):
- random_question = random.choice(questions)
- futures.append(executor.submit(make_openai_completion, random_question))
-
-# Wait for all futures to complete
-concurrent.futures.wait(futures)
-
-# Summarize the results
-successful_calls = 0
-failed_calls = 0
-
-for future in futures:
- if future.result() is not None:
- successful_calls += 1
- else:
- failed_calls += 1
-
-print("Load test Summary:")
-print(f"Total Requests: {concurrent_calls}")
-print(f"Successful Calls: {successful_calls}")
-print(f"Failed Calls: {failed_calls}")
-
-# Display content of the logs
-with open("request_log.txt", "r") as log_file:
- print("\nRequest Log:\n", log_file.read())
-
-with open("error_log.txt", "r") as error_log_file:
- print("\nError Log:\n", error_log_file.read())
diff --git a/cookbook/litellm_router/load_test_queuing.py b/cookbook/litellm_router/load_test_queuing.py
deleted file mode 100644
index 7d4d44b2528..00000000000
--- a/cookbook/litellm_router/load_test_queuing.py
+++ /dev/null
@@ -1,164 +0,0 @@
-import sys
-import os
-from dotenv import load_dotenv
-
-load_dotenv()
-
-sys.path.insert(
- 0, os.path.abspath("../..")
-) # Adds the parent directory to the system path
-
-from litellm import Router
-import litellm
-
-litellm.set_verbose = False
-# os.environ.pop("AZURE_AD_TOKEN")
-
-model_list = [
- { # list of model deployments
- "model_name": "gpt-3.5-turbo", # model alias
- "litellm_params": { # params for litellm completion/embedding call
- "model": "azure/chatgpt-v-2", # actual model name
- "api_key": os.getenv("AZURE_API_KEY"),
- "api_version": os.getenv("AZURE_API_VERSION"),
- "api_base": os.getenv("AZURE_API_BASE"),
- },
- },
- {
- "model_name": "gpt-3.5-turbo",
- "litellm_params": { # params for litellm completion/embedding call
- "model": "azure/chatgpt-functioncalling",
- "api_key": os.getenv("AZURE_API_KEY"),
- "api_version": os.getenv("AZURE_API_VERSION"),
- "api_base": os.getenv("AZURE_API_BASE"),
- },
- },
- {
- "model_name": "gpt-3.5-turbo",
- "litellm_params": { # params for litellm completion/embedding call
- "model": "gpt-3.5-turbo",
- "api_key": os.getenv("OPENAI_API_KEY"),
- },
- },
-]
-router = Router(model_list=model_list)
-
-
-file_paths = [
- "test_questions/question1.txt",
- "test_questions/question2.txt",
- "test_questions/question3.txt",
-]
-questions = []
-
-for file_path in file_paths:
- try:
- print(file_path)
- with open(file_path, "r") as file:
- content = file.read()
- questions.append(content)
- except FileNotFoundError as e:
- print(f"File not found: {e}")
- except Exception as e:
- print(f"An error occurred: {e}")
-
-# for q in questions:
-# print(q)
-
-
-# make X concurrent calls to litellm.completion(model=gpt-35-turbo, messages=[]), pick a random question in questions array.
-# Allow me to tune X concurrent calls.. Log question, output/exception, response time somewhere
-# show me a summary of requests made, success full calls, failed calls. For failed calls show me the exceptions
-
-import concurrent.futures
-import random
-import time
-
-
-# Function to make concurrent calls to OpenAI API
-def make_openai_completion(question):
- try:
- start_time = time.time()
- import requests
-
- data = {
- "model": "gpt-3.5-turbo",
- "messages": [
- {
- "role": "system",
- "content": f"You are a helpful assistant. Answer this question{question}",
- },
- ],
- }
- response = requests.post("http://0.0.0.0:8000/queue/request", json=data)
- response = response.json()
- end_time = time.time()
- # Log the request details
- with open("request_log.txt", "a") as log_file:
- log_file.write(
- f"Question: {question[:100]}\nResponse ID: {response.get('id', 'N/A')} Url: {response.get('url', 'N/A')}\nTime: {end_time - start_time:.2f} seconds\n\n"
- )
-
- # polling the url
- while True:
- try:
- url = response["url"]
- polling_url = f"http://0.0.0.0:8000{url}"
- polling_response = requests.get(polling_url)
- polling_response = polling_response.json()
- print("\n RESPONSE FROM POLLING JoB", polling_response)
- status = polling_response["status"]
- if status == "finished":
- llm_response = polling_response["result"]
- with open("response_log.txt", "a") as log_file:
- log_file.write(
- f"Response ID: {llm_response.get('id', 'NA')}\nLLM Response: {llm_response}\nTime: {end_time - start_time:.2f} seconds\n\n"
- )
-
- break
- print(
- f"POLLING JOB{polling_url}\nSTATUS: {status}, \n Response {polling_response}"
- )
- time.sleep(0.5)
- except Exception as e:
- print("got exception in polling", e)
- break
-
- return response
- except Exception as e:
- # Log exceptions for failed calls
- with open("error_log.txt", "a") as error_log_file:
- error_log_file.write(f"Question: {question[:100]}\nException: {str(e)}\n\n")
- return None
-
-
-# Number of concurrent calls (you can adjust this)
-concurrent_calls = 10
-
-# List to store the futures of concurrent calls
-futures = []
-
-# Make concurrent calls
-with concurrent.futures.ThreadPoolExecutor(max_workers=concurrent_calls) as executor:
- for _ in range(concurrent_calls):
- random_question = random.choice(questions)
- futures.append(executor.submit(make_openai_completion, random_question))
-
-# Wait for all futures to complete
-concurrent.futures.wait(futures)
-
-# Summarize the results
-successful_calls = 0
-failed_calls = 0
-
-for future in futures:
- if future.done():
- if future.result() is not None:
- successful_calls += 1
- else:
- failed_calls += 1
-
-print("Load test Summary:")
-print(f"Total Requests: {concurrent_calls}")
-print(f"Successful Calls: {successful_calls}")
-print(f"Failed Calls: {failed_calls}")
diff --git a/cookbook/litellm_router/load_test_router.py b/cookbook/litellm_router/load_test_router.py
deleted file mode 100644
index 92533b6c929..00000000000
--- a/cookbook/litellm_router/load_test_router.py
+++ /dev/null
@@ -1,143 +0,0 @@
-import sys
-import os
-from dotenv import load_dotenv
-
-load_dotenv()
-
-sys.path.insert(
- 0, os.path.abspath("../..")
-) # Adds the parent directory to the system path
-
-from litellm import Router
-import litellm
-
-litellm.set_verbose = False
-os.environ.pop("AZURE_AD_TOKEN")
-
-model_list = [
- { # list of model deployments
- "model_name": "gpt-3.5-turbo", # model alias
- "litellm_params": { # params for litellm completion/embedding call
- "model": "azure/chatgpt-v-2", # actual model name
- "api_key": os.getenv("AZURE_API_KEY"),
- "api_version": os.getenv("AZURE_API_VERSION"),
- "api_base": os.getenv("AZURE_API_BASE"),
- },
- },
- {
- "model_name": "gpt-3.5-turbo",
- "litellm_params": { # params for litellm completion/embedding call
- "model": "azure/chatgpt-functioncalling",
- "api_key": os.getenv("AZURE_API_KEY"),
- "api_version": os.getenv("AZURE_API_VERSION"),
- "api_base": os.getenv("AZURE_API_BASE"),
- },
- },
- {
- "model_name": "gpt-3.5-turbo",
- "litellm_params": { # params for litellm completion/embedding call
- "model": "gpt-3.5-turbo",
- "api_key": os.getenv("OPENAI_API_KEY"),
- },
- },
-]
-router = Router(model_list=model_list)
-
-
-file_paths = [
- "test_questions/question1.txt",
- "test_questions/question2.txt",
- "test_questions/question3.txt",
-]
-questions = []
-
-for file_path in file_paths:
- try:
- print(file_path)
- with open(file_path, "r") as file:
- content = file.read()
- questions.append(content)
- except FileNotFoundError as e:
- print(f"File not found: {e}")
- except Exception as e:
- print(f"An error occurred: {e}")
-
-# for q in questions:
-# print(q)
-
-
-# make X concurrent calls to litellm.completion(model=gpt-35-turbo, messages=[]), pick a random question in questions array.
-# Allow me to tune X concurrent calls.. Log question, output/exception, response time somewhere
-# show me a summary of requests made, success full calls, failed calls. For failed calls show me the exceptions
-
-import concurrent.futures
-import random
-import time
-
-
-# Function to make concurrent calls to OpenAI API
-def make_openai_completion(question):
- try:
- start_time = time.time()
- response = router.completion(
- model="gpt-3.5-turbo",
- messages=[
- {
- "role": "system",
- "content": f"You are a helpful assistant. Answer this question{question}",
- }
- ],
- )
- print(response)
- end_time = time.time()
-
- # Log the request details
- with open("request_log.txt", "a") as log_file:
- log_file.write(
- f"Question: {question[:100]}\nResponse: {response.choices[0].message.content}\nTime: {end_time - start_time:.2f} seconds\n\n"
- )
-
- return response
- except Exception as e:
- # Log exceptions for failed calls
- with open("error_log.txt", "a") as error_log_file:
- error_log_file.write(f"Question: {question[:100]}\nException: {str(e)}\n\n")
- return None
-
-
-# Number of concurrent calls (you can adjust this)
-concurrent_calls = 150
-
-# List to store the futures of concurrent calls
-futures = []
-
-# Make concurrent calls
-with concurrent.futures.ThreadPoolExecutor(max_workers=concurrent_calls) as executor:
- for _ in range(concurrent_calls):
- random_question = random.choice(questions)
- futures.append(executor.submit(make_openai_completion, random_question))
-
-# Wait for all futures to complete
-concurrent.futures.wait(futures)
-
-# Summarize the results
-successful_calls = 0
-failed_calls = 0
-
-for future in futures:
- if future.result() is not None:
- successful_calls += 1
- else:
- failed_calls += 1
-
-print("Load test Summary:")
-print(f"Total Requests: {concurrent_calls}")
-print(f"Successful Calls: {successful_calls}")
-print(f"Failed Calls: {failed_calls}")
-
-# Display content of the logs
-with open("request_log.txt", "r") as log_file:
- print("\nRequest Log:\n", log_file.read())
-
-with open("error_log.txt", "r") as error_log_file:
- print("\nError Log:\n", error_log_file.read())
diff --git a/cookbook/litellm_router/request_log.txt b/cookbook/litellm_router/request_log.txt
deleted file mode 100644
index 821d87ab56a..00000000000
--- a/cookbook/litellm_router/request_log.txt
+++ /dev/null
@@ -1,48 +0,0 @@
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Response ID: 71a47cd4-92d9-4091-9429-8d22af6b56bf Url: /queue/response/71a47cd4-92d9-4091-9429-8d22af6b56bf
-Time: 0.77 seconds
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Response ID: a0855c20-59ba-4eed-85c1-e0719eebdeab Url: /queue/response/a0855c20-59ba-4eed-85c1-e0719eebdeab
-Time: 1.46 seconds
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Response ID: b131cdcd-0693-495b-ad41-b0cf2afc4833 Url: /queue/response/b131cdcd-0693-495b-ad41-b0cf2afc4833
-Time: 2.13 seconds
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Response ID: a58e5185-90e7-4832-9f28-e5a5ac167a40 Url: /queue/response/a58e5185-90e7-4832-9f28-e5a5ac167a40
-Time: 2.83 seconds
-
-Question: Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format.
-Response ID: 52dbbd49-eedb-4c11-8382-3ca7deb1af35 Url: /queue/response/52dbbd49-eedb-4c11-8382-3ca7deb1af35
-Time: 3.50 seconds
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Response ID: eedda05f-61e1-4081-b49d-27f9449bcf69 Url: /queue/response/eedda05f-61e1-4081-b49d-27f9449bcf69
-Time: 4.20 seconds
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Response ID: 8a484722-66ec-4193-b19b-2dfc4265cfd2 Url: /queue/response/8a484722-66ec-4193-b19b-2dfc4265cfd2
-Time: 4.89 seconds
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Response ID: ae1e2b71-d711-456d-8df0-13ce0709eb04 Url: /queue/response/ae1e2b71-d711-456d-8df0-13ce0709eb04
-Time: 5.60 seconds
-
-Question: What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 10
-Response ID: cfabd174-838e-4252-b82b-648923573db8 Url: /queue/response/cfabd174-838e-4252-b82b-648923573db8
-Time: 6.29 seconds
-
-Question: Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the Ope
-Response ID: 02d5b7d6-5443-41e9-94e4-90d8b00d49fb Url: /queue/response/02d5b7d6-5443-41e9-94e4-90d8b00d49fb
-Time: 7.01 seconds
-
diff --git a/cookbook/litellm_router/response_log.txt b/cookbook/litellm_router/response_log.txt
deleted file mode 100644
index e69de29bb2d..00000000000
diff --git a/cookbook/litellm_router/test_questions/question1.txt b/cookbook/litellm_router/test_questions/question1.txt
deleted file mode 100644
index d633a8ea22e..00000000000
--- a/cookbook/litellm_router/test_questions/question1.txt
+++ /dev/null
@@ -1,43 +0,0 @@
-Given this context, what is litellm? LiteLLM about: About
-Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs). LiteLLM manages
-
-Translating inputs to the provider's completion and embedding endpoints
-Guarantees consistent output, text responses will always be available at ['choices'][0]['message']['content']
-Exception mapping - common exceptions across providers are mapped to the OpenAI exception types.
-10/05/2023: LiteLLM is adopting Semantic Versioning for all commits. Learn more
-10/16/2023: Self-hosted OpenAI-proxy server Learn more
-
-Usage (Docs)
-Important
-LiteLLM v1.0.0 is being launched to require openai>=1.0.0. Track this here
-
-Open In Colab
-pip install litellm
-from litellm import completion
-import os
-
-## set ENV variables
-os.environ["OPENAI_API_KEY"] = "your-openai-key"
-os.environ["COHERE_API_KEY"] = "your-cohere-key"
-
-messages = [{ "content": "Hello, how are you?","role": "user"}]
-
-# openai call
-response = completion(model="gpt-3.5-turbo", messages=messages)
-
-# cohere call
-response = completion(model="command-nightly", messages=messages)
-print(response)
-Streaming (Docs)
-liteLLM supports streaming the model response back, pass stream=True to get a streaming iterator in response.
-Streaming is supported for all models (Bedrock, Huggingface, TogetherAI, Azure, OpenAI, etc.)
-
-from litellm import completion
-response = completion(model="gpt-3.5-turbo", messages=messages, stream=True)
-for chunk in response:
- print(chunk['choices'][0]['delta'])
-
-# claude 2
-result = completion('claude-2', messages, stream=True)
-for chunk in result:
- print(chunk['choices'][0]['delta'])
\ No newline at end of file
diff --git a/cookbook/litellm_router/test_questions/question2.txt b/cookbook/litellm_router/test_questions/question2.txt
deleted file mode 100644
index 78188d06668..00000000000
--- a/cookbook/litellm_router/test_questions/question2.txt
+++ /dev/null
@@ -1,65 +0,0 @@
-Does litellm support ooobagooba llms? how can i call oobagooba llms. Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs). LiteLLM manages
-
-Translating inputs to the provider's completion and embedding endpoints
-Guarantees consistent output, text responses will always be available at ['choices'][0]['message']['content']
-Exception mapping - common exceptions across providers are mapped to the OpenAI exception types.
-10/05/2023: LiteLLM is adopting Semantic Versioning for all commits. Learn more
-10/16/2023: Self-hosted OpenAI-proxy server Learn more
-
-Usage (Docs)
-Important
-LiteLLM v1.0.0 is being launched to require openai>=1.0.0. Track this here
-
-Open In Colab
-pip install litellm
-from litellm import completion
-import os
-
-## set ENV variables
-os.environ["OPENAI_API_KEY"] = "your-openai-key"
-os.environ["COHERE_API_KEY"] = "your-cohere-key"
-
-messages = [{ "content": "Hello, how are you?","role": "user"}]
-
-# openai call
-response = completion(model="gpt-3.5-turbo", messages=messages)
-
-# cohere call
-response = completion(model="command-nightly", messages=messages)
-print(response)
-Streaming (Docs)
-liteLLM supports streaming the model response back, pass stream=True to get a streaming iterator in response.
-Streaming is supported for all models (Bedrock, Huggingface, TogetherAI, Azure, OpenAI, etc.)
-
-from litellm import completion
-response = completion(model="gpt-3.5-turbo", messages=messages, stream=True)
-for chunk in response:
- print(chunk['choices'][0]['delta'])
-
-# claude 2
-result = completion('claude-2', messages, stream=True)
-for chunk in result:
- print(chunk['choices'][0]['delta']) Supported LiteLLM providers Supported Provider (Docs)
-Provider Completion Streaming Async Completion Async Streaming
-openai ā
ā
ā
ā
-azure ā
ā
ā
ā
-aws - sagemaker ā
ā
ā
ā
-aws - bedrock ā
ā
ā
ā
-cohere ā
ā
ā
ā
-anthropic ā
ā
ā
ā
-huggingface ā
ā
ā
ā
-replicate ā
ā
ā
ā
-together_ai ā
ā
ā
ā
-openrouter ā
ā
ā
ā
-google - vertex_ai ā
ā
ā
ā
-google - palm ā
ā
ā
ā
-ai21 ā
ā
ā
ā
-baseten ā
ā
ā
ā
-vllm ā
ā
ā
ā
-nlp_cloud ā
ā
ā
ā
-aleph alpha ā
ā
ā
ā
-petals ā
ā
ā
ā
-ollama ā
ā
ā
ā
-deepinfra ā
ā
ā
ā
-perplexity-ai ā
ā
ā
ā
-anyscale ā
ā
ā
ā
\ No newline at end of file
diff --git a/cookbook/litellm_router/test_questions/question3.txt b/cookbook/litellm_router/test_questions/question3.txt
deleted file mode 100644
index d6006f9c73c..00000000000
--- a/cookbook/litellm_router/test_questions/question3.txt
+++ /dev/null
@@ -1,50 +0,0 @@
-What endpoints does the litellm proxy have š„ LiteLLM Proxy Server
-LiteLLM Server manages:
-
-Calling 100+ LLMs Huggingface/Bedrock/TogetherAI/etc. in the OpenAI ChatCompletions & Completions format
-Set custom prompt templates + model-specific configs (temperature, max_tokens, etc.)
-Quick Start
-View all the supported args for the Proxy CLI here
-
-$ litellm --model huggingface/bigcode/starcoder
-
-#INFO: Proxy running on http://0.0.0.0:8000
-
-Test
-In a new shell, run, this will make an openai.ChatCompletion request
-
-litellm --test
-
-This will now automatically route any requests for gpt-3.5-turbo to bigcode starcoder, hosted on huggingface inference endpoints.
-
-Replace openai base
-import openai
-
-openai.api_base = "http://0.0.0.0:8000"
-
-print(openai.chat.completions.create(model="test", messages=[{"role":"user", "content":"Hey!"}]))
-
-Supported LLMs
-Bedrock
-Huggingface (TGI)
-Anthropic
-VLLM
-OpenAI Compatible Server
-TogetherAI
-Replicate
-Petals
-Palm
-Azure OpenAI
-AI21
-Cohere
-$ export AWS_ACCESS_KEY_ID=""
-$ export AWS_REGION_NAME="" # e.g. us-west-2
-$ export AWS_SECRET_ACCESS_KEY=""
-
-$ litellm --model bedrock/anthropic.claude-v2
-
-Server Endpoints
-POST /chat/completions - chat completions endpoint to call 100+ LLMs
-POST /completions - completions endpoint
-POST /embeddings - embedding endpoint for Azure, OpenAI, Huggingface endpoints
-GET /models - available models on server
\ No newline at end of file
diff --git a/cookbook/litellm_router_load_test/memory_usage/router_endpoint.py b/cookbook/litellm_router_load_test/memory_usage/router_endpoint.py
deleted file mode 100644
index 1dc2d914857..00000000000
--- a/cookbook/litellm_router_load_test/memory_usage/router_endpoint.py
+++ /dev/null
@@ -1,65 +0,0 @@
-from fastapi import FastAPI
-import uvicorn
-from memory_profiler import profile
-import os
-import litellm
-from litellm import Router
-from dotenv import load_dotenv
-from litellm._uuid import uuid
-
-load_dotenv()
-
-model_list = [
- {
- "model_name": "gpt-3.5-turbo",
- "litellm_params": {
- "model": "azure/chatgpt-v-2",
- "api_key": os.getenv("AZURE_API_KEY"),
- "api_version": os.getenv("AZURE_API_VERSION"),
- "api_base": os.getenv("AZURE_API_BASE"),
- },
- "tpm": 240000,
- "rpm": 1800,
- },
- {
- "model_name": "text-embedding-ada-002",
- "litellm_params": {
- "model": "azure/azure-embedding-model",
- "api_key": os.getenv("AZURE_API_KEY"),
- "api_base": os.getenv("AZURE_API_BASE"),
- },
- "tpm": 100000,
- "rpm": 10000,
- },
-]
-
-litellm.set_verbose = True
-litellm.cache = litellm.Cache(
- type="s3", s3_bucket_name="litellm-my-test-bucket-2", s3_region_name="us-east-1"
-)
-router = Router(model_list=model_list, set_verbose=True)
-
-app = FastAPI()
-
-
-@app.get("/")
-async def read_root():
- return {"message": "Welcome to the FastAPI endpoint!"}
-
-
-@profile
-@app.post("/router_acompletion")
-async def router_acompletion():
- question = f"This is a test: {uuid.uuid4()}" * 100
- resp = await router.aembedding(model="text-embedding-ada-002", input=question)
- print("embedding-resp", resp)
-
- response = await router.acompletion(
- model="gpt-3.5-turbo", messages=[{"role": "user", "content": question}]
- )
- print("completion-resp", response)
- return response
-
-
-if __name__ == "__main__":
- uvicorn.run(app, host="0.0.0.0", port=8000)
diff --git a/cookbook/litellm_router_load_test/memory_usage/router_memory_usage copy.py b/cookbook/litellm_router_load_test/memory_usage/router_memory_usage copy.py
deleted file mode 100644
index 76d5d3913f5..00000000000
--- a/cookbook/litellm_router_load_test/memory_usage/router_memory_usage copy.py
+++ /dev/null
@@ -1,91 +0,0 @@
-#### What this tests ####
-
-from memory_profiler import profile
-import sys
-import os
-import time
-import asyncio
-
-sys.path.insert(
- 0, os.path.abspath("../..")
-) # Adds the parent directory to the system path
-import litellm
-from litellm import Router
-from dotenv import load_dotenv
-from litellm._uuid import uuid
-
-load_dotenv()
-
-
-model_list = [
- {
- "model_name": "gpt-3.5-turbo", # openai model name
- "litellm_params": { # params for litellm completion/embedding call
- "model": "azure/chatgpt-v-2",
- "api_key": os.getenv("AZURE_API_KEY"),
- "api_version": os.getenv("AZURE_API_VERSION"),
- "api_base": os.getenv("AZURE_API_BASE"),
- },
- "tpm": 240000,
- "rpm": 1800,
- },
- {
- "model_name": "text-embedding-ada-002",
- "litellm_params": {
- "model": "azure/azure-embedding-model",
- "api_key": os.environ["AZURE_API_KEY"],
- "api_base": os.environ["AZURE_API_BASE"],
- },
- "tpm": 100000,
- "rpm": 10000,
- },
-]
-litellm.set_verbose = True
-litellm.cache = litellm.Cache(
- type="s3", s3_bucket_name="litellm-my-test-bucket-2", s3_region_name="us-east-1"
-)
-router = Router(
- model_list=model_list,
- set_verbose=True,
-) # type: ignore
-
-
-@profile
-async def router_acompletion():
- # embedding call
- question = f"This is a test: {uuid.uuid4()}" * 100
- resp = await router.aembedding(model="text-embedding-ada-002", input=question)
- print("embedding-resp", resp)
-
- response = await router.acompletion(
- model="gpt-3.5-turbo", messages=[{"role": "user", "content": question}]
- )
- print("completion-resp", response)
- return response
-
-
-async def main():
- for i in range(1):
- start = time.time()
- n = 50 # Number of concurrent tasks
- tasks = [router_acompletion() for _ in range(n)]
-
- chat_completions = await asyncio.gather(*tasks)
-
- successful_completions = [c for c in chat_completions if c is not None]
-
- # Write errors to error_log.txt
- with open("error_log.txt", "a") as error_log:
- for completion in chat_completions:
- if isinstance(completion, str):
- error_log.write(completion + "\n")
-
- print(n, time.time() - start, len(successful_completions))
- time.sleep(10)
-
-
-if __name__ == "__main__":
- # Blank out contents of error_log.txt
- open("error_log.txt", "w").close()
-
- asyncio.run(main())
diff --git a/cookbook/litellm_router_load_test/memory_usage/router_memory_usage.py b/cookbook/litellm_router_load_test/memory_usage/router_memory_usage.py
deleted file mode 100644
index 76d5d3913f5..00000000000
--- a/cookbook/litellm_router_load_test/memory_usage/router_memory_usage.py
+++ /dev/null
@@ -1,91 +0,0 @@
-#### What this tests ####
-
-from memory_profiler import profile
-import sys
-import os
-import time
-import asyncio
-
-sys.path.insert(
- 0, os.path.abspath("../..")
-) # Adds the parent directory to the system path
-import litellm
-from litellm import Router
-from dotenv import load_dotenv
-from litellm._uuid import uuid
-
-load_dotenv()
-
-
-model_list = [
- {
- "model_name": "gpt-3.5-turbo", # openai model name
- "litellm_params": { # params for litellm completion/embedding call
- "model": "azure/chatgpt-v-2",
- "api_key": os.getenv("AZURE_API_KEY"),
- "api_version": os.getenv("AZURE_API_VERSION"),
- "api_base": os.getenv("AZURE_API_BASE"),
- },
- "tpm": 240000,
- "rpm": 1800,
- },
- {
- "model_name": "text-embedding-ada-002",
- "litellm_params": {
- "model": "azure/azure-embedding-model",
- "api_key": os.environ["AZURE_API_KEY"],
- "api_base": os.environ["AZURE_API_BASE"],
- },
- "tpm": 100000,
- "rpm": 10000,
- },
-]
-litellm.set_verbose = True
-litellm.cache = litellm.Cache(
- type="s3", s3_bucket_name="litellm-my-test-bucket-2", s3_region_name="us-east-1"
-)
-router = Router(
- model_list=model_list,
- set_verbose=True,
-) # type: ignore
-
-
-@profile
-async def router_acompletion():
- # embedding call
- question = f"This is a test: {uuid.uuid4()}" * 100
- resp = await router.aembedding(model="text-embedding-ada-002", input=question)
- print("embedding-resp", resp)
-
- response = await router.acompletion(
- model="gpt-3.5-turbo", messages=[{"role": "user", "content": question}]
- )
- print("completion-resp", response)
- return response
-
-
-async def main():
- for i in range(1):
- start = time.time()
- n = 50 # Number of concurrent tasks
- tasks = [router_acompletion() for _ in range(n)]
-
- chat_completions = await asyncio.gather(*tasks)
-
- successful_completions = [c for c in chat_completions if c is not None]
-
- # Write errors to error_log.txt
- with open("error_log.txt", "a") as error_log:
- for completion in chat_completions:
- if isinstance(completion, str):
- error_log.write(completion + "\n")
-
- print(n, time.time() - start, len(successful_completions))
- time.sleep(10)
-
-
-if __name__ == "__main__":
- # Blank out contents of error_log.txt
- open("error_log.txt", "w").close()
-
- asyncio.run(main())
diff --git a/cookbook/litellm_router_load_test/memory_usage/send_request.py b/cookbook/litellm_router_load_test/memory_usage/send_request.py
deleted file mode 100644
index 6a3473e230f..00000000000
--- a/cookbook/litellm_router_load_test/memory_usage/send_request.py
+++ /dev/null
@@ -1,28 +0,0 @@
-import requests
-from concurrent.futures import ThreadPoolExecutor
-
-# Replace the URL with your actual endpoint
-url = "http://localhost:8000/router_acompletion"
-
-
-def make_request(session):
- headers = {"Content-Type": "application/json"}
- data = {} # Replace with your JSON payload if needed
-
- response = session.post(url, headers=headers, json=data)
- print(f"Status code: {response.status_code}")
-
-
-# Number of concurrent requests
-num_requests = 20
-
-# Create a session to reuse the underlying TCP connection
-with requests.Session() as session:
- # Use ThreadPoolExecutor for concurrent requests
- with ThreadPoolExecutor(max_workers=num_requests) as executor:
- # Use list comprehension to submit tasks
- futures = [executor.submit(make_request, session) for _ in range(num_requests)]
-
- # Wait for all futures to complete
- for future in futures:
- future.result()
diff --git a/cookbook/litellm_router_load_test/test_loadtest_openai_client.py b/cookbook/litellm_router_load_test/test_loadtest_openai_client.py
deleted file mode 100644
index 8c50825be12..00000000000
--- a/cookbook/litellm_router_load_test/test_loadtest_openai_client.py
+++ /dev/null
@@ -1,73 +0,0 @@
-import sys
-import os
-from dotenv import load_dotenv
-
-load_dotenv()
-sys.path.insert(
- 0, os.path.abspath("../..")
-) # Adds the parent directory to the system path
-import asyncio
-from litellm import Timeout
-import time
-import openai
-
-### Test just calling AsyncAzureOpenAI
-
-openai_client = openai.AsyncAzureOpenAI(
- azure_endpoint=os.getenv("AZURE_API_BASE"),
- api_key=os.getenv("AZURE_API_KEY"),
-)
-
-
-async def call_acompletion(semaphore, input_data):
- async with semaphore:
- try:
- # Use asyncio.wait_for to set a timeout for the task
- response = await openai_client.chat.completions.create(**input_data)
- # Handle the response as needed
- print(response)
- return response
- except Timeout:
- print(f"Task timed out: {input_data}")
- return None # You may choose to return something else or raise an exception
-
-
-async def main():
- # Initialize the Router
-
- # Create a semaphore with a capacity of 100
- semaphore = asyncio.Semaphore(100)
-
- # List to hold all task references
- tasks = []
- start_time_all_tasks = time.time()
- # Launch 1000 tasks
- for _ in range(500):
- task = asyncio.create_task(
- call_acompletion(
- semaphore,
- {
- "model": "chatgpt-v-2",
- "messages": [{"role": "user", "content": "Hey, how's it going?"}],
- },
- )
- )
- tasks.append(task)
-
- # Wait for all tasks to complete
- responses = await asyncio.gather(*tasks)
- # Process responses as needed
- # Record the end time for all tasks
- end_time_all_tasks = time.time()
- # Calculate the total time for all tasks
- total_time_all_tasks = end_time_all_tasks - start_time_all_tasks
- print(f"Total time for all tasks: {total_time_all_tasks} seconds")
-
- # Calculate the average time per response
- average_time_per_response = total_time_all_tasks / len(responses)
- print(f"Average time per response: {average_time_per_response} seconds")
- print(f"NUMBER OF COMPLETED TASKS: {len(responses)}")
-
-
-# Run the main function
-asyncio.run(main())
diff --git a/cookbook/litellm_router_load_test/test_loadtest_router.py b/cookbook/litellm_router_load_test/test_loadtest_router.py
deleted file mode 100644
index 280e495e771..00000000000
--- a/cookbook/litellm_router_load_test/test_loadtest_router.py
+++ /dev/null
@@ -1,87 +0,0 @@
-import sys
-import os
-from dotenv import load_dotenv
-
-load_dotenv()
-sys.path.insert(
- 0, os.path.abspath("../..")
-) # Adds the parent directory to the system path
-import asyncio
-from litellm import Router, Timeout
-import time
-
-### Test calling router async
-
-
-async def call_acompletion(semaphore, router: Router, input_data):
- async with semaphore:
- try:
- # Use asyncio.wait_for to set a timeout for the task
- response = await router.acompletion(**input_data)
- # Handle the response as needed
- print(response)
- return response
- except Timeout:
- print(f"Task timed out: {input_data}")
- return None # You may choose to return something else or raise an exception
-
-
-async def main():
- # Initialize the Router
- model_list = [
- {
- "model_name": "gpt-3.5-turbo",
- "litellm_params": {
- "model": "gpt-3.5-turbo",
- "api_key": os.getenv("OPENAI_API_KEY"),
- },
- },
- {
- "model_name": "gpt-3.5-turbo",
- "litellm_params": {
- "model": "azure/chatgpt-v-2",
- "api_key": os.getenv("AZURE_API_KEY"),
- "api_base": os.getenv("AZURE_API_BASE"),
- "api_version": os.getenv("AZURE_API_VERSION"),
- },
- },
- ]
- router = Router(model_list=model_list, num_retries=3, timeout=10)
-
- # Create a semaphore with a capacity of 100
- semaphore = asyncio.Semaphore(100)
-
- # List to hold all task references
- tasks = []
- start_time_all_tasks = time.time()
- # Launch 1000 tasks
- for _ in range(500):
- task = asyncio.create_task(
- call_acompletion(
- semaphore,
- router,
- {
- "model": "gpt-3.5-turbo",
- "messages": [{"role": "user", "content": "Hey, how's it going?"}],
- },
- )
- )
- tasks.append(task)
-
- # Wait for all tasks to complete
- responses = await asyncio.gather(*tasks)
- # Process responses as needed
- # Record the end time for all tasks
- end_time_all_tasks = time.time()
- # Calculate the total time for all tasks
- total_time_all_tasks = end_time_all_tasks - start_time_all_tasks
- print(f"Total time for all tasks: {total_time_all_tasks} seconds")
-
- # Calculate the average time per response
- average_time_per_response = total_time_all_tasks / len(responses)
- print(f"Average time per response: {average_time_per_response} seconds")
- print(f"NUMBER OF COMPLETED TASKS: {len(responses)}")
-
-
-# Run the main function
-asyncio.run(main())
diff --git a/cookbook/litellm_router_load_test/test_loadtest_router_withs3_cache.py b/cookbook/litellm_router_load_test/test_loadtest_router_withs3_cache.py
deleted file mode 100644
index b093489be1b..00000000000
--- a/cookbook/litellm_router_load_test/test_loadtest_router_withs3_cache.py
+++ /dev/null
@@ -1,93 +0,0 @@
-import sys
-import os
-from dotenv import load_dotenv
-
-load_dotenv()
-sys.path.insert(
- 0, os.path.abspath("../..")
-) # Adds the parent directory to the system path
-import asyncio
-from litellm import Router, Timeout
-import time
-from litellm.caching.caching import Cache
-import litellm
-
-litellm.cache = Cache(
- type="s3", s3_bucket_name="cache-bucket-litellm", s3_region_name="us-west-2"
-)
-
-### Test calling router with s3 Cache
-
-
-async def call_acompletion(semaphore, router: Router, input_data):
- async with semaphore:
- try:
- # Use asyncio.wait_for to set a timeout for the task
- response = await router.acompletion(**input_data)
- # Handle the response as needed
- print(response)
- return response
- except Timeout:
- print(f"Task timed out: {input_data}")
- return None # You may choose to return something else or raise an exception
-
-
-async def main():
- # Initialize the Router
- model_list = [
- {
- "model_name": "gpt-3.5-turbo",
- "litellm_params": {
- "model": "gpt-3.5-turbo",
- "api_key": os.getenv("OPENAI_API_KEY"),
- },
- },
- {
- "model_name": "gpt-3.5-turbo",
- "litellm_params": {
- "model": "azure/chatgpt-v-2",
- "api_key": os.getenv("AZURE_API_KEY"),
- "api_base": os.getenv("AZURE_API_BASE"),
- "api_version": os.getenv("AZURE_API_VERSION"),
- },
- },
- ]
- router = Router(model_list=model_list, num_retries=3, timeout=10)
-
- # Create a semaphore with a capacity of 100
- semaphore = asyncio.Semaphore(100)
-
- # List to hold all task references
- tasks = []
- start_time_all_tasks = time.time()
- # Launch 1000 tasks
- for _ in range(500):
- task = asyncio.create_task(
- call_acompletion(
- semaphore,
- router,
- {
- "model": "gpt-3.5-turbo",
- "messages": [{"role": "user", "content": "Hey, how's it going?"}],
- },
- )
- )
- tasks.append(task)
-
- # Wait for all tasks to complete
- responses = await asyncio.gather(*tasks)
- # Process responses as needed
- # Record the end time for all tasks
- end_time_all_tasks = time.time()
- # Calculate the total time for all tasks
- total_time_all_tasks = end_time_all_tasks - start_time_all_tasks
- print(f"Total time for all tasks: {total_time_all_tasks} seconds")
-
- # Calculate the average time per response
- average_time_per_response = total_time_all_tasks / len(responses)
- print(f"Average time per response: {average_time_per_response} seconds")
- print(f"NUMBER OF COMPLETED TASKS: {len(responses)}")
-
-
-# Run the main function
-asyncio.run(main())
diff --git a/cookbook/litellm_test_multiple_llm_demo.ipynb b/cookbook/litellm_test_multiple_llm_demo.ipynb
deleted file mode 100644
index f22448e46b9..00000000000
--- a/cookbook/litellm_test_multiple_llm_demo.ipynb
+++ /dev/null
@@ -1,55 +0,0 @@
-{
- "nbformat": 4,
- "nbformat_minor": 0,
- "metadata": {
- "colab": {
- "provenance": []
- },
- "kernelspec": {
- "name": "python3",
- "display_name": "Python 3"
- },
- "language_info": {
- "name": "python"
- }
- },
- "cells": [
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "adotBkqZSh5g"
- },
- "outputs": [],
- "source": [
- "!pip install litellm"
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "from litellm import completion\n",
- "\n",
- "## set ENV variables\n",
- "os.environ[\"OPENAI_API_KEY\"] = \"openai key\"\n",
- "os.environ[\"COHERE_API_KEY\"] = \"cohere key\"\n",
- "os.environ[\"REPLICATE_API_KEY\"] = \"replicate key\"\n",
- "messages = [{ \"content\": \"Hello, how are you?\",\"role\": \"user\"}]\n",
- "\n",
- "# openai call\n",
- "response = completion(model=\"gpt-3.5-turbo\", messages=messages)\n",
- "\n",
- "# cohere call\n",
- "response = completion(\"command-nightly\", messages)\n",
- "\n",
- "# replicate call\n",
- "response = completion(\"replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1\", messages)"
- ],
- "metadata": {
- "id": "LeOqznSgSj-z"
- },
- "execution_count": null,
- "outputs": []
- }
- ]
-}
diff --git a/cookbook/livekit_agent_sdk/README.md b/cookbook/livekit_agent_sdk/README.md
deleted file mode 100644
index 6c794165e96..00000000000
--- a/cookbook/livekit_agent_sdk/README.md
+++ /dev/null
@@ -1,116 +0,0 @@
-# LiveKit Voice Agent with LiteLLM Gateway
-
-Simple example showing how to use LiveKit's xAI realtime plugin with LiteLLM as a proxy. This lets you switch between xAI, OpenAI, and Azure realtime APIs without changing your code.
-
-## Quick Start
-
-### 1. Install dependencies
-
-```bash
-pip install livekit-agents[xai] websockets
-```
-
-### 2. Start LiteLLM proxy
-
-```bash
-# With xAI
-export XAI_API_KEY="your-xai-key"
-export LITELLM_MASTER_KEY="sk-$(openssl rand -hex 32)"
-litellm --config config.yaml --port 4000
-```
-
-### 3. Run the voice agent
-
-```bash
-export LITELLM_API_KEY="$LITELLM_MASTER_KEY"
-python main.py
-```
-
-Type your message and get a voice response from Grok!
-
-## Configuration
-
-Set these environment variables if needed:
-
-```bash
-export LITELLM_PROXY_URL="http://localhost:4000"
-export LITELLM_API_KEY="$LITELLM_MASTER_KEY"
-export LITELLM_MODEL="grok-voice-agent"
-```
-
-Or use the defaults - connects to `http://localhost:4000` by default.
-
-## Example Config File
-
-Create a `config.yaml` with your realtime models:
-
-```yaml
-model_list:
- - model_name: grok-voice-agent
- litellm_params:
- model: xai/grok-2-vision-1212
- api_key: os.environ/XAI_API_KEY
- model_info:
- mode: realtime
-
- - model_name: openai-voice-agent
- litellm_params:
- model: gpt-4o-realtime-preview
- api_key: os.environ/OPENAI_API_KEY
- model_info:
- mode: realtime
-
-general_settings:
- master_key: os.environ/LITELLM_MASTER_KEY
-```
-
-Then start: `litellm --config config.yaml --port 4000`
-
-## How It Works
-
-LiveKit's xAI plugin connects through LiteLLM proxy by setting `base_url`:
-
-```python
-from livekit.plugins import xai
-
-model = xai.realtime.RealtimeModel(
- voice="ara",
- api_key=os.environ["LITELLM_MASTER_KEY"],
- base_url="http://localhost:4000", # Point to LiteLLM
-)
-```
-
-## Switching Providers
-
-Just change the model in your config - no code changes needed:
-
-**xAI Grok:**
-```yaml
-model: xai/grok-2-vision-1212
-```
-
-**OpenAI:**
-```yaml
-model: gpt-4o-realtime-preview
-```
-
-**Azure OpenAI:**
-```yaml
-model: azure/gpt-4o-realtime-preview
-api_base: https://your-endpoint.openai.azure.com/
-```
-
-## Why Use LiteLLM?
-
-- ā
**Switch providers** without changing agent code
-- ā
**Cost tracking** across all voice sessions
-- ā
**Rate limiting** and budgets
-- ā
**Load balancing** across multiple API keys
-- ā
**Fallbacks** to backup models
-
-## Learn More
-
-- [LiveKit xAI Realtime Tutorial](/docs/tutorials/livekit_xai_realtime)
-- [xAI Realtime Docs](/docs/providers/xai_realtime)
-- [LiveKit Agents Documentation](https://docs.livekit.io/agents/)
-- [LiteLLM Realtime API](/docs/realtime)
diff --git a/cookbook/livekit_agent_sdk/config.example.yaml b/cookbook/livekit_agent_sdk/config.example.yaml
deleted file mode 100644
index 388aa24f501..00000000000
--- a/cookbook/livekit_agent_sdk/config.example.yaml
+++ /dev/null
@@ -1,20 +0,0 @@
-model_list:
- - model_name: grok-voice-agent
- litellm_params:
- model: xai/grok-2-vision-1212
- api_key: os.environ/XAI_API_KEY
- model_info:
- mode: realtime
-
- - model_name: openai-voice-agent
- litellm_params:
- model: gpt-4o-realtime-preview
- api_key: os.environ/OPENAI_API_KEY
- model_info:
- mode: realtime
-
-litellm_settings:
- drop_params: True
-
-general_settings:
- master_key: os.environ/LITELLM_MASTER_KEY
diff --git a/cookbook/livekit_agent_sdk/main.py b/cookbook/livekit_agent_sdk/main.py
deleted file mode 100644
index fdc72e518b4..00000000000
--- a/cookbook/livekit_agent_sdk/main.py
+++ /dev/null
@@ -1,121 +0,0 @@
-"""
-Simple xAI Voice Agent using LiveKit SDK with LiteLLM Gateway
-
-This example shows how to use LiveKit's xAI realtime plugin through LiteLLM proxy.
-LiteLLM acts as a unified interface, allowing you to switch between xAI, OpenAI,
-and Azure realtime APIs without changing your agent code.
-"""
-
-import asyncio
-import json
-import os
-import websockets
-
-# Configuration
-PROXY_URL = os.getenv("LITELLM_PROXY_URL", "http://localhost:4000")
-API_KEY = os.environ["LITELLM_API_KEY"]
-MODEL = os.getenv("LITELLM_MODEL", "grok-voice-agent")
-
-
-async def run_voice_agent():
- """
- Simple voice agent that:
- 1. Connects to xAI realtime API through LiteLLM proxy
- 2. Sends a user message
- 3. Streams back the response
- """
-
- url = f"ws://{PROXY_URL.replace('http://', '').replace('https://', '')}/v1/realtime?model={MODEL}"
- headers = {"Authorization": f"Bearer {API_KEY}"}
-
- print(f"šļø Connecting to voice agent...")
- print(f" Model: {MODEL}")
- print(f" Proxy: {PROXY_URL}")
- print()
-
- async with websockets.connect(url, additional_headers=headers) as ws:
- # Receive initial connection event
- initial = json.loads(await ws.recv())
- print(f"ā
Connected! Event: {initial['type']}\n")
-
- # Get user input
- user_message = input("š¬ Your message: ").strip()
- if not user_message:
- user_message = "Tell me a fun fact about AI!"
-
- print(f"\nš¤ Sending to {MODEL}...\n")
-
- # Send user message
- await ws.send(
- json.dumps(
- {
- "type": "conversation.item.create",
- "item": {
- "type": "message",
- "role": "user",
- "content": [{"type": "input_text", "text": user_message}],
- },
- }
- )
- )
-
- # Request response
- await ws.send(
- json.dumps(
- {
- "type": "response.create",
- "response": {"modalities": ["text", "audio"]},
- }
- )
- )
-
- # Stream response
- print("š¤ Response: ", end="", flush=True)
- transcript = []
-
- try:
- while True:
- msg = await asyncio.wait_for(ws.recv(), timeout=15.0)
- event = json.loads(msg)
-
- # Capture transcript deltas
- if event["type"] == "response.output_audio_transcript.delta":
- delta = event.get("delta", "")
- if delta:
- print(delta, end="", flush=True)
- transcript.append(delta)
-
- # Done when response completes
- elif event["type"] == "response.done":
- break
-
- except asyncio.TimeoutError:
- pass
-
- print("\n")
-
- if transcript:
- print(f"ā
Complete response: {''.join(transcript)}")
-
- await ws.close()
-
-
-def main():
- """Run the voice agent"""
- print("=" * 70)
- print("LiveKit xAI Voice Agent via LiteLLM Proxy")
- print("=" * 70)
- print()
-
- try:
- asyncio.run(run_voice_agent())
- except KeyboardInterrupt:
- print("\n\nš Goodbye!")
- except Exception as e:
- print(f"\nā Error: {e}")
- print("\nMake sure LiteLLM proxy is running:")
- print(f" litellm --config config.yaml --port 4000")
-
-
-if __name__ == "__main__":
- main()
diff --git a/cookbook/livekit_agent_sdk/requirements.txt b/cookbook/livekit_agent_sdk/requirements.txt
deleted file mode 100644
index 9e3542fac27..00000000000
--- a/cookbook/livekit_agent_sdk/requirements.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-livekit-agents[xai]>=1.3.12
-websockets>=15.0.1
diff --git a/cookbook/logging_observability/LiteLLM_Arize.ipynb b/cookbook/logging_observability/LiteLLM_Arize.ipynb
deleted file mode 100644
index 72a082f874d..00000000000
--- a/cookbook/logging_observability/LiteLLM_Arize.ipynb
+++ /dev/null
@@ -1,172 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "4FbDOmcj2VkM"
- },
- "source": [
- "## Use LiteLLM with Arize\n",
- "https://docs.litellm.ai/docs/observability/arize_integration\n",
- "\n",
- "This method uses the litellm proxy to send the data to Arize. The callback is set in the litellm config below, instead of using OpenInference tracing."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "21W8Woog26Ns"
- },
- "source": [
- "## Install Dependencies"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {
- "id": "xrjKLBxhxu2L"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Requirement already satisfied: litellm in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (1.54.1)\n",
- "Requirement already satisfied: aiohttp in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (3.11.10)\n",
- "Requirement already satisfied: click in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (8.1.7)\n",
- "Requirement already satisfied: httpx<0.28.0,>=0.23.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (0.27.2)\n",
- "Requirement already satisfied: importlib-metadata>=6.8.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (8.5.0)\n",
- "Requirement already satisfied: jinja2<4.0.0,>=3.1.2 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (3.1.4)\n",
- "Requirement already satisfied: jsonschema<5.0.0,>=4.22.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (4.23.0)\n",
- "Requirement already satisfied: openai>=1.55.3 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (1.57.1)\n",
- "Requirement already satisfied: pydantic<3.0.0,>=2.0.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (2.10.3)\n",
- "Requirement already satisfied: python-dotenv>=0.2.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (1.0.1)\n",
- "Requirement already satisfied: requests<3.0.0,>=2.31.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (2.32.3)\n",
- "Requirement already satisfied: tiktoken>=0.7.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (0.7.0)\n",
- "Requirement already satisfied: tokenizers in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from litellm) (0.21.0)\n",
- "Requirement already satisfied: anyio in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from httpx<0.28.0,>=0.23.0->litellm) (4.7.0)\n",
- "Requirement already satisfied: certifi in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from httpx<0.28.0,>=0.23.0->litellm) (2024.8.30)\n",
- "Requirement already satisfied: httpcore==1.* in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from httpx<0.28.0,>=0.23.0->litellm) (1.0.7)\n",
- "Requirement already satisfied: idna in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from httpx<0.28.0,>=0.23.0->litellm) (3.10)\n",
- "Requirement already satisfied: sniffio in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from httpx<0.28.0,>=0.23.0->litellm) (1.3.1)\n",
- "Requirement already satisfied: h11<0.15,>=0.13 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from httpcore==1.*->httpx<0.28.0,>=0.23.0->litellm) (0.14.0)\n",
- "Requirement already satisfied: zipp>=3.20 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from importlib-metadata>=6.8.0->litellm) (3.21.0)\n",
- "Requirement already satisfied: MarkupSafe>=2.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from jinja2<4.0.0,>=3.1.2->litellm) (3.0.2)\n",
- "Requirement already satisfied: attrs>=22.2.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from jsonschema<5.0.0,>=4.22.0->litellm) (24.2.0)\n",
- "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from jsonschema<5.0.0,>=4.22.0->litellm) (2024.10.1)\n",
- "Requirement already satisfied: referencing>=0.28.4 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from jsonschema<5.0.0,>=4.22.0->litellm) (0.35.1)\n",
- "Requirement already satisfied: rpds-py>=0.7.1 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from jsonschema<5.0.0,>=4.22.0->litellm) (0.22.3)\n",
- "Requirement already satisfied: distro<2,>=1.7.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from openai>=1.55.3->litellm) (1.9.0)\n",
- "Requirement already satisfied: jiter<1,>=0.4.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from openai>=1.55.3->litellm) (0.6.1)\n",
- "Requirement already satisfied: tqdm>4 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from openai>=1.55.3->litellm) (4.67.1)\n",
- "Requirement already satisfied: typing-extensions<5,>=4.11 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from openai>=1.55.3->litellm) (4.12.2)\n",
- "Requirement already satisfied: annotated-types>=0.6.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from pydantic<3.0.0,>=2.0.0->litellm) (0.7.0)\n",
- "Requirement already satisfied: pydantic-core==2.27.1 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from pydantic<3.0.0,>=2.0.0->litellm) (2.27.1)\n",
- "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from requests<3.0.0,>=2.31.0->litellm) (3.4.0)\n",
- "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from requests<3.0.0,>=2.31.0->litellm) (2.0.7)\n",
- "Requirement already satisfied: regex>=2022.1.18 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from tiktoken>=0.7.0->litellm) (2024.11.6)\n",
- "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from aiohttp->litellm) (2.4.4)\n",
- "Requirement already satisfied: aiosignal>=1.1.2 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from aiohttp->litellm) (1.3.1)\n",
- "Requirement already satisfied: frozenlist>=1.1.1 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from aiohttp->litellm) (1.5.0)\n",
- "Requirement already satisfied: multidict<7.0,>=4.5 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from aiohttp->litellm) (6.1.0)\n",
- "Requirement already satisfied: propcache>=0.2.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from aiohttp->litellm) (0.2.1)\n",
- "Requirement already satisfied: yarl<2.0,>=1.17.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from aiohttp->litellm) (1.18.3)\n",
- "Requirement already satisfied: huggingface-hub<1.0,>=0.16.4 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from tokenizers->litellm) (0.26.5)\n",
- "Requirement already satisfied: filelock in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers->litellm) (3.16.1)\n",
- "Requirement already satisfied: fsspec>=2023.5.0 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers->litellm) (2024.10.0)\n",
- "Requirement already satisfied: packaging>=20.9 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers->litellm) (24.2)\n",
- "Requirement already satisfied: pyyaml>=5.1 in /Users/ericxiao/Documents/arize/.venv/lib/python3.11/site-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers->litellm) (6.0.2)\n"
- ]
- }
- ],
- "source": [
- "!pip install litellm"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "jHEu-TjZ29PJ"
- },
- "source": [
- "## Set Env Variables"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {
- "id": "QWd9rTysxsWO"
- },
- "outputs": [],
- "source": [
- "import litellm\n",
- "import os\n",
- "from getpass import getpass\n",
- "\n",
- "os.environ[\"ARIZE_SPACE_KEY\"] = getpass(\"Enter your Arize space key: \")\n",
- "os.environ[\"ARIZE_API_KEY\"] = getpass(\"Enter your Arize API key: \")\n",
- "os.environ['OPENAI_API_KEY']= getpass(\"Enter your OpenAI API key: \")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's run a completion call and see the traces in Arize"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Hello! Nice to meet you, OpenAI. How can I assist you today?\n"
- ]
- }
- ],
- "source": [
- "# set arize as a callback, litellm will send the data to arize\n",
- "litellm.callbacks = [\"arize\"]\n",
- " \n",
- "# openai call\n",
- "response = litellm.completion(\n",
- " model=\"gpt-3.5-turbo\",\n",
- " messages=[\n",
- " {\"role\": \"user\", \"content\": \"Hi š - i'm openai\"}\n",
- " ]\n",
- ")\n",
- "print(response.choices[0].message.content)"
- ]
- }
- ],
- "metadata": {
- "colab": {
- "provenance": []
- },
- "kernelspec": {
- "display_name": ".venv",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.6"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}
diff --git a/cookbook/logging_observability/LiteLLM_Langfuse.ipynb b/cookbook/logging_observability/LiteLLM_Langfuse.ipynb
deleted file mode 100644
index 2a63666e069..00000000000
--- a/cookbook/logging_observability/LiteLLM_Langfuse.ipynb
+++ /dev/null
@@ -1,197 +0,0 @@
-{
- "nbformat": 4,
- "nbformat_minor": 0,
- "metadata": {
- "colab": {
- "provenance": []
- },
- "kernelspec": {
- "name": "python3",
- "display_name": "Python 3"
- },
- "language_info": {
- "name": "python"
- }
- },
- "cells": [
- {
- "cell_type": "markdown",
- "source": [
- "## Use LiteLLM with Langfuse\n",
- "https://docs.litellm.ai/docs/observability/langfuse_integration"
- ],
- "metadata": {
- "id": "4FbDOmcj2VkM"
- }
- },
- {
- "cell_type": "markdown",
- "source": [
- "## Install Dependencies"
- ],
- "metadata": {
- "id": "21W8Woog26Ns"
- }
- },
- {
- "cell_type": "code",
- "source": [
- "!pip install litellm langfuse"
- ],
- "metadata": {
- "id": "xrjKLBxhxu2L"
- },
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "source": [
- "## Set Env Variables"
- ],
- "metadata": {
- "id": "jHEu-TjZ29PJ"
- }
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {
- "id": "QWd9rTysxsWO"
- },
- "outputs": [],
- "source": [
- "import litellm\n",
- "from litellm import completion\n",
- "import os\n",
- "\n",
- "# from https://cloud.langfuse.com/\n",
- "os.environ[\"LANGFUSE_PUBLIC_KEY\"] = \"\"\n",
- "os.environ[\"LANGFUSE_SECRET_KEY\"] = \"\"\n",
- "\n",
- "\n",
- "# OpenAI and Cohere keys\n",
- "# You can use any of the litellm supported providers: https://docs.litellm.ai/docs/providers\n",
- "os.environ['OPENAI_API_KEY']=\"\"\n",
- "os.environ['COHERE_API_KEY']=\"\"\n"
- ]
- },
- {
- "cell_type": "markdown",
- "source": [
- "## Set LangFuse as a callback for sending data\n",
- "## OpenAI completion call"
- ],
- "metadata": {
- "id": "NodQl0hp3Lma"
- }
- },
- {
- "cell_type": "code",
- "source": [
- "# set langfuse as a callback, litellm will send the data to langfuse\n",
- "litellm.success_callback = [\"langfuse\"]\n",
- "\n",
- "# openai call\n",
- "response = completion(\n",
- " model=\"gpt-3.5-turbo\",\n",
- " messages=[\n",
- " {\"role\": \"user\", \"content\": \"Hi š - i'm openai\"}\n",
- " ]\n",
- ")\n",
- "\n",
- "print(response)"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "vNAuwJY1yp_F",
- "outputId": "c3a71e26-13f5-4379-fac9-409290ba79bb"
- },
- "execution_count": 8,
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "{\n",
- " \"id\": \"chatcmpl-85nP4xHdAP3jAcGneIguWATS9qdoO\",\n",
- " \"object\": \"chat.completion\",\n",
- " \"created\": 1696392238,\n",
- " \"model\": \"gpt-3.5-turbo-0613\",\n",
- " \"choices\": [\n",
- " {\n",
- " \"index\": 0,\n",
- " \"message\": {\n",
- " \"role\": \"assistant\",\n",
- " \"content\": \"Hello! How can I assist you today?\"\n",
- " },\n",
- " \"finish_reason\": \"stop\"\n",
- " }\n",
- " ],\n",
- " \"usage\": {\n",
- " \"prompt_tokens\": 15,\n",
- " \"completion_tokens\": 9,\n",
- " \"total_tokens\": 24\n",
- " }\n",
- "}\n"
- ]
- }
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "# we set langfuse as a callback in the prev cell\n",
- "# cohere call\n",
- "response = completion(\n",
- " model=\"command-nightly\",\n",
- " messages=[\n",
- " {\"role\": \"user\", \"content\": \"Hi š - i'm cohere\"}\n",
- " ]\n",
- ")\n",
- "\n",
- "print(response)"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "2PMSLc_FziJL",
- "outputId": "1c37605e-b406-4ffc-aafd-e1983489c6be"
- },
- "execution_count": 9,
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "{\n",
- " \"object\": \"chat.completion\",\n",
- " \"choices\": [\n",
- " {\n",
- " \"finish_reason\": \"stop\",\n",
- " \"index\": 0,\n",
- " \"message\": {\n",
- " \"content\": \" Nice to meet you, Cohere! I'm excited to be meeting new members of the AI community\",\n",
- " \"role\": \"assistant\",\n",
- " \"logprobs\": null\n",
- " }\n",
- " }\n",
- " ],\n",
- " \"id\": \"chatcmpl-a14e903f-4608-4ceb-b996-8ebdf21360ca\",\n",
- " \"created\": 1696392247.3313863,\n",
- " \"model\": \"command-nightly\",\n",
- " \"usage\": {\n",
- " \"prompt_tokens\": 8,\n",
- " \"completion_tokens\": 20,\n",
- " \"total_tokens\": 28\n",
- " }\n",
- "}\n"
- ]
- }
- ]
- }
- ]
-}
\ No newline at end of file
diff --git a/cookbook/logging_observability/LiteLLM_Lunary.ipynb b/cookbook/logging_observability/LiteLLM_Lunary.ipynb
deleted file mode 100644
index 3b1dc5d5e25..00000000000
--- a/cookbook/logging_observability/LiteLLM_Lunary.ipynb
+++ /dev/null
@@ -1,348 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "4FbDOmcj2VkM"
- },
- "source": [
- "## Use LiteLLM with Langfuse\n",
- "https://docs.litellm.ai/docs/observability/langfuse_integration"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "21W8Woog26Ns"
- },
- "source": [
- "## Install Dependencies"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "xrjKLBxhxu2L"
- },
- "outputs": [],
- "source": [
- "%pip install litellm lunary"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "jHEu-TjZ29PJ"
- },
- "source": [
- "## Set Env Variables"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {
- "id": "QWd9rTysxsWO"
- },
- "outputs": [],
- "source": [
- "import litellm\n",
- "from litellm import completion\n",
- "import os\n",
- "\n",
- "# from https://app.lunary.ai/\n",
- "os.environ[\"LUNARY_PUBLIC_KEY\"] = \"\"\n",
- "\n",
- "\n",
- "# LLM provider keys\n",
- "# You can use any of the litellm supported providers: https://docs.litellm.ai/docs/providers\n",
- "os.environ['OPENAI_API_KEY'] = \"\"\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "NodQl0hp3Lma"
- },
- "source": [
- "## Set Lunary as a callback for sending data\n",
- "## OpenAI completion call"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "vNAuwJY1yp_F",
- "outputId": "c3a71e26-13f5-4379-fac9-409290ba79bb"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[Choices(finish_reason='stop', index=0, message=Message(content='Hello! How can I assist you today?', role='assistant'))]ModelResponse(id='chatcmpl-8xIWykI0GiJSmYtXYuB8Z363kpIBm', choices=[Choices(finish_reason='stop', index=0, message=Message(content='Hello! How can I assist you today?', role='assistant'))], created=1709143276, model='gpt-3.5-turbo-0125', object='chat.completion', system_fingerprint='fp_86156a94a0', usage=Usage(completion_tokens=9, prompt_tokens=15, total_tokens=24))\n",
- "\n",
- "[Lunary] Add event: {\n",
- " \"event\": \"start\",\n",
- " \"type\": \"llm\",\n",
- " \"name\": \"gpt-3.5-turbo\",\n",
- " \"runId\": \"a363776a-bd07-4474-bce2-193067f01b2e\",\n",
- " \"timestamp\": \"2024-02-28T18:01:15.188153+00:00\",\n",
- " \"input\": {\n",
- " \"role\": \"user\",\n",
- " \"content\": \"Hi \\ud83d\\udc4b - i'm openai\"\n",
- " },\n",
- " \"extra\": {},\n",
- " \"runtime\": \"litellm\",\n",
- " \"metadata\": {}\n",
- "}\n",
- "\n",
- "\n",
- "[Lunary] Add event: {\n",
- " \"event\": \"end\",\n",
- " \"type\": \"llm\",\n",
- " \"runId\": \"a363776a-bd07-4474-bce2-193067f01b2e\",\n",
- " \"timestamp\": \"2024-02-28T18:01:16.846581+00:00\",\n",
- " \"output\": {\n",
- " \"role\": \"assistant\",\n",
- " \"content\": \"Hello! How can I assist you today?\"\n",
- " },\n",
- " \"runtime\": \"litellm\",\n",
- " \"tokensUsage\": {\n",
- " \"completion\": 9,\n",
- " \"prompt\": 15\n",
- " }\n",
- "}\n",
- "\n",
- "\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "--- Logging error ---\n",
- "Traceback (most recent call last):\n",
- " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 537, in _make_request\n",
- " response = conn.getresponse()\n",
- " ^^^^^^^^^^^^^^^^^^\n",
- " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/urllib3/connection.py\", line 466, in getresponse\n",
- " httplib_response = super().getresponse()\n",
- " ^^^^^^^^^^^^^^^^^^^^^\n",
- " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py\", line 1423, in getresponse\n",
- " response.begin()\n",
- " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py\", line 331, in begin\n",
- " version, status, reason = self._read_status()\n",
- " ^^^^^^^^^^^^^^^^^^^\n",
- " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py\", line 292, in _read_status\n",
- " line = str(self.fp.readline(_MAXLINE + 1), \"iso-8859-1\")\n",
- " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
- " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/socket.py\", line 707, in readinto\n",
- " return self._sock.recv_into(b)\n",
- " ^^^^^^^^^^^^^^^^^^^^^^^\n",
- "TimeoutError: timed out\n",
- "\n",
- "The above exception was the direct cause of the following exception:\n",
- "\n",
- "Traceback (most recent call last):\n",
- " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/requests/adapters.py\", line 486, in send\n",
- " resp = conn.urlopen(\n",
- " ^^^^^^^^^^^^^\n",
- " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 847, in urlopen\n",
- " retries = retries.increment(\n",
- " ^^^^^^^^^^^^^^^^^^\n",
- " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/urllib3/util/retry.py\", line 470, in increment\n",
- " raise reraise(type(error), error, _stacktrace)\n",
- " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
- " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/urllib3/util/util.py\", line 39, in reraise\n",
- " raise value\n",
- " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 793, in urlopen\n",
- " response = self._make_request(\n",
- " ^^^^^^^^^^^^^^^^^^^\n",
- " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 539, in _make_request\n",
- " self._raise_timeout(err=e, url=url, timeout_value=read_timeout)\n",
- " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 370, in _raise_timeout\n",
- " raise ReadTimeoutError(\n",
- "urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=3333): Read timed out. (read timeout=5)\n",
- "\n",
- "During handling of the above exception, another exception occurred:\n",
- "\n",
- "Traceback (most recent call last):\n",
- " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/lunary/consumer.py\", line 59, in send_batch\n",
- " response = requests.post(\n",
- " ^^^^^^^^^^^^^^\n",
- " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/requests/api.py\", line 115, in post\n",
- " return request(\"post\", url, data=data, json=json, **kwargs)\n",
- " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
- " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/requests/api.py\", line 59, in request\n",
- " return session.request(method=method, url=url, **kwargs)\n",
- " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
- " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/requests/sessions.py\", line 589, in request\n",
- " resp = self.send(prep, **send_kwargs)\n",
- " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
- " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/requests/sessions.py\", line 703, in send\n",
- " r = adapter.send(request, **kwargs)\n",
- " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
- " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/requests/adapters.py\", line 532, in send\n",
- " raise ReadTimeout(e, request=request)\n",
- "requests.exceptions.ReadTimeout: HTTPConnectionPool(host='localhost', port=3333): Read timed out. (read timeout=5)\n",
- "\n",
- "During handling of the above exception, another exception occurred:\n",
- "\n",
- "Traceback (most recent call last):\n",
- " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/logging/__init__.py\", line 1160, in emit\n",
- " msg = self.format(record)\n",
- " ^^^^^^^^^^^^^^^^^^^\n",
- " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/logging/__init__.py\", line 999, in format\n",
- " return fmt.format(record)\n",
- " ^^^^^^^^^^^^^^^^^^\n",
- " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/logging/__init__.py\", line 703, in format\n",
- " record.message = record.getMessage()\n",
- " ^^^^^^^^^^^^^^^^^^^\n",
- " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/logging/__init__.py\", line 392, in getMessage\n",
- " msg = msg % self.args\n",
- " ~~~~^~~~~~~~~~~\n",
- "TypeError: not all arguments converted during string formatting\n",
- "Call stack:\n",
- " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py\", line 1030, in _bootstrap\n",
- " self._bootstrap_inner()\n",
- " File \"/opt/homebrew/Cellar/python@3.12/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py\", line 1073, in _bootstrap_inner\n",
- " self.run()\n",
- " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/lunary/consumer.py\", line 24, in run\n",
- " self.send_batch()\n",
- " File \"/Users/vince/Library/Caches/pypoetry/virtualenvs/litellm-7WKnDWGw-py3.12/lib/python3.12/site-packages/lunary/consumer.py\", line 73, in send_batch\n",
- " logging.error(\"[Lunary] Error sending events\", e)\n",
- "Message: '[Lunary] Error sending events'\n",
- "Arguments: (ReadTimeout(ReadTimeoutError(\"HTTPConnectionPool(host='localhost', port=3333): Read timed out. (read timeout=5)\")),)\n"
- ]
- }
- ],
- "source": [
- "# set langfuse as a callback, litellm will send the data to langfuse\n",
- "litellm.success_callback = [\"lunary\"]\n",
- "\n",
- "# openai call\n",
- "response = completion(\n",
- " model=\"gpt-3.5-turbo\",\n",
- " messages=[\n",
- " {\"role\": \"user\", \"content\": \"Hi š - i'm openai\"}\n",
- " ]\n",
- ")\n",
- "\n",
- "print(response)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Using LiteLLM with Lunary Templates\n",
- "\n",
- "You can use LiteLLM seamlessly with Lunary templates to manage your prompts and completions.\n",
- "\n",
- "Assuming you have created a template \"test-template\" with a variable \"question\", you can use it like this:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "2PMSLc_FziJL",
- "outputId": "1c37605e-b406-4ffc-aafd-e1983489c6be"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[Choices(finish_reason='stop', index=0, message=Message(content='Hello! How can I assist you today?', role='assistant'))]ModelResponse(id='chatcmpl-8xIXegwpudg4YKnLB6pmpFGXqTHcH', choices=[Choices(finish_reason='stop', index=0, message=Message(content='Hello! How can I assist you today?', role='assistant'))], created=1709143318, model='gpt-4-0125-preview', object='chat.completion', system_fingerprint='fp_c8aa5a06d6', usage=Usage(completion_tokens=9, prompt_tokens=21, total_tokens=30))\n",
- "\n",
- "[Lunary] Add event: {\n",
- " \"event\": \"start\",\n",
- " \"type\": \"llm\",\n",
- " \"name\": \"gpt-4-turbo-preview\",\n",
- " \"runId\": \"3a5b698d-cb55-4b3b-ab6d-04d2b99e40cb\",\n",
- " \"timestamp\": \"2024-02-28T18:01:56.746249+00:00\",\n",
- " \"input\": [\n",
- " {\n",
- " \"role\": \"system\",\n",
- " \"content\": \"You are an helpful assistant.\"\n",
- " },\n",
- " {\n",
- " \"role\": \"user\",\n",
- " \"content\": \"Hi! Hello!\"\n",
- " }\n",
- " ],\n",
- " \"extra\": {\n",
- " \"temperature\": 1,\n",
- " \"max_tokens\": 100\n",
- " },\n",
- " \"runtime\": \"litellm\",\n",
- " \"metadata\": {}\n",
- "}\n",
- "\n",
- "\n",
- "[Lunary] Add event: {\n",
- " \"event\": \"end\",\n",
- " \"type\": \"llm\",\n",
- " \"runId\": \"3a5b698d-cb55-4b3b-ab6d-04d2b99e40cb\",\n",
- " \"timestamp\": \"2024-02-28T18:01:58.741244+00:00\",\n",
- " \"output\": {\n",
- " \"role\": \"assistant\",\n",
- " \"content\": \"Hello! How can I assist you today?\"\n",
- " },\n",
- " \"runtime\": \"litellm\",\n",
- " \"tokensUsage\": {\n",
- " \"completion\": 9,\n",
- " \"prompt\": 21\n",
- " }\n",
- "}\n",
- "\n",
- "\n"
- ]
- }
- ],
- "source": [
- "import lunary\n",
- "from litellm import completion\n",
- "\n",
- "template = lunary.render_template(\"test-template\", {\"question\": \"Hello!\"})\n",
- "\n",
- "response = completion(**template)\n",
- "\n",
- "print(response)"
- ]
- }
- ],
- "metadata": {
- "colab": {
- "provenance": []
- },
- "kernelspec": {
- "display_name": "Python 3",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.12.2"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}
diff --git a/cookbook/logging_observability/LiteLLM_Proxy_Langfuse.ipynb b/cookbook/logging_observability/LiteLLM_Proxy_Langfuse.ipynb
deleted file mode 100644
index 0baaab3f49f..00000000000
--- a/cookbook/logging_observability/LiteLLM_Proxy_Langfuse.ipynb
+++ /dev/null
@@ -1,252 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## LLM Ops Stack - LiteLLM Proxy + Langfuse \n",
- "\n",
- "This notebook demonstrates how to use LiteLLM Proxy with Langfuse \n",
- "- Use LiteLLM Proxy for calling 100+ LLMs in OpenAI format\n",
- "- Use Langfuse for viewing request / response traces \n",
- "\n",
- "\n",
- "In this notebook we will setup LiteLLM Proxy to make requests to OpenAI, Anthropic, Bedrock and automatically log traces to Langfuse."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 1. Setup LiteLLM Proxy\n",
- "\n",
- "### 1.1 Define .env variables \n",
- "Define .env variables on the container that litellm proxy is running on.\n",
- "```bash\n",
- "## LLM API Keys\n",
- "OPENAI_API_KEY=sk-proj-1234567890\n",
- "ANTHROPIC_API_KEY=sk-ant-api03-1234567890\n",
- "AWS_ACCESS_KEY_ID=1234567890\n",
- "AWS_SECRET_ACCESS_KEY=1234567890\n",
- "\n",
- "## Langfuse Logging \n",
- "LANGFUSE_PUBLIC_KEY=\"pk-lf-xxxx9\"\n",
- "LANGFUSE_SECRET_KEY=\"sk-lf-xxxx9\"\n",
- "LANGFUSE_HOST=\"https://us.cloud.langfuse.com\"\n",
- "```\n",
- "\n",
- "\n",
- "### 1.1 Setup LiteLLM Proxy Config yaml \n",
- "```yaml\n",
- "model_list:\n",
- " - model_name: gpt-4o\n",
- " litellm_params:\n",
- " model: openai/gpt-4o\n",
- " api_key: os.environ/OPENAI_API_KEY\n",
- " - model_name: claude-3-5-sonnet-20241022\n",
- " litellm_params:\n",
- " model: anthropic/claude-3-5-sonnet-20241022\n",
- " api_key: os.environ/ANTHROPIC_API_KEY\n",
- " - model_name: us.amazon.nova-micro-v1:0\n",
- " litellm_params:\n",
- " model: bedrock/us.amazon.nova-micro-v1:0\n",
- " aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID\n",
- " aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY\n",
- "\n",
- "litellm_settings:\n",
- " callbacks: [\"langfuse\"]\n",
- "\n",
- "\n",
- "```"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 2. Make LLM Requests to LiteLLM Proxy\n",
- "\n",
- "Now we will make our first LLM request to LiteLLM Proxy"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 2.1 Setup Client Side Variables to point to LiteLLM Proxy\n",
- "Set `LITELLM_PROXY_BASE_URL` to the base url of the LiteLLM Proxy and `LITELLM_VIRTUAL_KEY` to the virtual key you want to use for Authentication to LiteLLM Proxy. (Note: In this initial setup you can)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 23,
- "metadata": {},
- "outputs": [],
- "source": [
- "\n",
- "LITELLM_PROXY_BASE_URL=\"http://0.0.0.0:4000\"\n",
- "LITELLM_VIRTUAL_KEY=\"sk-oXXRa1xxxxxxxxxxx\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 22,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "ChatCompletion(id='chatcmpl-B0sq6QkOKNMJ0dwP3x7OoMqk1jZcI', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='Langfuse is a platform designed to monitor, observe, and troubleshoot AI and large language model (LLM) applications. It provides features that help developers gain insights into how their AI systems are performing, make debugging easier, and optimize the deployment of models. Langfuse allows for tracking of model interactions, collecting telemetry, and visualizing data, which is crucial for understanding the behavior of AI models in production environments. This kind of tool is particularly useful for developers working with language models who need to ensure reliability and efficiency in their applications.', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1739550502, model='gpt-4o-2024-08-06', object='chat.completion', service_tier='default', system_fingerprint='fp_523b9b6e5f', usage=CompletionUsage(completion_tokens=109, prompt_tokens=13, total_tokens=122, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)))"
- ]
- },
- "execution_count": 22,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "import openai\n",
- "client = openai.OpenAI(\n",
- " api_key=LITELLM_VIRTUAL_KEY,\n",
- " base_url=LITELLM_PROXY_BASE_URL\n",
- ")\n",
- "\n",
- "response = client.chat.completions.create(\n",
- " model=\"gpt-4o\",\n",
- " messages = [\n",
- " {\n",
- " \"role\": \"user\",\n",
- " \"content\": \"what is Langfuse?\"\n",
- " }\n",
- " ],\n",
- ")\n",
- "\n",
- "response"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 2.3 View Traces on Langfuse\n",
- "LiteLLM will send the request / response, model, tokens (input + output), cost to Langfuse.\n",
- "\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 2.4 Call Anthropic, Bedrock models \n",
- "\n",
- "Now we can call `us.amazon.nova-micro-v1:0` and `claude-3-5-sonnet-20241022` models defined on your config.yaml both in the OpenAI request / response format."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 24,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "ChatCompletion(id='chatcmpl-7756e509-e61f-4f5e-b5ae-b7a41013522a', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=\"Langfuse is an observability tool designed specifically for machine learning models and applications built with natural language processing (NLP) and large language models (LLMs). It focuses on providing detailed insights into how these models perform in real-world scenarios. Here are some key features and purposes of Langfuse:\\n\\n1. **Real-time Monitoring**: Langfuse allows developers to monitor the performance of their NLP and LLM applications in real time. This includes tracking the inputs and outputs of the models, as well as any errors or issues that arise during operation.\\n\\n2. **Error Tracking**: It helps in identifying and tracking errors in the models' outputs. By analyzing incorrect or unexpected responses, developers can pinpoint where and why errors occur, facilitating more effective debugging and improvement.\\n\\n3. **Performance Metrics**: Langfuse provides various performance metrics, such as latency, throughput, and error rates. These metrics help developers understand how well their models are performing under different conditions and workloads.\\n\\n4. **Traceability**: It offers detailed traceability of requests and responses, allowing developers to follow the path of a request through the system and see how it is processed by the model at each step.\\n\\n5. **User Feedback Integration**: Langfuse can integrate user feedback to provide context for model outputs. This helps in understanding how real users are interacting with the model and how its outputs align with user expectations.\\n\\n6. **Customizable Dashboards**: Users can create custom dashboards to visualize the data collected by Langfuse. These dashboards can be tailored to highlight the most important metrics and insights for a specific application or team.\\n\\n7. **Alerting and Notifications**: It can set up alerts for specific conditions or errors, notifying developers when something goes wrong or when performance metrics fall outside of acceptable ranges.\\n\\nBy providing comprehensive observability for NLP and LLM applications, Langfuse helps developers to build more reliable, accurate, and user-friendly models and services.\", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1739554005, model='us.amazon.nova-micro-v1:0', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=380, prompt_tokens=5, total_tokens=385, completion_tokens_details=None, prompt_tokens_details=None))"
- ]
- },
- "execution_count": 24,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "import openai\n",
- "client = openai.OpenAI(\n",
- " api_key=LITELLM_VIRTUAL_KEY,\n",
- " base_url=LITELLM_PROXY_BASE_URL\n",
- ")\n",
- "\n",
- "response = client.chat.completions.create(\n",
- " model=\"us.amazon.nova-micro-v1:0\",\n",
- " messages = [\n",
- " {\n",
- " \"role\": \"user\",\n",
- " \"content\": \"what is Langfuse?\"\n",
- " }\n",
- " ],\n",
- ")\n",
- "\n",
- "response"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 3. Advanced - Set Langfuse Trace ID, Tags, Metadata \n",
- "\n",
- "Here is an example of how you can set Langfuse specific params on your client side request. See full list of supported langfuse params [here](https://docs.litellm.ai/docs/observability/langfuse_integration)\n",
- "\n",
- "You can view the logged trace of this request [here](https://us.cloud.langfuse.com/project/clvlhdfat0007vwb74m9lvfvi/traces/567890?timestamp=2025-02-14T17%3A30%3A26.709Z)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 27,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "ChatCompletion(id='chatcmpl-789babd5-c064-4939-9093-46e4cd2e208a', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=\"Langfuse is an observability platform designed specifically for monitoring and improving the performance of natural language processing (NLP) models and applications. It provides developers with tools to track, analyze, and optimize how their language models interact with users and handle natural language inputs.\\n\\nHere are some key features and benefits of Langfuse:\\n\\n1. **Real-Time Monitoring**: Langfuse allows developers to monitor their NLP applications in real time. This includes tracking user interactions, model responses, and overall performance metrics.\\n\\n2. **Error Tracking**: It helps in identifying and tracking errors in the model's responses. This can include incorrect, irrelevant, or unsafe outputs.\\n\\n3. **User Feedback Integration**: Langfuse enables the collection of user feedback directly within the platform. This feedback can be used to identify areas for improvement in the model's performance.\\n\\n4. **Performance Metrics**: The platform provides detailed metrics and analytics on model performance, including latency, throughput, and accuracy.\\n\\n5. **Alerts and Notifications**: Developers can set up alerts to notify them of any significant issues or anomalies in model performance.\\n\\n6. **Debugging Tools**: Langfuse offers tools to help developers debug and refine their models by providing insights into how the model processes different types of inputs.\\n\\n7. **Integration with Development Workflows**: It integrates seamlessly with various development environments and CI/CD pipelines, making it easier to incorporate observability into the development process.\\n\\n8. **Customizable Dashboards**: Users can create custom dashboards to visualize the data in a way that best suits their needs.\\n\\nLangfuse aims to help developers build more reliable, accurate, and user-friendly NLP applications by providing them with the tools to observe and improve how their models perform in real-world scenarios.\", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1739554281, model='us.amazon.nova-micro-v1:0', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=346, prompt_tokens=5, total_tokens=351, completion_tokens_details=None, prompt_tokens_details=None))"
- ]
- },
- "execution_count": 27,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "import openai\n",
- "client = openai.OpenAI(\n",
- " api_key=LITELLM_VIRTUAL_KEY,\n",
- " base_url=LITELLM_PROXY_BASE_URL\n",
- ")\n",
- "\n",
- "response = client.chat.completions.create(\n",
- " model=\"us.amazon.nova-micro-v1:0\",\n",
- " messages = [\n",
- " {\n",
- " \"role\": \"user\",\n",
- " \"content\": \"what is Langfuse?\"\n",
- " }\n",
- " ],\n",
- " extra_body={\n",
- " \"metadata\": {\n",
- " \"generation_id\": \"1234567890\",\n",
- " \"trace_id\": \"567890\",\n",
- " \"trace_user_id\": \"user_1234567890\",\n",
- " \"tags\": [\"tag1\", \"tag2\"]\n",
- " }\n",
- " }\n",
- ")\n",
- "\n",
- "response"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## "
- ]
- }
- ],
- "metadata": {
- "language_info": {
- "name": "python"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/cookbook/logging_observability/litellm_proxy_langfuse.png b/cookbook/logging_observability/litellm_proxy_langfuse.png
deleted file mode 100644
index 6b0691e6a5f..00000000000
Binary files a/cookbook/logging_observability/litellm_proxy_langfuse.png and /dev/null differ
diff --git a/cookbook/misc/RELEASE_NOTES_GENERATION_INSTRUCTIONS.md b/cookbook/misc/RELEASE_NOTES_GENERATION_INSTRUCTIONS.md
deleted file mode 100644
index 4a6fa9367fc..00000000000
--- a/cookbook/misc/RELEASE_NOTES_GENERATION_INSTRUCTIONS.md
+++ /dev/null
@@ -1,543 +0,0 @@
-# LiteLLM Release Notes Generation Instructions
-
-This document provides comprehensive instructions for AI agents to generate release notes for LiteLLM following the established format and style.
-
-## Required Inputs
-
-1. **Release Version** (e.g., `v1.77.3-stable`)
-2. **PR Diff/Changelog** - List of PRs with titles and contributors
-3. **Previous Version Commit Hash** - To compare model pricing changes
-4. **Reference Release Notes** - Use recent stable releases (v1.76.3-stable, v1.77.2-stable) as templates for consistent formatting
-
-### Resolving Staging PRs
-
-The GitHub release page (e.g. `https://github.com/BerriAI/litellm/releases/tag/v1.83.3-stable`) does **not** list the real changelog directly. The "What's Changed" section contains **staging PRs** that each bundle many individual commits/PRs. For example:
-
-- `Litellm oss staging 03 14 2026 by @RheagalFire in #23686`
-- `Litellm ryan march 16 by @ryan-crabbe in #23822`
-
-To get the real changelog, you MUST click into each staging PR (e.g. `#23686`, `#23822`), open its **Commits** tab, and extract every underlying commit/PR (look for the `(#NNNNN)` suffix on commit titles). Those underlying PRs ā not the staging PRs ā are what get categorized in the release notes. Never treat a staging PR title as a single changelog entry.
-
-**IMPORTANT ā staging PRs are not the complete source.** Some PRs land on the release branch *before* the staging PRs and are therefore not reachable via `gh api /pulls/