scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-07 02:59:05 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 103

Merge remote-tracking branch 'origin' into litellm_ui_config_add_sso

Browse source
This commit is contained in:
yuneng-jiang 2025-12-03 15:36:33 -08:00
commit 8a1cf104e0
121 changed files with 6121 additions and 1795 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.circleci/requirements.txt
View file

@ -13,7 +13,7 @@ google-cloud-aiplatform==1.43.0
google-cloud-iam==2.19.1
fastapi-sso==0.16.0
uvloop==0.21.0
mcp==1.10.1 # for MCP server
mcp==1.23.0 # for MCP server
semantic_router==0.1.10 # for auto-routing with litellm
fastuuid==0.12.0
responses==0.25.7 # for proxy client tests

15
Dockerfile
View file

@ -1,8 +1,8 @@
# Base image for building
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/python:latest-dev
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base
# Runtime image
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/python:latest-dev
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base
# Builder stage
FROM $LITELLM_BUILD_IMAGE AS builder
@ -12,11 +12,9 @@ WORKDIR /app
USER root
# Install build dependencies
RUN apk add --no-cache gcc python3-dev openssl openssl-dev
RUN apk add --no-cache bash gcc py3-pip python3 python3-dev openssl openssl-dev
RUN pip install --upgrade pip>=24.3.1 && \
pip install build
RUN python -m pip install build
# Copy the current directory contents into the container at /app
COPY . .
@ -48,10 +46,7 @@ FROM $LITELLM_RUNTIME_IMAGE AS runtime
USER root
# Install runtime dependencies
RUN apk add --no-cache openssl tzdata nodejs npm
# Upgrade pip to fix CVE-2025-8869
RUN pip install --upgrade pip>=24.3.1
RUN apk add --no-cache bash openssl tzdata nodejs npm python3 py3-pip
WORKDIR /app
# Copy the current directory contents into the container at /app

4
deploy/charts/litellm-helm/Chart.yaml
View file

@ -18,7 +18,7 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.4.8
version: 0.4.9
# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
@ -33,5 +33,5 @@ dependencies:
condition: db.deployStandalone
- name: redis
version: ">=18.0.0"
repository: oci://registry-1.docker.io/bitnamicharts
repository: oci://registry-1.docker.io/bitnamicharts
condition: redis.enabled

125
deploy/charts/litellm-helm/README.md
View file

@ -10,46 +10,48 @@
- Helm 3.8.0+
If `db.deployStandalone` is used:
- PV provisioner support in the underlying infrastructure
If `db.useStackgresOperator` is used (not yet implemented):
- The Stackgres Operator must already be installed in the Kubernetes Cluster. This chart will **not** install the operator if it is missing.
- The Stackgres Operator must already be installed in the Kubernetes Cluster. This chart will **not** install the operator if it is missing.
## Parameters
### LiteLLM Proxy Deployment Settings
| Name | Description | Value |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
| `replicaCount` | The number of LiteLLM Proxy pods to be deployed | `1` |
| `masterkeySecretName` | The name of the Kubernetes Secret that contains the Master API Key for LiteLLM. If not specified, use the generated secret name. | N/A |
| `masterkeySecretKey` | The key within the Kubernetes Secret that contains the Master API Key for LiteLLM. If not specified, use `masterkey` as the key. | N/A |
| `masterkey` | The Master API Key for LiteLLM. If not specified, a random key in the `sk-...` format is generated. | N/A |
| `environmentSecrets` | An optional array of Secret object names. The keys and values in these secrets will be presented to the LiteLLM proxy pod as environment variables. See below for an example Secret object. | `[]` |
| `environmentConfigMaps` | An optional array of ConfigMap object names. The keys and values in these configmaps will be presented to the LiteLLM proxy pod as environment variables. See below for an example Secret object. | `[]` |
| `image.repository` | LiteLLM Proxy image repository | `ghcr.io/berriai/litellm` |
| `image.pullPolicy` | LiteLLM Proxy image pull policy | `IfNotPresent` |
| `image.tag` | Overrides the image tag whose default the latest version of LiteLLM at the time this chart was published. | `""` |
| `imagePullSecrets` | Registry credentials for the LiteLLM and initContainer images. | `[]` |
| `serviceAccount.create` | Whether or not to create a Kubernetes Service Account for this deployment. The default is `false` because LiteLLM has no need to access the Kubernetes API. | `false` |
| `service.type` | Kubernetes Service type (e.g. `LoadBalancer`, `ClusterIP`, etc.) | `ClusterIP` |
| `service.port` | TCP port that the Kubernetes Service will listen on. Also the TCP port within the Pod that the proxy will listen on. | `4000` |
| `service.loadBalancerClass` | Optional LoadBalancer implementation class (only used when `service.type` is `LoadBalancer`) | `""` |
| `ingress.*` | See [values.yaml](./values.yaml) for example settings | N/A |
| `proxyConfigMap.create` | When `true`, render a ConfigMap from `.Values.proxy_config` and mount it. | `true` |
| `proxyConfigMap.name` | When `create=false`, name of the existing ConfigMap to mount. | `""` |
| `proxyConfigMap.key` | Key in the ConfigMap that contains the proxy config file. | `"config.yaml"` |
| `proxy_config.*` | See [values.yaml](./values.yaml) for default settings. Rendered into the ConfigMap’s `config.yaml` only when `proxyConfigMap.create=true`. See [example_config_yaml](../../../litellm/proxy/example_config_yaml/) for configuration examples. | `N/A` |
| `extraContainers[]` | An array of additional containers to be deployed as sidecars alongside the LiteLLM Proxy.
| `pdb.enabled` | Enable a PodDisruptionBudget for the LiteLLM proxy Deployment | `false` |
| `pdb.minAvailable` | Minimum number/percentage of pods that must be available during **voluntary** disruptions (choose **one** of minAvailable/maxUnavailable) | `null` |
| `pdb.maxUnavailable` | Maximum number/percentage of pods that can be unavailable during **voluntary** disruptions (choose **one** of minAvailable/maxUnavailable) | `null` |
| `pdb.annotations` | Extra metadata annotations to add to the PDB | `{}` |
| `pdb.labels` | Extra metadata labels to add to the PDB | `{}` |
| Name | Description | Value |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| `replicaCount` | The number of LiteLLM Proxy pods to be deployed | `1` |
| `masterkeySecretName` | The name of the Kubernetes Secret that contains the Master API Key for LiteLLM. If not specified, use the generated secret name. | N/A |
| `masterkeySecretKey` | The key within the Kubernetes Secret that contains the Master API Key for LiteLLM. If not specified, use `masterkey` as the key. | N/A |
| `masterkey` | The Master API Key for LiteLLM. If not specified, a random key in the `sk-...` format is generated. | N/A |
| `environmentSecrets` | An optional array of Secret object names. The keys and values in these secrets will be presented to the LiteLLM proxy pod as environment variables. See below for an example Secret object. | `[]` |
| `environmentConfigMaps` | An optional array of ConfigMap object names. The keys and values in these configmaps will be presented to the LiteLLM proxy pod as environment variables. See below for an example Secret object. | `[]` |
| `image.repository` | LiteLLM Proxy image repository | `ghcr.io/berriai/litellm` |
| `image.pullPolicy` | LiteLLM Proxy image pull policy | `IfNotPresent` |
| `image.tag` | Overrides the image tag whose default the latest version of LiteLLM at the time this chart was published. | `""` |
| `imagePullSecrets` | Registry credentials for the LiteLLM and initContainer images. | `[]` |
| `serviceAccount.create` | Whether or not to create a Kubernetes Service Account for this deployment. The default is `false` because LiteLLM has no need to access the Kubernetes API. | `false` |
| `service.type` | Kubernetes Service type (e.g. `LoadBalancer`, `ClusterIP`, etc.) | `ClusterIP` |
| `service.port` | TCP port that the Kubernetes Service will listen on. Also the TCP port within the Pod that the proxy will listen on. | `4000` |
| `service.loadBalancerClass` | Optional LoadBalancer implementation class (only used when `service.type` is `LoadBalancer`) | `""` |
| `ingress.labels` | Additional labels for the Ingress resource | `{}` |
| `ingress.*` | See [values.yaml](./values.yaml) for example settings | N/A |
| `proxyConfigMap.create` | When `true`, render a ConfigMap from `.Values.proxy_config` and mount it. | `true` |
| `proxyConfigMap.name` | When `create=false`, name of the existing ConfigMap to mount. | `""` |
| `proxyConfigMap.key` | Key in the ConfigMap that contains the proxy config file. | `"config.yaml"` |
| `proxy_config.*` | See [values.yaml](./values.yaml) for default settings. Rendered into the ConfigMap’s `config.yaml` only when `proxyConfigMap.create=true`. See [example_config_yaml](../../../litellm/proxy/example_config_yaml/) for configuration examples. | `N/A` |
| `extraContainers[]` | An array of additional containers to be deployed as sidecars alongside the LiteLLM Proxy. |
| `pdb.enabled` | Enable a PodDisruptionBudget for the LiteLLM proxy Deployment | `false` |
| `pdb.minAvailable` | Minimum number/percentage of pods that must be available during **voluntary** disruptions (choose **one** of minAvailable/maxUnavailable) | `null` |
| `pdb.maxUnavailable` | Maximum number/percentage of pods that can be unavailable during **voluntary** disruptions (choose **one** of minAvailable/maxUnavailable) | `null` |
| `pdb.annotations` | Extra metadata annotations to add to the PDB | `{}` |
| `pdb.labels` | Extra metadata labels to add to the PDB | `{}` |
#### Example `proxy_config` ConfigMap from values (default):
```
proxyConfigMap:
create: true
@ -67,7 +69,6 @@ proxy_config:
#### Example using existing `proxyConfigMap` instead of creating it:
```
proxyConfigMap:
create: false
@ -77,8 +78,7 @@ proxyConfigMap:
# proxy_config is ignored in this mode
```
#### Example `environmentSecrets` Secret
#### Example `environmentSecrets` Secret
```
apiVersion: v1
@ -91,21 +91,23 @@ type: Opaque
```
### Database Settings
| Name | Description | Value |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
| `db.useExisting` | Use an existing Postgres database. A Kubernetes Secret object must exist that contains credentials for connecting to the database. An example secret object definition is provided below. | `false` |
| `db.endpoint` | If `db.useExisting` is `true`, this is the IP, Hostname or Service Name of the Postgres server to connect to. | `localhost` |
| `db.database` | If `db.useExisting` is `true`, the name of the existing database to connect to. | `litellm` |
| `db.url` | If `db.useExisting` is `true`, the connection url of the existing database to connect to can be overwritten with this value. | `postgresql://$(DATABASE_USERNAME):$(DATABASE_PASSWORD)@$(DATABASE_HOST)/$(DATABASE_NAME)` |
| `db.secret.name` | If `db.useExisting` is `true`, the name of the Kubernetes Secret that contains credentials. | `postgres` |
| `db.secret.usernameKey` | If `db.useExisting` is `true`, the name of the key within the Kubernetes Secret that holds the username for authenticating with the Postgres instance. | `username` |
| `db.secret.passwordKey` | If `db.useExisting` is `true`, the name of the key within the Kubernetes Secret that holds the password associates with the above user. | `password` |
| `db.useStackgresOperator` | Not yet implemented. | `false` |
| `db.deployStandalone` | Deploy a standalone, single instance deployment of Postgres, using the Bitnami postgresql chart. This is useful for getting started but doesn't provide HA or (by default) data backups. | `true` |
| `postgresql.*` | If `db.deployStandalone` is `true`, configuration passed to the Bitnami postgresql chart. See the [Bitnami Documentation](https://github.com/bitnami/charts/tree/main/bitnami/postgresql) for full configuration details. See [values.yaml](./values.yaml) for the default configuration. | See [values.yaml](./values.yaml) |
| `postgresql.auth.*` | If `db.deployStandalone` is `true`, care should be taken to ensure the default `password` and `postgres-password` values are **NOT** used. | `NoTaGrEaTpAsSwOrD` |
| Name | Description | Value |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| `db.useExisting` | Use an existing Postgres database. A Kubernetes Secret object must exist that contains credentials for connecting to the database. An example secret object definition is provided below. | `false` |
| `db.endpoint` | If `db.useExisting` is `true`, this is the IP, Hostname or Service Name of the Postgres server to connect to. | `localhost` |
| `db.database` | If `db.useExisting` is `true`, the name of the existing database to connect to. | `litellm` |
| `db.url` | If `db.useExisting` is `true`, the connection url of the existing database to connect to can be overwritten with this value. | `postgresql://$(DATABASE_USERNAME):$(DATABASE_PASSWORD)@$(DATABASE_HOST)/$(DATABASE_NAME)` |
| `db.secret.name` | If `db.useExisting` is `true`, the name of the Kubernetes Secret that contains credentials. | `postgres` |
| `db.secret.usernameKey` | If `db.useExisting` is `true`, the name of the key within the Kubernetes Secret that holds the username for authenticating with the Postgres instance. | `username` |
| `db.secret.passwordKey` | If `db.useExisting` is `true`, the name of the key within the Kubernetes Secret that holds the password associates with the above user. | `password` |
| `db.useStackgresOperator` | Not yet implemented. | `false` |
| `db.deployStandalone` | Deploy a standalone, single instance deployment of Postgres, using the Bitnami postgresql chart. This is useful for getting started but doesn't provide HA or (by default) data backups. | `true` |
| `postgresql.*` | If `db.deployStandalone` is `true`, configuration passed to the Bitnami postgresql chart. See the [Bitnami Documentation](https://github.com/bitnami/charts/tree/main/bitnami/postgresql) for full configuration details. See [values.yaml](./values.yaml) for the default configuration. | See [values.yaml](./values.yaml) |
| `postgresql.auth.*` | If `db.deployStandalone` is `true`, care should be taken to ensure the default `password` and `postgres-password` values are **NOT** used. | `NoTaGrEaTpAsSwOrD` |
#### Example Postgres `db.useExisting` Secret
```yaml
apiVersion: v1
kind: Secret
@ -143,7 +145,7 @@ metadata:
name: litellm-env-secret
type: Opaque
data:
SOME_PASSWORD: cDZbUGVXeU5e0ZW # base64 encoded
SOME_PASSWORD: cDZbUGVXeU5e0ZW # base64 encoded
ANOTHER_PASSWORD: AAZbUGVXeU5e0ZB # base64 encoded
```
@ -153,23 +155,23 @@ Source: [GitHub Gist from troyharvey](https://gist.github.com/troyharvey/4506472
The migration job supports both ArgoCD and Helm hooks to ensure database migrations run at the appropriate time during deployments.
| Name | Description | Value |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
| `migrationJob.enabled` | Enable or disable the schema migration Job | `true` |
| `migrationJob.backoffLimit` | Backoff limit for Job restarts | `4` |
| `migrationJob.ttlSecondsAfterFinished` | TTL for completed migration jobs | `120` |
| `migrationJob.annotations` | Additional annotations for the migration job pod | `{}` |
| `migrationJob.extraContainers` | Additional containers to run alongside the migration job | `[]` |
| `migrationJob.hooks.argocd.enabled` | Enable ArgoCD hooks for the migration job (uses PreSync hook with BeforeHookCreation delete policy) | `true` |
| `migrationJob.hooks.helm.enabled` | Enable Helm hooks for the migration job (uses pre-install,pre-upgrade hooks with before-hook-creation delete policy) | `false` |
| `migrationJob.hooks.helm.weight` | Helm hook execution order (lower weights executed first). Optional - defaults to "1" if not specified. | N/A |
| Name | Description | Value |
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ------- |
| `migrationJob.enabled` | Enable or disable the schema migration Job | `true` |
| `migrationJob.backoffLimit` | Backoff limit for Job restarts | `4` |
| `migrationJob.ttlSecondsAfterFinished` | TTL for completed migration jobs | `120` |
| `migrationJob.annotations` | Additional annotations for the migration job pod | `{}` |
| `migrationJob.extraContainers` | Additional containers to run alongside the migration job | `[]` |
| `migrationJob.hooks.argocd.enabled` | Enable ArgoCD hooks for the migration job (uses PreSync hook with BeforeHookCreation delete policy) | `true` |
| `migrationJob.hooks.helm.enabled` | Enable Helm hooks for the migration job (uses pre-install,pre-upgrade hooks with before-hook-creation delete policy) | `false` |
| `migrationJob.hooks.helm.weight` | Helm hook execution order (lower weights executed first). Optional - defaults to "1" if not specified. | N/A |
## Accessing the Admin UI
When browsing to the URL published per the settings in `ingress.*`, you will
be prompted for **Admin Configuration**. The **Proxy Endpoint** is the internal
be prompted for **Admin Configuration**. The **Proxy Endpoint** is the internal
(from the `litellm` pod's perspective) URL published by the `<RELEASE>-litellm`
Kubernetes Service. If the deployment uses the default settings for this
Kubernetes Service. If the deployment uses the default settings for this
service, the **Proxy Endpoint** should be set to `http://<RELEASE>-litellm:4000`.
The **Proxy Key** is the value specified for `masterkey` or, if a `masterkey`
@ -181,7 +183,8 @@ kubectl -n litellm get secret <RELEASE>-litellm-masterkey -o jsonpath="{.data.ma
```
## Admin UI Limitations
At the time of writing, the Admin UI is unable to add models. This is because
At the time of writing, the Admin UI is unable to add models. This is because
it would need to update the `config.yaml` file which is a exposed ConfigMap, and
therefore, read-only. This is a limitation of this helm chart, not the Admin UI
therefore, read-only. This is a limitation of this helm chart, not the Admin UI
itself.

3
deploy/charts/litellm-helm/templates/ingress.yaml
View file

@ -18,6 +18,9 @@ metadata:
name: {{ $fullName }}
labels:
{{- include "litellm.labels" . | nindent 4 }}
{{- with .Values.ingress.labels }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}

45
deploy/charts/litellm-helm/tests/ingress_tests.yaml Normal file
View file

@ -0,0 +1,45 @@
suite: Ingress Configuration Tests
templates:
- ingress.yaml
tests:
- it: should not create Ingress by default
asserts:
- hasDocuments:
count: 0
- it: should create Ingress when enabled
set:
ingress.enabled: true
asserts:
- hasDocuments:
count: 1
- isKind:
of: Ingress
- it: should add custom labels
set:
ingress.enabled: true
ingress.labels:
custom-label: "true"
another-label: "value"
asserts:
- isKind:
of: Ingress
- equal:
path: metadata.labels.custom-label
value: "true"
- equal:
path: metadata.labels.another-label
value: "value"
- it: should add annotations
set:
ingress.enabled: true
ingress.annotations:
kubernetes.io/ingress.class: "nginx"
asserts:
- isKind:
of: Ingress
- equal:
path: metadata.annotations["kubernetes.io/ingress.class"]
value: "nginx"

45
deploy/charts/litellm-helm/values.yaml
View file

@ -35,7 +35,8 @@ podAnnotations: {}
podLabels: {}
terminationGracePeriodSeconds: 90
topologySpreadConstraints: []
topologySpreadConstraints:
[]
# - maxSkew: 1
# topologyKey: kubernetes.io/hostname
# whenUnsatisfiable: DoNotSchedule
@ -46,7 +47,8 @@ topologySpreadConstraints: []
# At the time of writing, the litellm docker image requires write access to the
# filesystem on startup so that prisma can install some dependencies.
podSecurityContext: {}
securityContext: {}
securityContext:
{}
# capabilities:
# drop:
# - ALL
@ -57,13 +59,15 @@ securityContext: {}
# A list of Kubernetes Secret objects that will be exported to the LiteLLM proxy
# pod as environment variables. These secrets can then be referenced in the
# configuration file (or "litellm" ConfigMap) with `os.environ/<Env Var Name>`
environmentSecrets: []
environmentSecrets:
[]
# - litellm-env-secret
# A list of Kubernetes ConfigMap objects that will be exported to the LiteLLM proxy
# pod as environment variables. The ConfigMap kv-pairs can then be referenced in the
# configuration file (or "litellm" ConfigMap) with `os.environ/<Env Var Name>`
environmentConfigMaps: []
environmentConfigMaps:
[]
# - litellm-env-configmap
service:
@ -82,7 +86,9 @@ separateHealthPort: 8081
ingress:
enabled: false
className: "nginx"
annotations: {}
labels: {}
annotations:
{}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts:
@ -129,7 +135,8 @@ proxy_config:
general_settings:
master_key: os.environ/PROXY_MASTER_KEY
resources: {}
resources:
{}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
@ -231,7 +238,7 @@ migrationJob:
# cpu: 100m
# memory: 100Mi
extraContainers: []
# Hook configuration
hooks:
argocd:
@ -240,30 +247,30 @@ migrationJob:
enabled: false
# Additional environment variables to be added to the deployment as a map of key-value pairs
envVars: {
# USE_DDTRACE: "true"
}
envVars: {}
# USE_DDTRACE: "true"
# Additional environment variables to be added to the deployment as a list of k8s env vars
extraEnvVars: {
# - name: EXTRA_ENV_VAR
# value: EXTRA_ENV_VAR_VALUE
}
extraEnvVars: {}
# - name: EXTRA_ENV_VAR
# value: EXTRA_ENV_VAR_VALUE
# Pod Disruption Budget
pdb:
enabled: false
# Set exactly one of the following. If both are set, minAvailable takes precedence.
minAvailable: null # e.g. "50%" or 1
maxUnavailable: null # e.g. 1 or "20%"
minAvailable: null # e.g. "50%" or 1
maxUnavailable: null # e.g. 1 or "20%"
annotations: {}
labels: {}
serviceMonitor:
enabled: false
labels: {}
labels:
{}
# test: test
annotations: {}
annotations:
{}
# kubernetes.io/test: test
interval: 15s
scrapeTimeout: 10s
@ -273,4 +280,4 @@ serviceMonitor:
# action: replace
namespaceSelector:
matchNames: []
# - test-namespace
# - test-namespace

16
docker/Dockerfile.database
View file

@ -1,8 +1,8 @@
# Base image for building
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/python:latest-dev
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base
# Runtime image
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/python:latest-dev
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base
# Builder stage
FROM $LITELLM_BUILD_IMAGE AS builder
@ -13,13 +13,15 @@ USER root
# Install build dependencies
RUN apk add --no-cache \
build-base \
bash \
gcc \
py3-pip \
python3 \
python3-dev \
openssl \
openssl-dev
RUN pip install --upgrade pip && \
pip install build
RUN python -m pip install build
# Copy the current directory contents into the container at /app
COPY . .
@ -46,7 +48,7 @@ FROM $LITELLM_RUNTIME_IMAGE AS runtime
USER root
# Install runtime dependencies
RUN apk add --no-cache openssl
RUN apk add --no-cache bash openssl tzdata nodejs npm python3 py3-pip
WORKDIR /app
# Copy the current directory contents into the container at /app

8
docker/Dockerfile.non_root
View file

@ -1,6 +1,6 @@
# Base images
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/python:latest-dev
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/python:latest-dev
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base
# -----------------
# Builder Stage
@ -11,6 +11,8 @@ WORKDIR /app
# Install build dependencies including Node.js for UI build
USER root
RUN apk add --no-cache \
python3 \
py3-pip \
clang \
llvm \
lld \
@ -71,7 +73,7 @@ WORKDIR /app
# Install runtime dependencies
USER root
RUN apk upgrade --no-cache && \
apk add --no-cache bash libstdc++ ca-certificates openssl supervisor
apk add --no-cache python3 py3-pip bash openssl tzdata nodejs npm supervisor
# Copy only necessary artifacts from builder stage for runtime
COPY . .

1
docs/my-website/docs/completion/knowledgebase.md
View file

@ -21,6 +21,7 @@ LiteLLM integrates with vector stores, allowing your models to access your organ
- [Azure Vector Stores](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/file-search?tabs=python#vector-stores) (Cannot be directly queried. Only available for calling in Assistants messages. We will be adding Azure AI Search Vector Store API support soon.)
- [Vertex AI RAG API](https://cloud.google.com/vertex-ai/generative-ai/docs/rag-overview)
- [Gemini File Search](https://ai.google.dev/gemini-api/docs/file-search)
- [RAGFlow Datasets](/docs/providers/ragflow_vector_store.md) (Dataset management only, search not supported)
## Quick Start

108
docs/my-website/docs/getting_started.md
View file

@ -1,108 +0,0 @@
# Getting Started
import QuickStart from '../src/components/QuickStart.js'
LiteLLM simplifies LLM API calls by mapping them all to the [OpenAI ChatCompletion format](https://platform.openai.com/docs/api-reference/chat).
## basic usage
By default we provide a free $10 community-key to try all providers supported on LiteLLM.
```python
from litellm import completion
## set ENV variables
os.environ["OPENAI_API_KEY"] = "your-api-key"
os.environ["COHERE_API_KEY"] = "your-api-key"
messages = [{ "content": "Hello, how are you?","role": "user"}]
# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)
# cohere call
response = completion("command-nightly", messages)
```
**Need a dedicated key?**
Email us @ krrish@berri.ai
Next Steps 👉 [Call all supported models - e.g. Claude-2, Llama2-70b, etc.](./proxy_api.md#supported-models)
More details 👉
- [Completion() function details](./completion/)
- [Overview of supported models / providers on LiteLLM](./providers/)
- [Search all models / providers](https://models.litellm.ai/)
- [Build your own OpenAI proxy](https://github.com/BerriAI/liteLLM-proxy/tree/main)
## streaming
Same example from before. Just pass in `stream=True` in the completion args.
```python
from litellm import completion
## set ENV variables
os.environ["OPENAI_API_KEY"] = "openai key"
os.environ["COHERE_API_KEY"] = "cohere key"
messages = [{ "content": "Hello, how are you?","role": "user"}]
# openai call
response = completion(model="gpt-3.5-turbo", messages=messages, stream=True)
# cohere call
response = completion("command-nightly", messages, stream=True)
print(response)
```
More details 👉
- [streaming + async](./completion/stream.md)
- [tutorial for streaming Llama2 on TogetherAI](./tutorials/TogetherAI_liteLLM.md)
## exception handling
LiteLLM maps exceptions across all supported providers to the OpenAI exceptions. All our exceptions inherit from OpenAI's exception types, so any error-handling you have for that, should work out of the box with LiteLLM.
```python
from openai.error import OpenAIError
from litellm import completion
os.environ["ANTHROPIC_API_KEY"] = "bad-key"
try:
# some code
completion(model="claude-instant-1", messages=[{"role": "user", "content": "Hey, how's it going?"}])
except OpenAIError as e:
print(e)
```
## Logging Observability - Log LLM Input/Output ([Docs](https://docs.litellm.ai/docs/observability/callbacks))
LiteLLM exposes pre defined callbacks to send data to MLflow, Lunary, Langfuse, Helicone, Promptlayer, Traceloop, Slack
```python
from litellm import completion
## set env variables for logging tools (API key set up is not required when using MLflow)
os.environ["LUNARY_PUBLIC_KEY"] = "your-lunary-public-key" # get your public key at https://app.lunary.ai/settings
os.environ["HELICONE_API_KEY"] = "your-helicone-key"
os.environ["LANGFUSE_PUBLIC_KEY"] = ""
os.environ["LANGFUSE_SECRET_KEY"] = ""
os.environ["OPENAI_API_KEY"]
# set callbacks
litellm.success_callback = ["lunary", "mlflow", "langfuse", "helicone"] # log input/output to MLflow, langfuse, lunary, helicone
#openai call
response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hi 👋 - i'm openai"}])
```
More details 👉
- [exception mapping](./exception_mapping.md)
- [retries + model fallbacks for completion()](./completion/reliable_completions.md)
- [tutorial for model fallbacks with completion()](./tutorials/fallbacks.md)

18
docs/my-website/docs/observability/datadog.md
View file

@ -71,17 +71,19 @@ DD_SOURCE="litellm_dev" # [OPTIONAL] your datadog source. use to different
Send logs through a local DataDog agent (useful for containerized environments):
```shell
DD_AGENT_HOST="localhost" # hostname or IP of DataDog agent
DD_AGENT_PORT="10518" # [OPTIONAL] port of DataDog agent (default: 10518)
DD_API_KEY="5f2d0f310***********" # [OPTIONAL] your datadog API Key (agent handles auth)
DD_SOURCE="litellm_dev" # [OPTIONAL] your datadog source
LITELLM_DD_AGENT_HOST="localhost" # hostname or IP of DataDog agent
LITELLM_DD_AGENT_PORT="10518" # [OPTIONAL] port of DataDog agent (default: 10518)
DD_API_KEY="5f2d0f310***********" # [OPTIONAL] your datadog API Key (agent handles auth)
DD_SOURCE="litellm_dev" # [OPTIONAL] your datadog source
```
When `DD_AGENT_HOST` is set, logs are sent to the agent instead of directly to DataDog API. This is useful for:
When `LITELLM_DD_AGENT_HOST` is set, logs are sent to the agent instead of directly to DataDog API. This is useful for:
- Centralized log shipping in containerized environments
- Reducing direct API calls from multiple services
- Leveraging agent-side processing and filtering
**Note:** We use `LITELLM_DD_AGENT_HOST` instead of `DD_AGENT_HOST` to avoid conflicts with `ddtrace` which automatically sets `DD_AGENT_HOST` for APM tracing.
**Step 3**: Start the proxy, make a test request
Start proxy
@ -191,8 +193,8 @@ LiteLLM supports customizing the following Datadog environment variables
|---------------------|-------------|---------------|----------|
| `DD_API_KEY` | Your Datadog API key for authentication (required for direct API, optional for agent) | None | Conditional* |
| `DD_SITE` | Your Datadog site (e.g., "us5.datadoghq.com") (required for direct API) | None | Conditional* |
| `DD_AGENT_HOST` | Hostname or IP of DataDog agent (e.g., "localhost"). When set, logs are sent to agent instead of direct API | None | ❌ No |
| `DD_AGENT_PORT` | Port of DataDog agent for log intake | "10518" | ❌ No |
| `LITELLM_DD_AGENT_HOST` | Hostname or IP of DataDog agent (e.g., "localhost"). When set, logs are sent to agent instead of direct API | None | ❌ No |
| `LITELLM_DD_AGENT_PORT` | Port of DataDog agent for log intake | "10518" | ❌ No |
| `DD_ENV` | Environment tag for your logs (e.g., "production", "staging") | "unknown" | ❌ No |
| `DD_SERVICE` | Service name for your logs | "litellm-server" | ❌ No |
| `DD_SOURCE` | Source name for your logs | "litellm" | ❌ No |
@ -201,5 +203,5 @@ LiteLLM supports customizing the following Datadog environment variables
| `POD_NAME` | Pod name tag (useful for Kubernetes deployments) | "unknown" | ❌ No |
\* **Required when using Direct API** (default): `DD_API_KEY` and `DD_SITE` are required
\* **Optional when using DataDog Agent**: Set `DD_AGENT_HOST` to use agent mode; `DD_API_KEY` and `DD_SITE` are not required
\* **Optional when using DataDog Agent**: Set `LITELLM_DD_AGENT_HOST` to use agent mode; `DD_API_KEY` and `DD_SITE` are not required

79
docs/my-website/docs/observability/phoenix_integration.md
View file

@ -6,7 +6,7 @@ Open source tracing and evaluation platform
:::tip
This is community maintained, Please make an issue if you run into a bug
This is community maintained. Please make an issue if you run into a bug:
https://github.com/BerriAI/litellm
:::
@ -31,19 +31,16 @@ litellm.callbacks = ["arize_phoenix"]
import litellm
import os
os.environ["PHOENIX_API_KEY"] = "" # Necessary only using Phoenix Cloud
os.environ["PHOENIX_COLLECTOR_HTTP_ENDPOINT"] = "" # The URL of your Phoenix OSS instance e.g. http://localhost:6006/v1/traces
os.environ["PHOENIX_PROJECT_NAME"]="litellm" # OPTIONAL: you can configure project names, otherwise traces would go to "default" project
# Set env variables
os.environ["PHOENIX_API_KEY"] = "d0*****" # Set the Phoenix API key here. It is necessary only when using Phoenix Cloud.
os.environ["PHOENIX_COLLECTOR_HTTP_ENDPOINT"] = "https://app.phoenix.arize.com/s/<space-name>/v1/traces" # Set the URL of your Phoenix OSS instance, otherwise tracer would use https://app.phoenix.arize.com/v1/traces for Phoenix Cloud.
os.environ["PHOENIX_PROJECT_NAME"] = "litellm" # Configure the project name, otherwise traces would go to "default" project.
os.environ['OPENAI_API_KEY'] = "fake-key" # Set the OpenAI API key here.
# This defaults to https://app.phoenix.arize.com/v1/traces for Phoenix Cloud
# LLM API Keys
os.environ['OPENAI_API_KEY']=""
# set arize as a callback, litellm will send the data to arize
# Set arize_phoenix as a callback & LiteLLM will send the data to Phoenix.
litellm.callbacks = ["arize_phoenix"]
# openai call
# OpenAI call
response = litellm.completion(
model="gpt-3.5-turbo",
messages=[
@ -52,8 +49,9 @@ response = litellm.completion(
)
```
### Using with LiteLLM Proxy
## Using with LiteLLM Proxy
1. Setup config.yaml
```yaml
model_list:
@ -66,12 +64,63 @@ model_list:
litellm_settings:
callbacks: ["arize_phoenix"]
general_settings:
master_key: "sk-1234"
environment_variables:
PHOENIX_API_KEY: "d0*****"
PHOENIX_COLLECTOR_ENDPOINT: "https://app.phoenix.arize.com/v1/traces" # OPTIONAL, for setting the GRPC endpoint
PHOENIX_COLLECTOR_HTTP_ENDPOINT: "https://app.phoenix.arize.com/v1/traces" # OPTIONAL, for setting the HTTP endpoint
PHOENIX_COLLECTOR_ENDPOINT: "https://app.phoenix.arize.com/s/<space-name>/v1/traces" # OPTIONAL - For setting the gRPC endpoint
PHOENIX_COLLECTOR_HTTP_ENDPOINT: "https://app.phoenix.arize.com/s/<space-name>/v1/traces" # OPTIONAL - For setting the HTTP endpoint
```
2. Start the proxy
```bash
litellm --config config.yaml
```
3. Test it!
```bash
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Hi 👋 - i'm openai"}]}'
```
## Supported Phoenix Endpoints
Phoenix now supports multiple deployment types. The correct endpoint depends on which version of Phoenix Cloud you are using.
**Phoenix Cloud (With Spaces - New Version)**
Use this if your Phoenix URL contains `/s/<space-name>` path.
```bash
https://app.phoenix.arize.com/s/<space-name>/v1/traces
```
**Phoenix Cloud (Legacy - Deprecated)**
Use this only if your deployment still shows the `/legacy` pattern.
```bash
https://app.phoenix.arize.com/legacy/v1/traces
```
**Phoenix Cloud (Without Spaces - Old Version)**
Use this if your Phoenix Cloud URL does not contain `/s/<space-name>` or `/legacy` path.
```bash
https://app.phoenix.arize.com/v1/traces
```
**Self-Hosted Phoenix (Local Instance)**
Use this when running Phoenix on your machine or a private server.
```bash
http://localhost:6006/v1/traces
```
Depending on which Phoenix Cloud version or deployment you are using, you should set the corresponding endpoint in `PHOENIX_COLLECTOR_HTTP_ENDPOINT` or `PHOENIX_COLLECTOR_ENDPOINT`.
## Support & Talk to Founders
- [Schedule Demo 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)

10
docs/my-website/docs/projects/Agent Lightning.md Normal file
View file

@ -0,0 +1,10 @@
# Agent Lightning
[Agent Lightning](https://github.com/microsoft/agent-lightning) is Microsoft's open-source framework for training and optimizing AI agents with Reinforcement Learning, Automatic Prompt Optimization, and Supervised Fine-tuning — with almost zero code changes.
It works with any agent framework including LangChain, OpenAI Agents SDK, AutoGen, and CrewAI. Agent Lightning uses LiteLLM Proxy under the hood to route LLM requests and collect traces that power its training algorithms.
- [GitHub](https://github.com/microsoft/agent-lightning)
- [Docs](https://microsoft.github.io/agent-lightning/)
- [arXiv Paper](https://arxiv.org/abs/2508.03680)

21
docs/my-website/docs/projects/Google ADK.md Normal file
View file

@ -0,0 +1,21 @@
# Google ADK (Agent Development Kit)
[Google ADK](https://github.com/google/adk-python) is an open-source, code-first Python framework for building, evaluating, and deploying sophisticated AI agents. While optimized for Gemini, ADK is model-agnostic and supports LiteLLM for using 100+ providers.
```python
from google.adk.agents.llm_agent import Agent
from google.adk.models.lite_llm import LiteLlm
root_agent = Agent(
model=LiteLlm(model="openai/gpt-4o"), # Or any LiteLLM-supported model
name="my_agent",
description="An agent using LiteLLM",
instruction="You are a helpful assistant.",
tools=[your_tools],
)
```
- [GitHub](https://github.com/google/adk-python)
- [Documentation](https://google.github.io/adk-docs)
- [LiteLLM Samples](https://github.com/google/adk-python/tree/main/contributing/samples/hello_world_litellm)

24
docs/my-website/docs/projects/Harbor.md Normal file
View file

@ -0,0 +1,24 @@
# Harbor
[Harbor](https://github.com/laude-institute/harbor) is a framework from the creators of Terminal-Bench for evaluating and optimizing agents and language models. It uses LiteLLM to call 100+ LLM providers.
```bash
# Install
pip install harbor
# Run a benchmark with any LiteLLM-supported model
harbor run --dataset terminal-bench@2.0 \
--agent claude-code \
--model anthropic/claude-opus-4-1 \
--n-concurrent 4
```
Key features:
- Evaluate agents like Claude Code, OpenHands, Codex CLI
- Build and share benchmarks and environments
- Run experiments in parallel across cloud providers (Daytona, Modal)
- Generate rollouts for RL optimization
- [GitHub](https://github.com/laude-institute/harbor)
- [Documentation](https://harborframework.com/docs)

14
docs/my-website/docs/providers/bedrock.md
View file

@ -43,6 +43,8 @@ export AWS_BEARER_TOKEN_BEDROCK="your-api-key"
Option 2: use the api_key parameter to pass in API key for completion, embedding, image_generation API calls.
<Tabs>
<TabItem value="sdk" label="SDK">
```python
response = completion(
model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
@ -50,7 +52,17 @@ response = completion(
api_key="your-api-key"
)
```
</TabItem>
<TabItem value="proxy" label="PROXY">
```yaml
model_list:
- model_name: bedrock-claude-3-sonnet
litellm_params:
model: bedrock/anthropic.claude-3-sonnet-20240229-v1:0
api_key: os.environ/AWS_BEARER_TOKEN_BEDROCK
```
</TabItem>
</Tabs>
## Usage

244
docs/my-website/docs/providers/ragflow.md Normal file
View file

@ -0,0 +1,244 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# RAGFlow
Litellm supports Ragflow's chat completions APIs
## Supported Features
- ✅ Chat completions
- ✅ Streaming responses
- ✅ Both chat and agent endpoints
- ✅ Multiple credential sources (params, env vars, litellm_params)
- ✅ OpenAI-compatible API format
## API Key
```python
# env variable
os.environ['RAGFLOW_API_KEY']
```
## API Base
```python
# env variable
os.environ['RAGFLOW_API_BASE']
```
## Overview
RAGFlow provides OpenAI-compatible APIs with unique path structures that include chat and agent IDs:
- **Chat endpoint**: `/api/v1/chats_openai/{chat_id}/chat/completions`
- **Agent endpoint**: `/api/v1/agents_openai/{agent_id}/chat/completions`
The model name format embeds the endpoint type and ID:
- Chat: `ragflow/chat/{chat_id}/{model_name}`
- Agent: `ragflow/agent/{agent_id}/{model_name}`
## Sample Usage - Chat Endpoint
```python
from litellm import completion
import os
os.environ['RAGFLOW_API_KEY'] = "your-ragflow-api-key"
os.environ['RAGFLOW_API_BASE'] = "http://localhost:9380" # or your hosted URL
response = completion(
model="ragflow/chat/my-chat-id/gpt-4o-mini",
messages=[{"role": "user", "content": "How does the deep doc understanding work?"}]
)
print(response)
```
## Sample Usage - Agent Endpoint
```python
from litellm import completion
import os
os.environ['RAGFLOW_API_KEY'] = "your-ragflow-api-key"
os.environ['RAGFLOW_API_BASE'] = "http://localhost:9380" # or your hosted URL
response = completion(
model="ragflow/agent/my-agent-id/gpt-4o-mini",
messages=[{"role": "user", "content": "What are the key features?"}]
)
print(response)
```
## Sample Usage - With Parameters
You can also pass `api_key` and `api_base` directly as parameters:
```python
from litellm import completion
response = completion(
model="ragflow/chat/my-chat-id/gpt-4o-mini",
messages=[{"role": "user", "content": "Hello!"}],
api_key="your-ragflow-api-key",
api_base="http://localhost:9380"
)
print(response)
```
## Sample Usage - Streaming
```python
from litellm import completion
import os
os.environ['RAGFLOW_API_KEY'] = "your-ragflow-api-key"
os.environ['RAGFLOW_API_BASE'] = "http://localhost:9380"
response = completion(
model="ragflow/agent/my-agent-id/gpt-4o-mini",
messages=[{"role": "user", "content": "Explain RAGFlow"}],
stream=True
)
for chunk in response:
print(chunk)
```
## Model Name Format
The model name must follow one of these formats:
### Chat Endpoint
```
ragflow/chat/{chat_id}/{model_name}
```
Example: `ragflow/chat/my-chat-id/gpt-4o-mini`
### Agent Endpoint
```
ragflow/agent/{agent_id}/{model_name}
```
Example: `ragflow/agent/my-agent-id/gpt-4o-mini`
Where:
- `{chat_id}` or `{agent_id}` is the ID of your chat or agent in RAGFlow
- `{model_name}` is the actual model name (e.g., `gpt-4o-mini`, `gpt-4o`, etc.)
## Configuration Sources
LiteLLM supports multiple ways to provide credentials, checked in this order:
1. **Function parameters**: `api_key="..."`, `api_base="..."`
2. **litellm_params**: `litellm_params={"api_key": "...", "api_base": "..."}`
3. **Environment variables**: `RAGFLOW_API_KEY`, `RAGFLOW_API_BASE`
4. **Global litellm settings**: `litellm.api_key`, `litellm.api_base`
## Usage - LiteLLM Proxy Server
### 1. Save key in your environment
```bash
export RAGFLOW_API_KEY="your-ragflow-api-key"
export RAGFLOW_API_BASE="http://localhost:9380"
```
### 2. Start the proxy
<Tabs>
<TabItem value="config" label="config.yaml">
```yaml
model_list:
- model_name: ragflow-chat-gpt4
litellm_params:
model: ragflow/chat/my-chat-id/gpt-4o-mini
api_key: os.environ/RAGFLOW_API_KEY
api_base: os.environ/RAGFLOW_API_BASE
- model_name: ragflow-agent-gpt4
litellm_params:
model: ragflow/agent/my-agent-id/gpt-4o-mini
api_key: os.environ/RAGFLOW_API_KEY
api_base: os.environ/RAGFLOW_API_BASE
```
</TabItem>
<TabItem value="cli" label="CLI">
```bash
$ litellm --config /path/to/config.yaml
# Server running on http://0.0.0.0:4000
```
</TabItem>
</Tabs>
### 3. Test it
<Tabs>
<TabItem value="Curl" label="Curl Request">
```bash
curl http://0.0.0.0:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "ragflow-chat-gpt4",
"messages": [
{"role": "user", "content": "How does RAGFlow work?"}
]
}'
```
</TabItem>
<TabItem value="Python" label="Python SDK">
```python
from openai import OpenAI
client = OpenAI(
api_key="sk-1234", # Your LiteLLM proxy key
base_url="http://0.0.0.0:4000"
)
response = client.chat.completions.create(
model="ragflow-chat-gpt4",
messages=[
{"role": "user", "content": "How does RAGFlow work?"}
]
)
print(response)
```
</TabItem>
</Tabs>
## API Base URL Handling
The `api_base` parameter can be provided with or without `/v1` suffix. LiteLLM will automatically handle it:
- `http://localhost:9380` → `http://localhost:9380/api/v1/chats_openai/{chat_id}/chat/completions`
- `http://localhost:9380/v1` → `http://localhost:9380/api/v1/chats_openai/{chat_id}/chat/completions`
- `http://localhost:9380/api/v1` → `http://localhost:9380/api/v1/chats_openai/{chat_id}/chat/completions`
All three formats will work correctly.
## Error Handling
If you encounter errors:
1. **Invalid model format**: Ensure your model name follows `ragflow/{chat|agent}/{id}/{model_name}` format
2. **Missing api_base**: Provide `api_base` via parameter, environment variable, or litellm_params
3. **Connection errors**: Verify your RAGFlow server is running and accessible at the provided `api_base`
:::info
For more information about passing provider-specific parameters, [go here](../completion/provider_specific_params.md)
:::

349
docs/my-website/docs/providers/ragflow_vector_store.md Normal file
View file

@ -0,0 +1,349 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import Image from '@theme/IdealImage';
# RAGFlow Vector Stores
Litellm support creation and management of datasets for document processing and knowledge base management in Ragflow.
| Property | Details |
|----------|---------|
| Description | RAGFlow datasets enable document processing, chunking, and knowledge base management for RAG applications. |
| Provider Route on LiteLLM | `ragflow` in the litellm vector_store_registry |
| Provider Doc | [RAGFlow API Documentation ↗](https://ragflow.io/docs) |
| Supported Operations | Dataset Management (Create, List, Update, Delete) |
| Search/Retrieval | ❌ Not supported (management only) |
## Quick Start
### LiteLLM Python SDK
```python showLineNumbers title="Example using LiteLLM Python SDK"
import os
import litellm
# Set RAGFlow credentials
os.environ["RAGFLOW_API_KEY"] = "your-ragflow-api-key"
os.environ["RAGFLOW_API_BASE"] = "http://localhost:9380" # Optional, defaults to localhost:9380
# Create a RAGFlow dataset
response = litellm.vector_stores.create(
name="my-dataset",
custom_llm_provider="ragflow",
metadata={
"description": "My knowledge base dataset",
"embedding_model": "BAAI/bge-large-zh-v1.5@BAAI",
"chunk_method": "naive"
}
)
print(f"Created dataset ID: {response.id}")
print(f"Dataset name: {response.name}")
```
### LiteLLM Proxy
#### 1. Configure your vector_store_registry
<Tabs>
<TabItem value="config-yaml" label="config.yaml">
```yaml
model_list:
- model_name: gpt-4o-mini
litellm_params:
model: gpt-4o-mini
api_key: os.environ/OPENAI_API_KEY
vector_store_registry:
- vector_store_name: "ragflow-knowledge-base"
litellm_params:
vector_store_id: "your-dataset-id"
custom_llm_provider: "ragflow"
api_key: os.environ/RAGFLOW_API_KEY
api_base: os.environ/RAGFLOW_API_BASE # Optional
vector_store_description: "RAGFlow dataset for knowledge base"
vector_store_metadata:
source: "Company documentation"
```
</TabItem>
<TabItem value="litellm-ui" label="LiteLLM UI">
On the LiteLLM UI, Navigate to Experimental > Vector Stores > Create Vector Store. On this page you can create a vector store with a name, vector store id and credentials.
<Image
img={require('../../img/kb_2.png')}
style={{width: '50%'}}
/>
</TabItem>
</Tabs>
#### 2. Create a dataset via Proxy
<Tabs>
<TabItem value="curl" label="Curl">
```bash
curl http://localhost:4000/v1/vector_stores \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-d '{
"name": "my-ragflow-dataset",
"custom_llm_provider": "ragflow",
"metadata": {
"description": "Test dataset",
"chunk_method": "naive"
}
}'
```
</TabItem>
<TabItem value="openai-sdk" label="OpenAI Python SDK">
```python
from openai import OpenAI
# Initialize client with your LiteLLM proxy URL
client = OpenAI(
base_url="http://localhost:4000",
api_key="your-litellm-api-key"
)
# Create a RAGFlow dataset
response = client.vector_stores.create(
name="my-ragflow-dataset",
custom_llm_provider="ragflow",
metadata={
"description": "Test dataset",
"chunk_method": "naive"
}
)
print(f"Created dataset: {response.id}")
```
</TabItem>
</Tabs>
## Configuration
### Environment Variables
RAGFlow vector stores support configuration via environment variables:
- `RAGFLOW_API_KEY` - Your RAGFlow API key (required)
- `RAGFLOW_API_BASE` - RAGFlow API base URL (optional, defaults to `http://localhost:9380`)
### Parameters
You can also pass these via `litellm_params`:
- `api_key` - RAGFlow API key (overrides `RAGFLOW_API_KEY` env var)
- `api_base` - RAGFlow API base URL (overrides `RAGFLOW_API_BASE` env var)
## Dataset Creation Options
### Basic Dataset Creation
```python
response = litellm.vector_stores.create(
name="basic-dataset",
custom_llm_provider="ragflow"
)
```
### Dataset with Chunk Method
RAGFlow supports various chunk methods for different document types:
<Tabs>
<TabItem value="naive" label="Naive (General)">
```python
response = litellm.vector_stores.create(
name="general-dataset",
custom_llm_provider="ragflow",
metadata={
"chunk_method": "naive",
"parser_config": {
"chunk_token_num": 512,
"delimiter": "\n",
"html4excel": False,
"layout_recognize": "DeepDOC"
}
}
)
```
</TabItem>
<TabItem value="book" label="Book">
```python
response = litellm.vector_stores.create(
name="book-dataset",
custom_llm_provider="ragflow",
metadata={
"chunk_method": "book",
"parser_config": {
"raptor": {
"use_raptor": False
}
}
}
)
```
</TabItem>
<TabItem value="qa" label="Q&A">
```python
response = litellm.vector_stores.create(
name="qa-dataset",
custom_llm_provider="ragflow",
metadata={
"chunk_method": "qa",
"parser_config": {
"raptor": {
"use_raptor": False
}
}
}
)
```
</TabItem>
<TabItem value="paper" label="Paper">
```python
response = litellm.vector_stores.create(
name="paper-dataset",
custom_llm_provider="ragflow",
metadata={
"chunk_method": "paper",
"parser_config": {
"raptor": {
"use_raptor": False
}
}
}
)
```
</TabItem>
</Tabs>
### Dataset with Ingestion Pipeline
Instead of using a chunk method, you can use an ingestion pipeline:
```python
response = litellm.vector_stores.create(
name="pipeline-dataset",
custom_llm_provider="ragflow",
metadata={
"parse_type": 2, # Number of parsers in your pipeline
"pipeline_id": "d0bebe30ae2211f0970942010a8e0005" # 32-character hex ID
}
)
```
**Note**: `chunk_method` and `pipeline_id` are mutually exclusive. Use one or the other.
### Advanced Parser Configuration
```python
response = litellm.vector_stores.create(
name="advanced-dataset",
custom_llm_provider="ragflow",
metadata={
"chunk_method": "naive",
"description": "Advanced dataset with custom parser config",
"embedding_model": "BAAI/bge-large-zh-v1.5@BAAI",
"permission": "me", # or "team"
"parser_config": {
"chunk_token_num": 1024,
"delimiter": "\n!?;。；！？",
"html4excel": True,
"layout_recognize": "DeepDOC",
"auto_keywords": 5,
"auto_questions": 3,
"task_page_size": 12,
"raptor": {
"use_raptor": True
},
"graphrag": {
"use_graphrag": False
}
}
}
)
```
## Supported Chunk Methods
RAGFlow supports the following chunk methods:
- `naive` - General purpose (default)
- `book` - For book documents
- `email` - For email documents
- `laws` - For legal documents
- `manual` - Manual chunking
- `one` - Single chunk
- `paper` - For academic papers
- `picture` - For image documents
- `presentation` - For presentation documents
- `qa` - Q&A format
- `table` - For table documents
- `tag` - Tag-based chunking
## RAGFlow-Specific Parameters
All RAGFlow-specific parameters should be passed via the `metadata` field:
| Parameter | Type | Description |
|-----------|------|-------------|
| `avatar` | string | Base64 encoding of the avatar (max 65535 chars) |
| `description` | string | Brief description of the dataset (max 65535 chars) |
| `embedding_model` | string | Embedding model name (e.g., "BAAI/bge-large-zh-v1.5@BAAI") |
| `permission` | string | Access permission: "me" (default) or "team" |
| `chunk_method` | string | Chunking method (see supported methods above) |
| `parser_config` | object | Parser configuration (varies by chunk_method) |
| `parse_type` | int | Number of parsers in pipeline (required with pipeline_id) |
| `pipeline_id` | string | 32-character hex pipeline ID (required with parse_type) |
## Error Handling
RAGFlow returns error responses in the following format:
```json
{
"code": 101,
"message": "Dataset name 'my-dataset' already exists"
}
```
LiteLLM automatically maps these to appropriate exceptions:
- `code != 0` → Raises exception with the error message
- Missing required fields → Raises `ValueError`
- Mutually exclusive parameters → Raises `ValueError`
## Limitations
- **Search/Retrieval**: RAGFlow vector stores support dataset management only. Search operations are not supported and will raise `NotImplementedError`.
- **List/Update/Delete**: These operations are not yet implemented through the standard vector store API. Use RAGFlow's native API endpoints directly.
## Further Reading
Vector Stores:
- [Vector Store Creation](../vector_stores/create.md)
- [Using Vector Stores with Completions](../completion/knowledgebase.md)
- [Vector Store Registry](../completion/knowledgebase.md#vectorstoreregistry)

1
docs/my-website/docs/vector_stores/create.md
View file

@ -14,6 +14,7 @@ Create a vector store which can be used to store and search document chunks for
| End-user Tracking | ✅ | |
| Support LLM Providers (OpenAI `/vector_stores` API) | **OpenAI** | Full vector stores API support across providers |
| Support LLM Providers (Passthrough API) | [**Azure AI**](/docs/providers/azure_ai/azure_ai_vector_stores_passthrough) | Full vector stores API support across providers |
| Support LLM Providers (Dataset Management) | [**RAGFlow**](/docs/providers/ragflow_vector_store.md) | Dataset creation and management (search not supported) |
## Usage

4
docs/my-website/release_notes/v1.80.5-stable/index.md
View file

@ -1,5 +1,5 @@
---
title: "[PREVIEW] v1.80.5.rc.2 - Gemini 3.0 Support"
title: "v1.80.5-stable - Gemini 3.0 Support"
slug: "v1-80-5"
date: 2025-11-22T10:00:00
authors:
@ -27,7 +27,7 @@ import TabItem from '@theme/TabItem';
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.80.5.rc.2
ghcr.io/berriai/litellm:v1.80.5-stable
```
</TabItem>

6
docs/my-website/sidebars.js
View file

@ -627,6 +627,7 @@ const sidebars = {
"providers/petals",
"providers/publicai",
"providers/predibase",
"providers/ragflow",
"providers/recraft",
"providers/replicate",
{
@ -820,10 +821,13 @@ const sidebars = {
"Learn how to deploy + call models from different providers on LiteLLM",
slug: "/project",
},
items: [
items: [
"projects/smolagents",
"projects/mini-swe-agent",
"projects/openai-agents",
"projects/Google ADK",
"projects/Agent Lightning",
"projects/Harbor",
"projects/Docq.AI",
"projects/PDL",
"projects/OpenInterpreter",

47
docs/my-website/src/pages/intro.md
View file

@ -1,47 +0,0 @@
---
sidebar_position: 1
---
# Tutorial Intro
Let's discover **Docusaurus in less than 5 minutes**.
## Getting Started
Get started by **creating a new site**.
Or **try Docusaurus immediately** with **[docusaurus.new](https://docusaurus.new)**.
### What you'll need
- [Node.js](https://nodejs.org/en/download/) version 16.14 or above:
- When installing Node.js, you are recommended to check all checkboxes related to dependencies.
## Generate a new site
Generate a new Docusaurus site using the **classic template**.
The classic template will automatically be added to your project after you run the command:
```bash
npm init docusaurus@latest my-website classic
```
You can type this command into Command Prompt, Powershell, Terminal, or any other integrated terminal of your code editor.
The command also installs all necessary dependencies you need to run Docusaurus.
## Start your site
Run the development server:
```bash
cd my-website
npm run start
```
The `cd` command changes the directory you're working with. In order to work with your newly created Docusaurus site, you'll need to navigate the terminal there.
The `npm run start` command builds your website locally and serves it through a development server, ready for you to view at http://localhost:3000/.
Open `docs/intro.md` (this page) and edit some lines: the site **reloads automatically** and displays your changes.

8
docs/my-website/src/pages/tutorial-basics/_category_.json
View file

@ -1,8 +0,0 @@
{
"label": "Tutorial - Basics",
"position": 2,
"link": {
"type": "generated-index",
"description": "5 minutes to learn the most important Docusaurus concepts."
}
}

23
docs/my-website/src/pages/tutorial-basics/congratulations.md
View file

@ -1,23 +0,0 @@
---
sidebar_position: 6
---
# Congratulations!
You have just learned the **basics of Docusaurus** and made some changes to the **initial template**.
Docusaurus has **much more to offer**!
Have **5 more minutes**? Take a look at **[versioning](../tutorial-extras/manage-docs-versions.md)** and **[i18n](../tutorial-extras/translate-your-site.md)**.
Anything **unclear** or **buggy** in this tutorial? [Please report it!](https://github.com/facebook/docusaurus/discussions/4610)
## What's next?
- Read the [official documentation](https://docusaurus.io/)
- Modify your site configuration with [`docusaurus.config.js`](https://docusaurus.io/docs/api/docusaurus-config)
- Add navbar and footer items with [`themeConfig`](https://docusaurus.io/docs/api/themes/configuration)
- Add a custom [Design and Layout](https://docusaurus.io/docs/styling-layout)
- Add a [search bar](https://docusaurus.io/docs/search)
- Find inspirations in the [Docusaurus showcase](https://docusaurus.io/showcase)
- Get involved in the [Docusaurus Community](https://docusaurus.io/community/support)

34
docs/my-website/src/pages/tutorial-basics/create-a-blog-post.md
View file

@ -1,34 +0,0 @@
---
sidebar_position: 3
---
# Create a Blog Post
Docusaurus creates a **page for each blog post**, but also a **blog index page**, a **tag system**, an **RSS** feed...
## Create your first Post
Create a file at `blog/2021-02-28-greetings.md`:
```md title="blog/2021-02-28-greetings.md"
---
slug: greetings
title: Greetings!
authors:
- name: Joel Marcey
title: Co-creator of Docusaurus 1
url: https://github.com/JoelMarcey
image_url: https://github.com/JoelMarcey.png
- name: Sébastien Lorber
title: Docusaurus maintainer
url: https://sebastienlorber.com
image_url: https://github.com/slorber.png
tags: [greetings]
---
Congratulations, you have made your first post!
Feel free to play around and edit this post as much you like.
```
A new blog post is now available at [http://localhost:3000/blog/greetings](http://localhost:3000/blog/greetings).

57
docs/my-website/src/pages/tutorial-basics/create-a-document.md
View file

@ -1,57 +0,0 @@
---
sidebar_position: 2
---
# Create a Document
Documents are **groups of pages** connected through:
- a **sidebar**
- **previous/next navigation**
- **versioning**
## Create your first Doc
Create a Markdown file at `docs/hello.md`:
```md title="docs/hello.md"
# Hello
This is my **first Docusaurus document**!
```
A new document is now available at [http://localhost:3000/docs/hello](http://localhost:3000/docs/hello).
## Configure the Sidebar
Docusaurus automatically **creates a sidebar** from the `docs` folder.
Add metadata to customize the sidebar label and position:
```md title="docs/hello.md" {1-4}
---
sidebar_label: 'Hi!'
sidebar_position: 3
---
# Hello
This is my **first Docusaurus document**!
```
It is also possible to create your sidebar explicitly in `sidebars.js`:
```js title="sidebars.js"
module.exports = {
tutorialSidebar: [
'intro',
// highlight-next-line
'hello',
{
type: 'category',
label: 'Tutorial',
items: ['tutorial-basics/create-a-document'],
},
],
};
```

43
docs/my-website/src/pages/tutorial-basics/create-a-page.md
View file

@ -1,43 +0,0 @@
---
sidebar_position: 1
---
# Create a Page
Add **Markdown or React** files to `src/pages` to create a **standalone page**:
- `src/pages/index.js` → `localhost:3000/`
- `src/pages/foo.md` → `localhost:3000/foo`
- `src/pages/foo/bar.js` → `localhost:3000/foo/bar`
## Create your first React Page
Create a file at `src/pages/my-react-page.js`:
```jsx title="src/pages/my-react-page.js"
import React from 'react';
import Layout from '@theme/Layout';
export default function MyReactPage() {
return (
<Layout>
<h1>My React page</h1>
<p>This is a React page</p>
</Layout>
);
}
```
A new page is now available at [http://localhost:3000/my-react-page](http://localhost:3000/my-react-page).
## Create your first Markdown Page
Create a file at `src/pages/my-markdown-page.md`:
```mdx title="src/pages/my-markdown-page.md"
# My Markdown page
This is a Markdown page
```
A new page is now available at [http://localhost:3000/my-markdown-page](http://localhost:3000/my-markdown-page).

31
docs/my-website/src/pages/tutorial-basics/deploy-your-site.md
View file

@ -1,31 +0,0 @@
---
sidebar_position: 5
---
# Deploy your site
Docusaurus is a **static-site-generator** (also called **[Jamstack](https://jamstack.org/)**).
It builds your site as simple **static HTML, JavaScript and CSS files**.
## Build your site
Build your site **for production**:
```bash
npm run build
```
The static files are generated in the `build` folder.
## Deploy your site
Test your production build locally:
```bash
npm run serve
```
The `build` folder is now served at [http://localhost:3000/](http://localhost:3000/).
You can now deploy the `build` folder **almost anywhere** easily, **for free** or very small cost (read the **[Deployment Guide](https://docusaurus.io/docs/deployment)**).

150
docs/my-website/src/pages/tutorial-basics/markdown-features.mdx
View file

@ -1,150 +0,0 @@
---
sidebar_position: 4
---
# Markdown Features
Docusaurus supports **[Markdown](https://daringfireball.net/projects/markdown/syntax)** and a few **additional features**.
## Front Matter
Markdown documents have metadata at the top called [Front Matter](https://jekyllrb.com/docs/front-matter/):
```text title="my-doc.md"
// highlight-start
---
id: my-doc-id
title: My document title
description: My document description
slug: /my-custom-url
---
// highlight-end
## Markdown heading
Markdown text with [links](./hello.md)
```
## Links
Regular Markdown links are supported, using url paths or relative file paths.
```md
Let's see how to [Create a page](/create-a-page).
```
```md
Let's see how to [Create a page](./create-a-page.md).
```
**Result:** Let's see how to [Create a page](./create-a-page.md).
## Images
Regular Markdown images are supported.
You can use absolute paths to reference images in the static directory (`static/img/docusaurus.png`):
```md
![Docusaurus logo](/img/docusaurus.png)
```
![Docusaurus logo](/img/docusaurus.png)
You can reference images relative to the current file as well. This is particularly useful to colocate images close to the Markdown files using them:
```md
![Docusaurus logo](./img/docusaurus.png)
```
## Code Blocks
Markdown code blocks are supported with Syntax highlighting.
```jsx title="src/components/HelloDocusaurus.js"
function HelloDocusaurus() {
return (
<h1>Hello, Docusaurus!</h1>
)
}
```
```jsx title="src/components/HelloDocusaurus.js"
function HelloDocusaurus() {
return <h1>Hello, Docusaurus!</h1>;
}
```
## Admonitions
Docusaurus has a special syntax to create admonitions and callouts:
:::tip My tip
Use this awesome feature option
:::
:::danger Take care
This action is dangerous
:::
:::tip My tip
Use this awesome feature option
:::
:::danger Take care
This action is dangerous
:::
## MDX and React Components
[MDX](https://mdxjs.com/) can make your documentation more **interactive** and allows using any **React components inside Markdown**:
```jsx
export const Highlight = ({children, color}) => (
<span
style={{
backgroundColor: color,
borderRadius: '20px',
color: '#fff',
padding: '10px',
cursor: 'pointer',
}}
onClick={() => {
alert(`You clicked the color ${color} with label ${children}`)
}}>
{children}
</span>
);
This is <Highlight color="#25c2a0">Docusaurus green</Highlight> !
This is <Highlight color="#1877F2">Facebook blue</Highlight> !
```
export const Highlight = ({children, color}) => (
<span
style={{
backgroundColor: color,
borderRadius: '20px',
color: '#fff',
padding: '10px',
cursor: 'pointer',
}}
onClick={() => {
alert(`You clicked the color ${color} with label ${children}`);
}}>
{children}
</span>
);
This is <Highlight color="#25c2a0">Docusaurus green</Highlight> !
This is <Highlight color="#1877F2">Facebook blue</Highlight> !

7
docs/my-website/src/pages/tutorial-extras/_category_.json
View file

@ -1,7 +0,0 @@
{
"label": "Tutorial - Extras",
"position": 3,
"link": {
"type": "generated-index"
}
}

BIN
docs/my-website/src/pages/tutorial-extras/img/docsVersionDropdown.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 25 KiB

BIN
docs/my-website/src/pages/tutorial-extras/img/localeDropdown.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 27 KiB

55
docs/my-website/src/pages/tutorial-extras/manage-docs-versions.md
View file

@ -1,55 +0,0 @@
---
sidebar_position: 1
---
# Manage Docs Versions
Docusaurus can manage multiple versions of your docs.
## Create a docs version
Release a version 1.0 of your project:
```bash
npm run docusaurus docs:version 1.0
```
The `docs` folder is copied into `versioned_docs/version-1.0` and `versions.json` is created.
Your docs now have 2 versions:
- `1.0` at `http://localhost:3000/docs/` for the version 1.0 docs
- `current` at `http://localhost:3000/docs/next/` for the **upcoming, unreleased docs**
## Add a Version Dropdown
To navigate seamlessly across versions, add a version dropdown.
Modify the `docusaurus.config.js` file:
```js title="docusaurus.config.js"
module.exports = {
themeConfig: {
navbar: {
items: [
// highlight-start
{
type: 'docsVersionDropdown',
},
// highlight-end
],
},
},
};
```
The docs version dropdown appears in your navbar:
![Docs Version Dropdown](./img/docsVersionDropdown.png)
## Update an existing version
It is possible to edit versioned docs in their respective folder:
- `versioned_docs/version-1.0/hello.md` updates `http://localhost:3000/docs/hello`
- `docs/hello.md` updates `http://localhost:3000/docs/next/hello`

88
docs/my-website/src/pages/tutorial-extras/translate-your-site.md
View file

@ -1,88 +0,0 @@
---
sidebar_position: 2
---
# Translate your site
Let's translate `docs/intro.md` to French.
## Configure i18n
Modify `docusaurus.config.js` to add support for the `fr` locale:
```js title="docusaurus.config.js"
module.exports = {
i18n: {
defaultLocale: 'en',
locales: ['en', 'fr'],
},
};
```
## Translate a doc
Copy the `docs/intro.md` file to the `i18n/fr` folder:
```bash
mkdir -p i18n/fr/docusaurus-plugin-content-docs/current/
cp docs/intro.md i18n/fr/docusaurus-plugin-content-docs/current/intro.md
```
Translate `i18n/fr/docusaurus-plugin-content-docs/current/intro.md` in French.
## Start your localized site
Start your site on the French locale:
```bash
npm run start -- --locale fr
```
Your localized site is accessible at [http://localhost:3000/fr/](http://localhost:3000/fr/) and the `Getting Started` page is translated.
:::caution
In development, you can only use one locale at a same time.
:::
## Add a Locale Dropdown
To navigate seamlessly across languages, add a locale dropdown.
Modify the `docusaurus.config.js` file:
```js title="docusaurus.config.js"
module.exports = {
themeConfig: {
navbar: {
items: [
// highlight-start
{
type: 'localeDropdown',
},
// highlight-end
],
},
},
};
```
The locale dropdown now appears in your navbar:
![Locale Dropdown](./img/localeDropdown.png)
## Build your localized site
Build your site for a specific locale:
```bash
npm run build -- --locale fr
```
Or build your site to include all the locales at once:
```bash
npm run build
```

148
litellm/__init__.py
View file

@ -1056,57 +1056,10 @@ from .timeout import timeout
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
from litellm.litellm_core_utils.core_helpers import remove_index_from_tool_calls
from litellm.litellm_core_utils.token_counter import get_modified_max_tokens
from .utils import (
client,
exception_type,
get_optional_params,
get_response_string,
token_counter,
create_pretrained_tokenizer,
create_tokenizer,
supports_function_calling,
supports_web_search,
supports_url_context,
supports_response_schema,
supports_parallel_function_calling,
supports_vision,
supports_audio_input,
supports_audio_output,
supports_system_messages,
supports_reasoning,
get_litellm_params,
acreate,
get_max_tokens,
get_model_info,
register_prompt_template,
validate_environment,
check_valid_key,
register_model,
encode,
decode,
_calculate_retry_after,
_should_retry,
get_supported_openai_params,
get_api_base,
get_first_chars_messages,
ModelResponse,
ModelResponseStream,
EmbeddingResponse,
ImageResponse,
TranscriptionResponse,
TextCompletionResponse,
get_provider_fields,
ModelResponseListIterator,
get_valid_models,
)
ALL_LITELLM_RESPONSE_TYPES = [
ModelResponse,
EmbeddingResponse,
ImageResponse,
TranscriptionResponse,
TextCompletionResponse,
]
# client must be imported immediately as it's used as a decorator at function definition time
from .utils import client
# Note: Most other utils imports are lazy-loaded via __getattr__ to avoid loading utils.py
# (which imports tiktoken) at import time
from .llms.bytez.chat.transformation import BytezChatConfig
from .llms.custom_llm import CustomLLM
@ -1387,6 +1340,7 @@ from .llms.docker_model_runner.chat.transformation import DockerModelRunnerChatC
from .llms.v0.chat.transformation import V0ChatConfig
from .llms.oci.chat.transformation import OCIChatConfig
from .llms.morph.chat.transformation import MorphChatConfig
from .llms.ragflow.chat.transformation import RAGFlowConfig
from .llms.lambda_ai.chat.transformation import LambdaAIChatConfig
from .llms.hyperbolic.chat.transformation import HyperbolicChatConfig
from .llms.vercel_ai_gateway.chat.transformation import VercelAIGatewayConfig
@ -1537,56 +1491,6 @@ def set_global_gitlab_config(config: Dict[str, Any]) -> None:
# Lazy loading system for heavy modules to reduce initial import time and memory usage
def _lazy_import_cost_calculator(name: str) -> Any:
"""Lazy import for cost_calculator functions."""
from .cost_calculator import (
completion_cost as _completion_cost,
cost_per_token as _cost_per_token,
response_cost_calculator as _response_cost_calculator,
)
_cost_functions = {
"completion_cost": _completion_cost,
"cost_per_token": _cost_per_token,
"response_cost_calculator": _response_cost_calculator,
}
func = _cost_functions[name]
globals()[name] = func
return func
def _lazy_import_litellm_logging(name: str) -> Any:
"""Lazy import for litellm_logging module."""
try:
from litellm.litellm_core_utils.litellm_logging import (
Logging as _Logging,
modify_integration as _modify_integration,
)
_logging_objects = {
"Logging": _Logging,
"modify_integration": _modify_integration,
}
obj = _logging_objects[name]
globals()[name] = obj
return obj
except Exception as e:
raise AttributeError(
f"module {__name__!r} has no attribute {name!r}. "
f"Lazy import failed: {e}"
) from e
_LAZY_LOAD_REGISTRY: Dict[str, Callable[[str], Any]] = {
"completion_cost": _lazy_import_cost_calculator,
"cost_per_token": _lazy_import_cost_calculator,
"response_cost_calculator": _lazy_import_cost_calculator,
"Logging": _lazy_import_litellm_logging,
"modify_integration": _lazy_import_litellm_logging,
}
if TYPE_CHECKING:
cost_per_token: Callable[..., Tuple[float, float]]
@ -1597,7 +1501,45 @@ if TYPE_CHECKING:
def __getattr__(name: str) -> Any:
"""Lazy import handler for cost_calculator and litellm_logging functions."""
if name in _LAZY_LOAD_REGISTRY:
return _LAZY_LOAD_REGISTRY[name](name)
# Lazy load cost_calculator functions
_cost_calculator_names = (
"completion_cost",
"cost_per_token",
"response_cost_calculator",
)
if name in _cost_calculator_names:
from ._lazy_imports import _lazy_import_cost_calculator
return _lazy_import_cost_calculator(name)
# Lazy load litellm_logging functions
_litellm_logging_names = (
"Logging",
"modify_integration",
)
if name in _litellm_logging_names:
from ._lazy_imports import _lazy_import_litellm_logging
return _lazy_import_litellm_logging(name)
# Lazy load utils functions
_utils_names = (
"exception_type", "get_optional_params", "get_response_string", "token_counter",
"create_pretrained_tokenizer", "create_tokenizer", "supports_function_calling",
"supports_web_search", "supports_url_context", "supports_response_schema",
"supports_parallel_function_calling", "supports_vision", "supports_audio_input",
"supports_audio_output", "supports_system_messages", "supports_reasoning",
"get_litellm_params", "acreate", "get_max_tokens", "get_model_info",
"register_prompt_template", "validate_environment", "check_valid_key",
"register_model", "encode", "decode", "_calculate_retry_after", "_should_retry",
"get_supported_openai_params", "get_api_base", "get_first_chars_messages",
"ModelResponse", "ModelResponseStream", "EmbeddingResponse", "ImageResponse",
"TranscriptionResponse", "TextCompletionResponse", "get_provider_fields",
"ModelResponseListIterator", "get_valid_models",
)
if name in _utils_names:
from ._lazy_imports import _lazy_import_utils
return _lazy_import_utils(name)
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
# ALL_LITELLM_RESPONSE_TYPES is lazy-loaded via __getattr__ to avoid loading utils at import time

259
litellm/_lazy_imports.py Normal file
View file

@ -0,0 +1,259 @@
from typing import Any
import sys
def _get_litellm_globals() -> dict:
"""Helper to get the globals dictionary of the litellm module."""
return sys.modules["litellm"].__dict__
# Lazy import for utils module - imports only the requested item by name.
# Note: PLR0915 (too many statements) is suppressed because the many if statements
# are intentional - each attribute is imported individually only when requested,
# ensuring true lazy imports rather than importing the entire utils module.
def _lazy_import_utils(name: str) -> Any: # noqa: PLR0915
"""Lazy import for utils module - imports only the requested item by name."""
_globals = _get_litellm_globals()
if name == "exception_type":
from .utils import exception_type as _exception_type
_globals["exception_type"] = _exception_type
return _exception_type
if name == "get_optional_params":
from .utils import get_optional_params as _get_optional_params
_globals["get_optional_params"] = _get_optional_params
return _get_optional_params
if name == "get_response_string":
from .utils import get_response_string as _get_response_string
_globals["get_response_string"] = _get_response_string
return _get_response_string
if name == "token_counter":
from .utils import token_counter as _token_counter
_globals["token_counter"] = _token_counter
return _token_counter
if name == "create_pretrained_tokenizer":
from .utils import create_pretrained_tokenizer as _create_pretrained_tokenizer
_globals["create_pretrained_tokenizer"] = _create_pretrained_tokenizer
return _create_pretrained_tokenizer
if name == "create_tokenizer":
from .utils import create_tokenizer as _create_tokenizer
_globals["create_tokenizer"] = _create_tokenizer
return _create_tokenizer
if name == "supports_function_calling":
from .utils import supports_function_calling as _supports_function_calling
_globals["supports_function_calling"] = _supports_function_calling
return _supports_function_calling
if name == "supports_web_search":
from .utils import supports_web_search as _supports_web_search
_globals["supports_web_search"] = _supports_web_search
return _supports_web_search
if name == "supports_url_context":
from .utils import supports_url_context as _supports_url_context
_globals["supports_url_context"] = _supports_url_context
return _supports_url_context
if name == "supports_response_schema":
from .utils import supports_response_schema as _supports_response_schema
_globals["supports_response_schema"] = _supports_response_schema
return _supports_response_schema
if name == "supports_parallel_function_calling":
from .utils import supports_parallel_function_calling as _supports_parallel_function_calling
_globals["supports_parallel_function_calling"] = _supports_parallel_function_calling
return _supports_parallel_function_calling
if name == "supports_vision":
from .utils import supports_vision as _supports_vision
_globals["supports_vision"] = _supports_vision
return _supports_vision
if name == "supports_audio_input":
from .utils import supports_audio_input as _supports_audio_input
_globals["supports_audio_input"] = _supports_audio_input
return _supports_audio_input
if name == "supports_audio_output":
from .utils import supports_audio_output as _supports_audio_output
_globals["supports_audio_output"] = _supports_audio_output
return _supports_audio_output
if name == "supports_system_messages":
from .utils import supports_system_messages as _supports_system_messages
_globals["supports_system_messages"] = _supports_system_messages
return _supports_system_messages
if name == "supports_reasoning":
from .utils import supports_reasoning as _supports_reasoning
_globals["supports_reasoning"] = _supports_reasoning
return _supports_reasoning
if name == "get_litellm_params":
from .utils import get_litellm_params as _get_litellm_params
_globals["get_litellm_params"] = _get_litellm_params
return _get_litellm_params
if name == "acreate":
from .utils import acreate as _acreate
_globals["acreate"] = _acreate
return _acreate
if name == "get_max_tokens":
from .utils import get_max_tokens as _get_max_tokens
_globals["get_max_tokens"] = _get_max_tokens
return _get_max_tokens
if name == "get_model_info":
from .utils import get_model_info as _get_model_info
_globals["get_model_info"] = _get_model_info
return _get_model_info
if name == "register_prompt_template":
from .utils import register_prompt_template as _register_prompt_template
_globals["register_prompt_template"] = _register_prompt_template
return _register_prompt_template
if name == "validate_environment":
from .utils import validate_environment as _validate_environment
_globals["validate_environment"] = _validate_environment
return _validate_environment
if name == "check_valid_key":
from .utils import check_valid_key as _check_valid_key
_globals["check_valid_key"] = _check_valid_key
return _check_valid_key
if name == "register_model":
from .utils import register_model as _register_model
_globals["register_model"] = _register_model
return _register_model
if name == "encode":
from .utils import encode as _encode
_globals["encode"] = _encode
return _encode
if name == "decode":
from .utils import decode as _decode
_globals["decode"] = _decode
return _decode
if name == "_calculate_retry_after":
from .utils import _calculate_retry_after as __calculate_retry_after
_globals["_calculate_retry_after"] = __calculate_retry_after
return __calculate_retry_after
if name == "_should_retry":
from .utils import _should_retry as __should_retry
_globals["_should_retry"] = __should_retry
return __should_retry
if name == "get_supported_openai_params":
from .utils import get_supported_openai_params as _get_supported_openai_params
_globals["get_supported_openai_params"] = _get_supported_openai_params
return _get_supported_openai_params
if name == "get_api_base":
from .utils import get_api_base as _get_api_base
_globals["get_api_base"] = _get_api_base
return _get_api_base
if name == "get_first_chars_messages":
from .utils import get_first_chars_messages as _get_first_chars_messages
_globals["get_first_chars_messages"] = _get_first_chars_messages
return _get_first_chars_messages
if name == "ModelResponse":
from .utils import ModelResponse as _ModelResponse
_globals["ModelResponse"] = _ModelResponse
return _ModelResponse
if name == "ModelResponseStream":
from .utils import ModelResponseStream as _ModelResponseStream
_globals["ModelResponseStream"] = _ModelResponseStream
return _ModelResponseStream
if name == "EmbeddingResponse":
from .utils import EmbeddingResponse as _EmbeddingResponse
_globals["EmbeddingResponse"] = _EmbeddingResponse
return _EmbeddingResponse
if name == "ImageResponse":
from .utils import ImageResponse as _ImageResponse
_globals["ImageResponse"] = _ImageResponse
return _ImageResponse
if name == "TranscriptionResponse":
from .utils import TranscriptionResponse as _TranscriptionResponse
_globals["TranscriptionResponse"] = _TranscriptionResponse
return _TranscriptionResponse
if name == "TextCompletionResponse":
from .utils import TextCompletionResponse as _TextCompletionResponse
_globals["TextCompletionResponse"] = _TextCompletionResponse
return _TextCompletionResponse
if name == "get_provider_fields":
from .utils import get_provider_fields as _get_provider_fields
_globals["get_provider_fields"] = _get_provider_fields
return _get_provider_fields
if name == "ModelResponseListIterator":
from .utils import ModelResponseListIterator as _ModelResponseListIterator
_globals["ModelResponseListIterator"] = _ModelResponseListIterator
return _ModelResponseListIterator
if name == "get_valid_models":
from .utils import get_valid_models as _get_valid_models
_globals["get_valid_models"] = _get_valid_models
return _get_valid_models
raise AttributeError(f"Utils lazy import: unknown attribute {name!r}")
def _lazy_import_cost_calculator(name: str) -> Any:
"""Lazy import for cost_calculator functions."""
_globals = _get_litellm_globals()
from .cost_calculator import (
completion_cost as _completion_cost,
cost_per_token as _cost_per_token,
response_cost_calculator as _response_cost_calculator,
)
_cost_functions = {
"completion_cost": _completion_cost,
"cost_per_token": _cost_per_token,
"response_cost_calculator": _response_cost_calculator,
}
func = _cost_functions[name]
_globals[name] = func
return func
def _lazy_import_litellm_logging(name: str) -> Any:
"""Lazy import for litellm_logging module."""
_globals = _get_litellm_globals()
try:
from litellm.litellm_core_utils.litellm_logging import (
Logging as _Logging,
modify_integration as _modify_integration,
)
_logging_objects = {
"Logging": _Logging,
"modify_integration": _modify_integration,
}
obj = _logging_objects[name]
_globals[name] = obj
return obj
except Exception as e:
raise AttributeError(
f"module 'litellm' has no attribute {name!r}. "
f"Lazy import failed: {e}"
) from e

2
litellm/batches/main.py
View file

@ -18,8 +18,6 @@ from typing import Any, Coroutine, Dict, Literal, Optional, Union, cast
import httpx
from openai.types.batch import BatchRequestCounts
from openai.types.batch import Metadata
from openai.types.batch import Metadata as OpenAIBatchMetadata
import litellm
from litellm._logging import verbose_logger

32
litellm/constants.py
View file

@ -1,4 +1,5 @@
import os
import sys
from typing import List, Literal
DEFAULT_HEALTH_CHECK_PROMPT = str(
@ -99,10 +100,18 @@ RUNWAYML_POLLING_TIMEOUT = int(
########## Networking constants ##############################################################
_DEFAULT_TTL_FOR_HTTPX_CLIENTS = 3600 # 1 hour, re-use the same httpx client for 1 hour
# Aiohttp connection pooling constants
AIOHTTP_CONNECTOR_LIMIT = int(os.getenv("AIOHTTP_CONNECTOR_LIMIT", 0))
# Aiohttp connection pooling - prevents memory leaks from unbounded connection growth
# Set to 0 for unlimited (not recommended for production)
AIOHTTP_CONNECTOR_LIMIT = int(os.getenv("AIOHTTP_CONNECTOR_LIMIT", 300))
AIOHTTP_CONNECTOR_LIMIT_PER_HOST = int(os.getenv("AIOHTTP_CONNECTOR_LIMIT_PER_HOST", 50))
AIOHTTP_KEEPALIVE_TIMEOUT = int(os.getenv("AIOHTTP_KEEPALIVE_TIMEOUT", 120))
AIOHTTP_TTL_DNS_CACHE = int(os.getenv("AIOHTTP_TTL_DNS_CACHE", 300))
# enable_cleanup_closed is only needed for Python versions with the SSL leak bug
# Fixed in Python 3.12.7+ and 3.13.1+ (see https://github.com/python/cpython/pull/118960)
# Reference: https://github.com/aio-libs/aiohttp/blob/master/aiohttp/connector.py#L74-L78
AIOHTTP_NEEDS_CLEANUP_CLOSED = (
(3, 13, 0) <= sys.version_info < (3, 13, 1) or sys.version_info < (3, 12, 7)
)
# WebSocket constants
# Default to None (unlimited) to match OpenAI's official agents SDK behavior
@ -255,7 +264,9 @@ TOGETHER_AI_EMBEDDING_350_M = int(os.getenv("TOGETHER_AI_EMBEDDING_350_M", 350))
QDRANT_SCALAR_QUANTILE = float(os.getenv("QDRANT_SCALAR_QUANTILE", 0.99))
QDRANT_VECTOR_SIZE = int(os.getenv("QDRANT_VECTOR_SIZE", 1536))
CACHED_STREAMING_CHUNK_DELAY = float(os.getenv("CACHED_STREAMING_CHUNK_DELAY", 0.02))
AUDIO_SPEECH_CHUNK_SIZE = 8192 # chunk_size for audio speech streaming. Balance between latency and memory usage
AUDIO_SPEECH_CHUNK_SIZE = int(
os.getenv("AUDIO_SPEECH_CHUNK_SIZE", 8192)
) # chunk_size for audio speech streaming. Balance between latency and memory usage
MAX_SIZE_PER_ITEM_IN_MEMORY_CACHE_IN_KB = int(
os.getenv("MAX_SIZE_PER_ITEM_IN_MEMORY_CACHE_IN_KB", 512)
)
@ -278,10 +289,16 @@ REDACTED_BY_LITELM_STRING = "REDACTED_BY_LITELM"
MAX_LANGFUSE_INITIALIZED_CLIENTS = int(
os.getenv("MAX_LANGFUSE_INITIALIZED_CLIENTS", 50)
)
LOGGING_WORKER_CONCURRENCY = int(os.getenv("LOGGING_WORKER_CONCURRENCY", 100)) # Must be above 0
LOGGING_WORKER_CONCURRENCY = int(
os.getenv("LOGGING_WORKER_CONCURRENCY", 100)
) # Must be above 0
LOGGING_WORKER_MAX_QUEUE_SIZE = int(os.getenv("LOGGING_WORKER_MAX_QUEUE_SIZE", 50_000))
LOGGING_WORKER_MAX_TIME_PER_COROUTINE = float(os.getenv("LOGGING_WORKER_MAX_TIME_PER_COROUTINE", 20.0))
LOGGING_WORKER_CLEAR_PERCENTAGE = int(os.getenv("LOGGING_WORKER_CLEAR_PERCENTAGE", 50)) # Percentage of queue to clear (default: 50%)
LOGGING_WORKER_MAX_TIME_PER_COROUTINE = float(
os.getenv("LOGGING_WORKER_MAX_TIME_PER_COROUTINE", 20.0)
)
LOGGING_WORKER_CLEAR_PERCENTAGE = int(
os.getenv("LOGGING_WORKER_CLEAR_PERCENTAGE", 50)
) # Percentage of queue to clear (default: 50%)
MAX_ITERATIONS_TO_CLEAR_QUEUE = int(os.getenv("MAX_ITERATIONS_TO_CLEAR_QUEUE", 200))
MAX_TIME_TO_CLEAR_QUEUE = float(os.getenv("MAX_TIME_TO_CLEAR_QUEUE", 5.0))
LOGGING_WORKER_AGGRESSIVE_CLEAR_COOLDOWN_SECONDS = float(
@ -586,6 +603,7 @@ openai_compatible_providers: List = [
"cometapi",
"clarifai",
"docker_model_runner",
"ragflow",
]
openai_text_completion_compatible_providers: List = (
[ # providers that support `/v1/completions`
@ -859,7 +877,7 @@ BEDROCK_INVOKE_PROVIDERS_LITERAL = Literal[
"deepseek_r1",
"qwen3",
"twelvelabs",
"openai"
"openai",
]
BEDROCK_EMBEDDING_PROVIDERS_LITERAL = Literal[

4
litellm/images/main.py
View file

@ -6,7 +6,9 @@ from typing import Any, Coroutine, Dict, List, Literal, Optional, Union, cast, o
import httpx
import litellm
from litellm import client, exception_type, get_litellm_params
from litellm.utils import exception_type, get_litellm_params
# client is imported from litellm as it's a decorator
from litellm import client
from litellm.constants import DEFAULT_IMAGE_ENDPOINT_MODEL
from litellm.constants import request_timeout as DEFAULT_REQUEST_TIMEOUT
from litellm.exceptions import LiteLLMUnknownProvider

10
litellm/integrations/SlackAlerting/budget_alert_types.py
View file

@ -50,6 +50,14 @@ class TeamBudgetAlert(BaseBudgetAlertType):
return user_info.team_id or "default_id"
class OrganizationBudgetAlert(BaseBudgetAlertType):
def get_event_message(self) -> str:
return "Organization Budget: "
def get_id(self, user_info: CallInfo) -> str:
return user_info.organization_id or "default_id"
class TokenBudgetAlert(BaseBudgetAlertType):
def get_event_message(self) -> str:
return "Key Budget: "
@ -72,6 +80,7 @@ def get_budget_alert_type(
"soft_budget",
"user_budget",
"team_budget",
"organization_budget",
"proxy_budget",
"projected_limit_exceeded",
],
@ -83,6 +92,7 @@ def get_budget_alert_type(
"soft_budget": SoftBudgetAlert(),
"user_budget": UserBudgetAlert(),
"team_budget": TeamBudgetAlert(),
"organization_budget": OrganizationBudgetAlert(),
"token_budget": TokenBudgetAlert(),
"projected_limit_exceeded": ProjectedLimitExceededAlert(),
}

19
litellm/integrations/SlackAlerting/slack_alerting.py
View file

@ -134,19 +134,25 @@ class SlackAlerting(CustomBatchLogger):
if llm_router is not None:
self.llm_router = llm_router
def _prepare_outage_value_for_cache(self, outage_value: Union[dict, ProviderRegionOutageModel, OutageModel]) -> dict:
def _prepare_outage_value_for_cache(
self, outage_value: Union[dict, ProviderRegionOutageModel, OutageModel]
) -> dict:
"""
Helper method to prepare outage value for Redis caching.
Converts set objects to lists for JSON serialization.
"""
# Convert to dict for processing
cache_value = dict(outage_value)
if "deployment_ids" in cache_value and isinstance(cache_value["deployment_ids"], set):
if "deployment_ids" in cache_value and isinstance(
cache_value["deployment_ids"], set
):
cache_value["deployment_ids"] = list(cache_value["deployment_ids"])
return cache_value
def _restore_outage_value_from_cache(self, outage_value: Optional[dict]) -> Optional[dict]:
def _restore_outage_value_from_cache(
self, outage_value: Optional[dict]
) -> Optional[dict]:
"""
Helper method to restore outage value after retrieving from cache.
Converts list objects back to sets for proper handling.
@ -528,6 +534,7 @@ class SlackAlerting(CustomBatchLogger):
"soft_budget",
"user_budget",
"team_budget",
"organization_budget",
"proxy_budget",
"projected_limit_exceeded",
],
@ -1338,7 +1345,7 @@ Model Info:
subject=email_event["subject"],
html=email_event["html"],
)
if webhook_event.event_group == "team":
if webhook_event.event_group == Litellm_EntityType.TEAM:
from litellm.integrations.email_alerting import send_team_budget_alert
await send_team_budget_alert(webhook_event=webhook_event)
@ -1399,7 +1406,7 @@ Model Info:
current_time = datetime.now().strftime("%H:%M:%S")
_proxy_base_url = os.getenv("PROXY_BASE_URL", None)
# Use .name if it's an enum, otherwise use as is
alert_type_name = getattr(alert_type, 'name', alert_type)
alert_type_name = getattr(alert_type, "name", alert_type)
alert_type_formatted = f"Alert type: `{alert_type_name}`"
if alert_type == "daily_reports" or alert_type == "new_model_added":
formatted_message = alert_type_formatted + message

1
litellm/integrations/custom_guardrail.py
View file

@ -20,7 +20,6 @@ from litellm.types.guardrails import (
GuardrailEventHooks,
LitellmParams,
Mode,
PiiEntityType,
)
from litellm.types.llms.openai import AllMessageValues
from litellm.types.proxy.guardrails.guardrail_hooks.base import GuardrailConfigModel

13
litellm/integrations/datadog/datadog.py
View file

@ -65,11 +65,11 @@ class DataDogLogger(
`DD_SITE` - your datadog site, example = `"us5.datadoghq.com"`
Optional environment variables (DataDog Agent):
`DD_AGENT_HOST` - hostname or IP of DataDog agent, example = `"localhost"`
`DD_AGENT_PORT` - port of DataDog agent (default: 10518 for logs)
`LITELLM_DD_AGENT_HOST` - hostname or IP of DataDog agent, example = `"localhost"`
`LITELLM_DD_AGENT_PORT` - port of DataDog agent (default: 10518 for logs)
Note: If DD_AGENT_HOST is set, logs will be sent to the agent instead of directly to DataDog API.
In this case, DD_API_KEY and DD_SITE are not required (agent handles authentication).
Note: We use LITELLM_DD_AGENT_HOST instead of DD_AGENT_HOST to avoid conflicts
with ddtrace which automatically sets DD_AGENT_HOST for APM tracing.
"""
try:
verbose_logger.debug("Datadog: in init datadog logger")
@ -85,7 +85,8 @@ class DataDogLogger(
)
# Configure DataDog endpoint (Agent or Direct API)
dd_agent_host = os.getenv("DD_AGENT_HOST")
# Use LITELLM_DD_AGENT_HOST to avoid conflicts with ddtrace's DD_AGENT_HOST
dd_agent_host = os.getenv("LITELLM_DD_AGENT_HOST")
if dd_agent_host:
self._configure_dd_agent(dd_agent_host=dd_agent_host)
else:
@ -127,7 +128,7 @@ class DataDogLogger(
Args:
dd_agent_host: Hostname or IP of DataDog agent
"""
dd_agent_port = os.getenv("DD_AGENT_PORT", "10518") # default port for logs
dd_agent_port = os.getenv("LITELLM_DD_AGENT_PORT", "10518") # default port for logs
self.intake_url = f"http://{dd_agent_host}:{dd_agent_port}/api/v2/logs"
self.DD_API_KEY = os.getenv("DD_API_KEY") # Optional when using agent
verbose_logger.debug(f"Datadog: Using DD Agent at {self.intake_url}")

1
litellm/litellm_core_utils/README.md
View file

@ -9,4 +9,5 @@ Core files:
- `default_encoding.py`: code for loading the default encoding (tiktoken)
- `get_llm_provider_logic.py`: code for inferring the LLM provider from a given model name.
- `duration_parser.py`: code for parsing durations - e.g. "1d", "1mo", "10s"
- `api_route_to_call_types.py`: mapping of API routes to their corresponding CallTypes (e.g., `/chat/completions` -> [acompletion, completion])

38
litellm/litellm_core_utils/api_route_to_call_types.py Normal file
View file

@ -0,0 +1,38 @@
"""
Dictionary mapping API routes to their corresponding CallTypes in LiteLLM.
This dictionary maps each API endpoint to the CallTypes that can be used for that route.
Each route can have both async (prefixed with 'a') and sync call types.
"""
from litellm.types.utils import API_ROUTE_TO_CALL_TYPES, CallTypes
def get_call_types_for_route(route: str) -> list:
"""
Get the list of CallTypes for a given API route.
Args:
route: API route path (e.g., "/chat/completions")
Returns:
List of CallTypes for that route, or empty list if route not found
"""
return API_ROUTE_TO_CALL_TYPES.get(route, [])
def get_routes_for_call_type(call_type: CallTypes) -> list:
"""
Get all routes that use a specific CallType.
Args:
call_type: The CallType to search for
Returns:
List of routes that use this CallType
"""
routes = []
for route, types in API_ROUTE_TO_CALL_TYPES.items():
if call_type in types:
routes.append(route)
return routes

10
litellm/litellm_core_utils/get_llm_provider_logic.py
View file

@ -840,6 +840,16 @@ def _get_openai_compatible_provider_info( # noqa: PLR0915
) = litellm.ClarifaiConfig()._get_openai_compatible_provider_info(
api_base, api_key
)
elif custom_llm_provider == "ragflow":
full_model = f"ragflow/{model}"
(
api_base,
dynamic_api_key,
_,
) = litellm.RAGFlowConfig()._get_openai_compatible_provider_info(
full_model, api_base, api_key, "ragflow"
)
model = full_model
if api_base is not None and not isinstance(api_base, str):
raise Exception("api base needs to be a string. api_base={}".format(api_base))

53
litellm/litellm_core_utils/streaming_handler.py
View file

@ -96,9 +96,9 @@ class CustomStreamWrapper:
self.system_fingerprint: Optional[str] = None
self.received_finish_reason: Optional[str] = None
self.intermittent_finish_reason: Optional[str] = (
None # finish reasons that show up mid-stream
)
self.intermittent_finish_reason: Optional[
str
] = None # finish reasons that show up mid-stream
self.special_tokens = [
"<|assistant|>",
"<|system|>",
@ -735,7 +735,7 @@ class CustomStreamWrapper:
and completion_obj["function_call"] is not None
)
or (
"tool_calls" in model_response.choices[0].delta
"tool_calls" in model_response.choices[0].delta
and model_response.choices[0].delta["tool_calls"] is not None
)
or (
@ -889,7 +889,6 @@ class CustomStreamWrapper:
## check if openai/azure chunk
original_chunk = response_obj.get("original_chunk", None)
if original_chunk:
if len(original_chunk.choices) > 0:
choices = []
for choice in original_chunk.choices:
@ -906,7 +905,6 @@ class CustomStreamWrapper:
print_verbose(f"choices in streaming: {choices}")
setattr(model_response, "choices", choices)
else:
return
model_response.system_fingerprint = (
original_chunk.system_fingerprint
@ -1435,9 +1433,9 @@ class CustomStreamWrapper:
_json_delta = delta.model_dump()
print_verbose(f"_json_delta: {_json_delta}")
if "role" not in _json_delta or _json_delta["role"] is None:
_json_delta["role"] = (
"assistant" # mistral's api returns role as None
)
_json_delta[
"role"
] = "assistant" # mistral's api returns role as None
if "tool_calls" in _json_delta and isinstance(
_json_delta["tool_calls"], list
):
@ -1533,7 +1531,7 @@ class CustomStreamWrapper:
async def _call_post_streaming_deployment_hook(self, chunk):
"""
Call the post-call streaming deployment hook for callbacks.
This allows callbacks to modify streaming chunks before they're returned.
"""
try:
@ -1544,15 +1542,17 @@ class CustomStreamWrapper:
# Get request kwargs from logging object
request_data = self.logging_obj.model_call_details
call_type_str = self.logging_obj.call_type
try:
typed_call_type = CallTypes(call_type_str)
except ValueError:
typed_call_type = None
# Call hooks for all callbacks
for callback in litellm.callbacks:
if isinstance(callback, CustomLogger) and hasattr(callback, "async_post_call_streaming_deployment_hook"):
if isinstance(callback, CustomLogger) and hasattr(
callback, "async_post_call_streaming_deployment_hook"
):
result = await callback.async_post_call_streaming_deployment_hook(
request_data=request_data,
response_chunk=chunk,
@ -1560,11 +1560,14 @@ class CustomStreamWrapper:
)
if result is not None:
chunk = result
return chunk
except Exception as e:
from litellm._logging import verbose_logger
verbose_logger.exception(f"Error in post-call streaming deployment hook: {str(e)}")
verbose_logger.exception(
f"Error in post-call streaming deployment hook: {str(e)}"
)
return chunk
def cache_streaming_response(self, processed_chunk, cache_hit: bool):
@ -1687,7 +1690,7 @@ class CustomStreamWrapper:
response, "usage"
): # remove usage from chunk, only send on final chunk
# Convert the object to a dictionary
obj_dict = response.dict()
obj_dict = response.model_dump()
# Remove an attribute (e.g., 'attr2')
if "usage" in obj_dict:
@ -1852,7 +1855,7 @@ class CustomStreamWrapper:
processed_chunk, "usage"
): # remove usage from chunk, only send on final chunk
# Convert the object to a dictionary
obj_dict = processed_chunk.dict()
obj_dict = processed_chunk.model_dump()
# Remove an attribute (e.g., 'attr2')
if "usage" in obj_dict:
@ -1872,11 +1875,15 @@ class CustomStreamWrapper:
if self.sent_last_chunk is True and self.stream_options is None:
usage = calculate_total_usage(chunks=self.chunks)
processed_chunk._hidden_params["usage"] = usage
# Call post-call streaming deployment hook for final chunk
if self.sent_last_chunk is True:
processed_chunk = await self._call_post_streaming_deployment_hook(processed_chunk)
processed_chunk = (
await self._call_post_streaming_deployment_hook(
processed_chunk
)
)
return processed_chunk
raise StopAsyncIteration
else: # temporary patch for non-aiohttp async calls
@ -1890,9 +1897,9 @@ class CustomStreamWrapper:
chunk = next(self.completion_stream)
if chunk is not None and chunk != b"":
print_verbose(f"PROCESSED CHUNK PRE CHUNK CREATOR: {chunk}")
processed_chunk: Optional[ModelResponseStream] = (
self.chunk_creator(chunk=chunk)
)
processed_chunk: Optional[
ModelResponseStream
] = self.chunk_creator(chunk=chunk)
print_verbose(
f"PROCESSED CHUNK POST CHUNK CREATOR: {processed_chunk}"
)

2
litellm/llms/azure/azure.py
View file

@ -1020,7 +1020,7 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
headers: dict,
client=None,
timeout=None,
) -> litellm.ImageResponse:
) -> ImageResponse:
response: Optional[dict] = None
try:

4
litellm/llms/azure_ai/embed/handler.py
View file

@ -58,7 +58,7 @@ class AzureAIEmbedding(OpenAIChatCompletion):
data: ImageEmbeddingRequest,
timeout: float,
logging_obj,
model_response: litellm.EmbeddingResponse,
model_response: EmbeddingResponse,
optional_params: dict,
api_key: Optional[str],
api_base: Optional[str],
@ -138,7 +138,7 @@ class AzureAIEmbedding(OpenAIChatCompletion):
input: List,
timeout: float,
logging_obj,
model_response: litellm.EmbeddingResponse,
model_response: EmbeddingResponse,
optional_params: dict,
api_key: Optional[str] = None,
api_base: Optional[str] = None,

14
litellm/llms/base_llm/guardrail_translation/base_translation.py
View file

@ -84,3 +84,17 @@ class BaseTranslation(ABC):
user_api_key_dict: User API key metadata (passed separately since response doesn't contain it)
"""
pass
async def process_output_streaming_response(
self,
response: Any,
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Optional["LiteLLMLoggingObj"] = None,
user_api_key_dict: Optional["UserAPIKeyAuth"] = None,
) -> Any:
"""
Process output streaming response with guardrails.
Optional to override in subclasses.
"""
return response

10
litellm/llms/bedrock/chat/converse_handler.py
View file

@ -29,6 +29,7 @@ def make_sync_call(
logging_obj: LiteLLMLoggingObject,
json_mode: Optional[bool] = False,
fake_stream: bool = False,
stream_chunk_size: int = 1024,
):
if client is None:
client = _get_httpx_client() # Create a new client if none provided
@ -66,7 +67,7 @@ def make_sync_call(
)
else:
decoder = AWSEventStreamDecoder(model=model)
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=1024))
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=stream_chunk_size))
# LOGGING
logging_obj.post_call(
@ -102,6 +103,7 @@ class BedrockConverseLLM(BaseAWSLLM):
fake_stream: bool = False,
json_mode: Optional[bool] = False,
api_key: Optional[str] = None,
stream_chunk_size: int = 1024,
) -> CustomStreamWrapper:
request_data = await litellm.AmazonConverseConfig()._async_transform_request(
model=model,
@ -143,6 +145,7 @@ class BedrockConverseLLM(BaseAWSLLM):
logging_obj=logging_obj,
fake_stream=fake_stream,
json_mode=json_mode,
stream_chunk_size=stream_chunk_size,
)
streaming_response = CustomStreamWrapper(
completion_stream=completion_stream,
@ -260,6 +263,7 @@ class BedrockConverseLLM(BaseAWSLLM):
):
## SETUP ##
stream = optional_params.pop("stream", None)
stream_chunk_size = optional_params.pop("stream_chunk_size", 1024)
unencoded_model_id = optional_params.pop("model_id", None)
fake_stream = optional_params.pop("fake_stream", False)
json_mode = optional_params.get("json_mode", False)
@ -356,7 +360,8 @@ class BedrockConverseLLM(BaseAWSLLM):
json_mode=json_mode,
fake_stream=fake_stream,
credentials=credentials,
api_key=api_key
api_key=api_key,
stream_chunk_size=stream_chunk_size,
) # type: ignore
### ASYNC COMPLETION
return self.async_completion(
@ -433,6 +438,7 @@ class BedrockConverseLLM(BaseAWSLLM):
logging_obj=logging_obj,
json_mode=json_mode,
fake_stream=fake_stream,
stream_chunk_size=stream_chunk_size,
)
streaming_response = CustomStreamWrapper(
completion_stream=completion_stream,

20
litellm/llms/bedrock/chat/invoke_handler.py
View file

@ -192,6 +192,7 @@ async def make_call(
fake_stream: bool = False,
json_mode: Optional[bool] = False,
bedrock_invoke_provider: Optional[litellm.BEDROCK_INVOKE_PROVIDERS_LITERAL] = None,
stream_chunk_size: int = 1024,
):
try:
if client is None:
@ -235,7 +236,7 @@ async def make_call(
json_mode=json_mode,
)
completion_stream = decoder.aiter_bytes(
response.aiter_bytes(chunk_size=1024)
response.aiter_bytes(chunk_size=stream_chunk_size)
)
elif bedrock_invoke_provider == "deepseek_r1":
decoder = AmazonDeepSeekR1StreamDecoder(
@ -243,12 +244,12 @@ async def make_call(
sync_stream=False,
)
completion_stream = decoder.aiter_bytes(
response.aiter_bytes(chunk_size=1024)
response.aiter_bytes(chunk_size=stream_chunk_size)
)
else:
decoder = AWSEventStreamDecoder(model=model)
completion_stream = decoder.aiter_bytes(
response.aiter_bytes(chunk_size=1024)
response.aiter_bytes(chunk_size=stream_chunk_size)
)
# LOGGING
@ -281,6 +282,7 @@ def make_sync_call(
fake_stream: bool = False,
json_mode: Optional[bool] = False,
bedrock_invoke_provider: Optional[litellm.BEDROCK_INVOKE_PROVIDERS_LITERAL] = None,
stream_chunk_size: int = 1024,
):
try:
if client is None:
@ -321,16 +323,16 @@ def make_sync_call(
sync_stream=True,
json_mode=json_mode,
)
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=1024))
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=stream_chunk_size))
elif bedrock_invoke_provider == "deepseek_r1":
decoder = AmazonDeepSeekR1StreamDecoder(
model=model,
sync_stream=True,
)
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=1024))
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=stream_chunk_size))
else:
decoder = AWSEventStreamDecoder(model=model)
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=1024))
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=stream_chunk_size))
# LOGGING
logging_obj.post_call(
@ -729,6 +731,7 @@ class BedrockLLM(BaseAWSLLM):
## SETUP ##
stream = optional_params.pop("stream", None)
stream_chunk_size = optional_params.pop("stream_chunk_size", 1024)
provider = self.get_bedrock_invoke_provider(model)
modelId = self.get_bedrock_model_id(
@ -1003,6 +1006,7 @@ class BedrockLLM(BaseAWSLLM):
headers=prepped.headers,
timeout=timeout,
client=client,
stream_chunk_size=stream_chunk_size,
) # type: ignore
### ASYNC COMPLETION
return self.async_completion(
@ -1048,7 +1052,7 @@ class BedrockLLM(BaseAWSLLM):
decoder = AWSEventStreamDecoder(model=model)
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=1024))
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=stream_chunk_size))
streaming_response = CustomStreamWrapper(
completion_stream=completion_stream,
model=model,
@ -1168,6 +1172,7 @@ class BedrockLLM(BaseAWSLLM):
logger_fn=None,
headers={},
client: Optional[AsyncHTTPHandler] = None,
stream_chunk_size: int = 1024,
) -> CustomStreamWrapper:
# The call is not made here; instead, we prepare the necessary objects for the stream.
@ -1183,6 +1188,7 @@ class BedrockLLM(BaseAWSLLM):
messages=messages,
logging_obj=logging_obj,
fake_stream=True if "ai21" in api_base else False,
stream_chunk_size=stream_chunk_size,
),
model=model,
custom_llm_provider="bedrock",

3
litellm/llms/bedrock/chat/invoke_transformations/amazon_nova_transformation.py
View file

@ -10,7 +10,6 @@ from typing import Any, List, Optional
import httpx
import litellm
from litellm.litellm_core_utils.litellm_logging import Logging
from litellm.types.llms.bedrock import BedrockInvokeNovaRequest
from litellm.types.llms.openai import AllMessageValues
@ -80,7 +79,7 @@ class AmazonInvokeNovaConfig(AmazonInvokeConfig, AmazonConverseConfig):
encoding: Any,
api_key: Optional[str] = None,
json_mode: Optional[bool] = None,
) -> litellm.ModelResponse:
) -> ModelResponse:
return AmazonConverseConfig.transform_response(
self,
model,

19
litellm/llms/bedrock/common_utils.py
View file

@ -27,6 +27,25 @@ class BedrockError(BaseLLMException):
pass
# Lazy import cache to avoid circular imports and performance impact
_get_model_info = None
def get_cached_model_info():
"""
Lazy import and cache get_model_info to avoid circular imports.
This function is used by bedrock transformation classes that need get_model_info
but cannot import it at module level due to circular import issues.
The function is cached after first use to avoid performance impact.
"""
global _get_model_info
if _get_model_info is None:
from litellm import get_model_info
_get_model_info = get_model_info
return _get_model_info
class AmazonBedrockGlobalConfig:
def __init__(self):
pass

3
litellm/llms/bedrock/image/amazon_nova_canvas_transformation.py
View file

@ -3,7 +3,6 @@ from typing import Any, Dict, List, Optional
from openai.types.image import Image
from litellm import get_model_info
from litellm.types.llms.bedrock import (
AmazonNovaCanvasColorGuidedGenerationParams,
AmazonNovaCanvasColorGuidedRequest,
@ -15,6 +14,7 @@ from litellm.types.llms.bedrock import (
AmazonNovaCanvasTextToImageRequest,
AmazonNovaCanvasTextToImageResponse,
)
from litellm.llms.bedrock.common_utils import get_cached_model_info
from litellm.types.utils import ImageResponse
@ -207,6 +207,7 @@ class AmazonNovaCanvasConfig:
size: Optional[str] = None,
optional_params: Optional[dict] = None,
) -> float:
get_model_info = get_cached_model_info()
model_info = get_model_info(
model=model,
custom_llm_provider="bedrock",

3
litellm/llms/bedrock/image/amazon_stability1_transformation.py
View file

@ -5,7 +5,7 @@ from typing import List, Optional
from openai.types.image import Image
from litellm import get_model_info
from litellm.llms.bedrock.common_utils import get_cached_model_info
from litellm.types.utils import ImageResponse
@ -151,6 +151,7 @@ class AmazonStabilityConfig:
size = size or "1024-x-1024"
model = f"{size}/{steps}/{model}"
get_model_info = get_cached_model_info()
model_info = get_model_info(
model=model,
custom_llm_provider="bedrock",

3
litellm/llms/bedrock/image/amazon_stability3_transformation.py
View file

@ -3,12 +3,12 @@ from typing import List, Optional
from openai.types.image import Image
from litellm import get_model_info
from litellm.llms.bedrock.common_utils import BedrockError
from litellm.types.llms.bedrock import (
AmazonStability3TextToImageRequest,
AmazonStability3TextToImageResponse,
)
from litellm.llms.bedrock.common_utils import get_cached_model_info
from litellm.types.utils import ImageResponse
@ -115,6 +115,7 @@ class AmazonStability3Config:
size: Optional[str] = None,
optional_params: Optional[dict] = None,
) -> float:
get_model_info = get_cached_model_info()
model_info = get_model_info(
model=model,
custom_llm_provider="bedrock",

2
litellm/llms/bedrock/image/amazon_titan_transformation.py
View file

@ -7,7 +7,7 @@ from typing import List, Optional
from openai.types.image import Image
from litellm import get_model_info
from litellm.utils import get_model_info
from litellm.types.llms.bedrock import (
AmazonNovaCanvasImageGenerationConfig,
AmazonTitanImageGenerationRequestBody,

21
litellm/llms/custom_httpx/http_handler.py
View file

@ -16,7 +16,9 @@ from litellm._logging import verbose_logger
from litellm.constants import (
_DEFAULT_TTL_FOR_HTTPX_CLIENTS,
AIOHTTP_CONNECTOR_LIMIT,
AIOHTTP_CONNECTOR_LIMIT_PER_HOST,
AIOHTTP_KEEPALIVE_TIMEOUT,
AIOHTTP_NEEDS_CLEANUP_CLOSED,
AIOHTTP_TTL_DNS_CACHE,
DEFAULT_SSL_CIPHERS,
)
@ -792,15 +794,20 @@ class AsyncHTTPHandler:
verbose_logger.debug(
"NEW SESSION: Creating new ClientSession (no shared session provided)"
)
transport_connector_kwargs = {
"keepalive_timeout": AIOHTTP_KEEPALIVE_TIMEOUT,
"ttl_dns_cache": AIOHTTP_TTL_DNS_CACHE,
"enable_cleanup_closed": True,
**connector_kwargs,
}
if AIOHTTP_CONNECTOR_LIMIT > 0:
transport_connector_kwargs["limit"] = AIOHTTP_CONNECTOR_LIMIT
if AIOHTTP_CONNECTOR_LIMIT_PER_HOST > 0:
transport_connector_kwargs["limit_per_host"] = AIOHTTP_CONNECTOR_LIMIT_PER_HOST
return LiteLLMAiohttpTransport(
client=lambda: ClientSession(
connector=TCPConnector(
limit=AIOHTTP_CONNECTOR_LIMIT,
keepalive_timeout=AIOHTTP_KEEPALIVE_TIMEOUT,
ttl_dns_cache=AIOHTTP_TTL_DNS_CACHE,
enable_cleanup_closed=True,
**connector_kwargs,
),
connector=TCPConnector(**transport_connector_kwargs),
trust_env=trust_env,
),
)

98
litellm/llms/openai/chat/guardrail_translation/handler.py
View file

@ -14,16 +14,16 @@ Pattern Overview:
This pattern can be replicated for other message formats (e.g., Anthropic).
"""
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, cast
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union, cast
import litellm
from litellm._logging import verbose_proxy_logger
from litellm.llms.base_llm.guardrail_translation.base_translation import BaseTranslation
from litellm.types.utils import Choices
from litellm.types.utils import Choices, StreamingChoices
if TYPE_CHECKING:
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.types.utils import ModelResponse
from litellm.types.utils import ModelResponse, ModelResponseStream
class OpenAIChatCompletionsHandler(BaseTranslation):
@ -241,21 +241,79 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
return response
def _has_text_content(self, response: "ModelResponse") -> bool:
async def process_output_streaming_response(
self,
response: "ModelResponseStream",
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Optional[Any] = None,
user_api_key_dict: Optional[Any] = None,
) -> Any:
"""
Process output streaming response by applying guardrails to text content.
Args:
response: LiteLLM ModelResponseStream object
guardrail_to_apply: The guardrail instance to apply
litellm_logging_obj: Optional logging object
user_api_key_dict: User API key metadata to pass to guardrails
Returns:
Modified response with guardrail applied to content
Response Format Support:
- String content: choice.message.content = "text here"
- List content: choice.message.content = [{"type": "text", "text": "text here"}, ...]
"""
# Step 0: Check if response has any text content to process
if not self._has_text_content(response):
return response
texts_to_check: List[str] = []
images_to_check: List[str] = []
task_mappings: List[Tuple[int, Optional[int]]] = []
# Track (choice_index, content_index) for each text
# Step 1: Extract all text content and images from response choices
for choice_idx, choice in enumerate(response.choices):
self._extract_output_text_and_images(
choice=choice,
choice_idx=choice_idx,
texts_to_check=texts_to_check,
images_to_check=images_to_check,
task_mappings=task_mappings,
)
def _has_text_content(
self, response: Union["ModelResponse", "ModelResponseStream"]
) -> bool:
"""
Check if response has any text content to process.
Override this method to customize text content detection.
"""
for choice in response.choices:
if isinstance(choice, litellm.Choices):
if choice.message.content and isinstance(choice.message.content, str):
return True
from litellm.types.utils import ModelResponse, ModelResponseStream
if isinstance(response, ModelResponse):
for choice in response.choices:
if isinstance(choice, litellm.Choices):
if choice.message.content and isinstance(
choice.message.content, str
):
return True
elif isinstance(response, ModelResponseStream):
for choice in response.choices:
if isinstance(choice, litellm.Choices):
if choice.message.content and isinstance(
choice.message.content, str
):
return True
return False
def _extract_output_text_and_images(
self,
choice: Any,
choice: Union[Choices, StreamingChoices],
choice_idx: int,
texts_to_check: List[str],
images_to_check: List[str],
@ -266,21 +324,29 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
Override this method to customize text/image extraction logic.
"""
if not isinstance(choice, litellm.Choices):
return
verbose_proxy_logger.debug(
"OpenAI Chat Completions: Processing choice: %s", choice
)
if choice.message.content and isinstance(choice.message.content, str):
# Determine content source based on choice type
content = None
if isinstance(choice, litellm.Choices):
content = choice.message.content
elif isinstance(choice, litellm.StreamingChoices):
content = choice.delta.content
else:
# Unknown choice type, skip processing
return
# Process content if it exists
if content and isinstance(content, str):
# Simple string content
texts_to_check.append(choice.message.content)
texts_to_check.append(content)
task_mappings.append((choice_idx, None))
elif choice.message.content and isinstance(choice.message.content, list):
elif content and isinstance(content, list):
# List content (e.g., multimodal response)
for content_idx, content_item in enumerate(choice.message.content):
for content_idx, content_item in enumerate(content):
# Extract text
content_text = content_item.get("text")
if content_text:

15
litellm/llms/openrouter/chat/transformation.py
View file

@ -10,6 +10,7 @@ from enum import Enum
from typing import Any, AsyncIterator, Iterator, List, Optional, Tuple, Union, cast
import httpx
import litellm
from litellm.llms.base_llm.base_model_iterator import BaseModelResponseIterator
from litellm.llms.base_llm.chat.transformation import BaseLLMException
@ -28,6 +29,20 @@ class CacheControlSupportedModels(str, Enum):
class OpenrouterConfig(OpenAIGPTConfig):
def get_supported_openai_params(self, model: str) -> list:
"""
Allow reasoning parameters for models flagged as reasoning-capable.
"""
supported_params = super().get_supported_openai_params(model=model)
try:
if litellm.supports_reasoning(
model=model, custom_llm_provider="openrouter"
) or litellm.supports_reasoning(model=model):
supported_params.append("reasoning_effort")
except Exception:
pass
return list(dict.fromkeys(supported_params))
def map_openai_params(
self,
non_default_params: dict,

4
litellm/llms/ovhcloud/chat/transformation.py
View file

@ -7,7 +7,9 @@ More information on our website: https://endpoints.ai.cloud.ovh.net
from typing import Optional, Union, List
import httpx
from litellm import ModelResponseStream, OpenAIGPTConfig, get_model_info, verbose_logger
from litellm.utils import ModelResponseStream, get_model_info
from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
from litellm._logging import verbose_logger
from litellm.llms.ovhcloud.utils import OVHCloudException
from litellm.llms.base_llm.base_model_iterator import BaseModelResponseIterator
from litellm.llms.base_llm.chat.transformation import BaseLLMException

8
litellm/llms/ragflow/__init__.py Normal file
View file

@ -0,0 +1,8 @@
"""
RAGFlow provider for LiteLLM.
RAGFlow provides OpenAI-compatible APIs with unique path structures:
- Chat endpoint: /api/v1/chats_openai/{chat_id}/chat/completions
- Agent endpoint: /api/v1/agents_openai/{agent_id}/chat/completions
"""

4
litellm/llms/ragflow/chat/__init__.py Normal file
View file

@ -0,0 +1,4 @@
"""
RAGFlow chat completion configuration.
"""

264
litellm/llms/ragflow/chat/transformation.py Normal file
View file

@ -0,0 +1,264 @@
"""
RAGFlow provider configuration for OpenAI-compatible API.
RAGFlow provides OpenAI-compatible APIs with unique path structures:
- Chat endpoint: /api/v1/chats_openai/{chat_id}/chat/completions
- Agent endpoint: /api/v1/agents_openai/{agent_id}/chat/completions
Model name format:
- Chat: ragflow/chat/{chat_id}/{model_name}
- Agent: ragflow/agent/{agent_id}/{model_name}
"""
from typing import List, Optional, Tuple
import litellm
from litellm.llms.openai.openai import OpenAIConfig
from litellm.secret_managers.main import get_secret, get_secret_str
from litellm.types.llms.openai import AllMessageValues
class RAGFlowConfig(OpenAIConfig):
"""
Configuration for RAGFlow OpenAI-compatible API.
Handles both chat and agent endpoints by parsing the model name format:
- ragflow/chat/{chat_id}/{model_name} for chat endpoints
- ragflow/agent/{agent_id}/{model_name} for agent endpoints
"""
def _parse_ragflow_model(self, model: str) -> Tuple[str, str, str]:
"""
Parse RAGFlow model name format: ragflow/{endpoint_type}/{id}/{model_name}
Args:
model: Model name in format ragflow/chat/{chat_id}/{model} or ragflow/agent/{agent_id}/{model}
Returns:
Tuple of (endpoint_type, id, model_name)
Raises:
ValueError: If model format is invalid
"""
parts = model.split("/")
if len(parts) < 4:
raise ValueError(
f"Invalid RAGFlow model format: {model}. "
f"Expected format: ragflow/chat/{{chat_id}}/{{model}} or ragflow/agent/{{agent_id}}/{{model}}"
)
if parts[0] != "ragflow":
raise ValueError(
f"Invalid RAGFlow model format: {model}. Must start with 'ragflow/'"
)
endpoint_type = parts[1]
if endpoint_type not in ["chat", "agent"]:
raise ValueError(
f"Invalid RAGFlow endpoint type: {endpoint_type}. Must be 'chat' or 'agent'"
)
entity_id = parts[2]
model_name = "/".join(parts[3:]) # Handle model names that might contain slashes
return endpoint_type, entity_id, model_name
def get_complete_url(
self,
api_base: Optional[str],
api_key: Optional[str],
model: str,
optional_params: dict,
litellm_params: dict,
stream: Optional[bool] = None,
) -> str:
"""
Get the complete URL for the RAGFlow API call.
Constructs URL based on endpoint type:
- Chat: /api/v1/chats_openai/{chat_id}/chat/completions
- Agent: /api/v1/agents_openai/{agent_id}/chat/completions
Args:
api_base: Base API URL (e.g., http://ragflow-server:port or http://ragflow-server:port/v1)
api_key: API key (not used in URL construction)
model: Model name in format ragflow/{endpoint_type}/{id}/{model}
optional_params: Optional parameters
litellm_params: LiteLLM parameters (may contain api_base)
stream: Whether streaming is enabled
Returns:
Complete URL for the API call
"""
# Get api_base from multiple sources: input param, litellm_params, environment, or global litellm setting
if litellm_params and hasattr(litellm_params, 'api_base') and litellm_params.api_base:
api_base = api_base or litellm_params.api_base
api_base = (
api_base
or litellm.api_base
or get_secret("RAGFLOW_API_BASE")
or get_secret_str("RAGFLOW_API_BASE")
)
if api_base is None:
raise ValueError("api_base is required for RAGFlow provider. Set it via api_base parameter, RAGFLOW_API_BASE environment variable, or litellm.api_base")
# Parse model name to extract endpoint type and ID
endpoint_type, entity_id, _ = self._parse_ragflow_model(model)
# Remove trailing slash from api_base if present
api_base = api_base.rstrip("/")
# Strip /v1 or /api/v1 from api_base if present, since we'll add the full path
# Check /api/v1 first because /api/v1 ends with /v1
if api_base.endswith("/api/v1"):
api_base = api_base[:-7] # Remove /api/v1
elif api_base.endswith("/v1"):
api_base = api_base[:-3] # Remove /v1
# Construct the RAGFlow-specific path
if endpoint_type == "chat":
path = f"/api/v1/chats_openai/{entity_id}/chat/completions"
else: # agent
path = f"/api/v1/agents_openai/{entity_id}/chat/completions"
# Ensure path starts with /
if not path.startswith("/"):
path = "/" + path
return f"{api_base}{path}"
def _get_openai_compatible_provider_info(
self,
model: str,
api_base: Optional[str],
api_key: Optional[str],
custom_llm_provider: str,
) -> Tuple[Optional[str], Optional[str], str]:
"""
Get OpenAI-compatible provider information for RAGFlow.
Args:
model: Model name (will be parsed to extract actual model name)
api_base: Base API URL (from input params)
api_key: API key (from input params)
custom_llm_provider: Custom LLM provider name
Returns:
Tuple of (api_base, api_key, custom_llm_provider)
"""
# Parse model to extract the actual model name
# The model name will be stored in litellm_params for use in requests
_, _, actual_model = self._parse_ragflow_model(model)
# Get api_base from multiple sources: input param, environment, or global litellm setting
dynamic_api_base = (
api_base
or litellm.api_base
or get_secret("RAGFLOW_API_BASE")
or get_secret_str("RAGFLOW_API_BASE")
)
# Get api_key from multiple sources: input param, environment, or global litellm setting
dynamic_api_key = (
api_key
or litellm.api_key
or get_secret_str("RAGFLOW_API_KEY")
)
return dynamic_api_base, dynamic_api_key, custom_llm_provider
def validate_environment(
self,
headers: dict,
model: str,
messages: List[AllMessageValues],
optional_params: dict,
litellm_params: dict,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
) -> dict:
"""
Validate environment and set up headers for RAGFlow API.
Args:
headers: Request headers
model: Model name
messages: Chat messages
optional_params: Optional parameters
litellm_params: LiteLLM parameters (may contain api_key)
api_key: API key (from input params)
api_base: Base API URL
Returns:
Updated headers dictionary
"""
# Use api_key from litellm_params if available, otherwise fall back to other sources
if litellm_params and hasattr(litellm_params, 'api_key') and litellm_params.api_key:
api_key = api_key or litellm_params.api_key
# Get api_key from multiple sources: input param, litellm_params, environment, or global litellm setting
api_key = (
api_key
or litellm.api_key
or get_secret_str("RAGFLOW_API_KEY")
)
if api_key is not None:
headers["Authorization"] = f"Bearer {api_key}"
# Ensure Content-Type is set to application/json
if "content-type" not in headers and "Content-Type" not in headers:
headers["Content-Type"] = "application/json"
# Parse model to extract actual model name and store it
# The actual model name should be used in the request body
try:
_, _, actual_model = self._parse_ragflow_model(model)
# Store the actual model name in litellm_params for use in transform_request
litellm_params["_ragflow_actual_model"] = actual_model
except ValueError:
# If parsing fails, use the original model name
pass
return headers
def transform_request(
self,
model: str,
messages: List[AllMessageValues],
optional_params: dict,
litellm_params: dict,
headers: dict,
) -> dict:
"""
Transform request for RAGFlow API.
Uses the actual model name extracted from the RAGFlow model format.
Args:
model: Model name in RAGFlow format
messages: Chat messages
optional_params: Optional parameters
litellm_params: LiteLLM parameters (may contain _ragflow_actual_model)
headers: Request headers
Returns:
Transformed request dictionary
"""
# Get the actual model name from litellm_params if available
actual_model = litellm_params.get("_ragflow_actual_model")
if actual_model is None:
# Fallback: try to parse the model name
try:
_, _, actual_model = self._parse_ragflow_model(model)
except ValueError:
# If parsing fails, use the original model name
actual_model = model
# Use parent's transform_request with the actual model name
return super().transform_request(
actual_model, messages, optional_params, litellm_params, headers
)

2
litellm/llms/ragflow/vector_stores/__init__.py Normal file
View file

@ -0,0 +1,2 @@
# RAGFlow vector stores module

249
litellm/llms/ragflow/vector_stores/transformation.py Normal file
View file

@ -0,0 +1,249 @@
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
import httpx
from litellm.llms.base_llm.vector_store.transformation import BaseVectorStoreConfig
from litellm.secret_managers.main import get_secret_str
from litellm.types.router import GenericLiteLLMParams
from litellm.types.vector_stores import (
BaseVectorStoreAuthCredentials,
VectorStoreCreateOptionalRequestParams,
VectorStoreCreateResponse,
VectorStoreFileCounts,
VectorStoreIndexEndpoints,
VectorStoreSearchOptionalRequestParams,
VectorStoreSearchResponse,
)
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
else:
LiteLLMLoggingObj = Any
class RAGFlowVectorStoreConfig(BaseVectorStoreConfig):
"""Vector store configuration for RAGFlow datasets."""
def get_auth_credentials(
self, litellm_params: dict
) -> BaseVectorStoreAuthCredentials:
api_key = litellm_params.get("api_key")
if api_key is None:
# Try to get from environment variable
api_key = get_secret_str("RAGFLOW_API_KEY")
if api_key is None:
raise ValueError("api_key is required (set RAGFLOW_API_KEY env var or pass in litellm_params)")
return {
"headers": {
"Authorization": f"Bearer {api_key}",
},
}
def get_vector_store_endpoints_by_type(self) -> VectorStoreIndexEndpoints:
"""RAGFlow vector stores are management-only, no search support."""
return {
"read": [],
"write": [],
}
def validate_environment(
self, headers: dict, litellm_params: Optional[GenericLiteLLMParams]
) -> dict:
"""Validate environment and set headers for RAGFlow API."""
litellm_params = litellm_params or GenericLiteLLMParams()
api_key = (
litellm_params.api_key
or get_secret_str("RAGFLOW_API_KEY")
)
if api_key is None:
raise ValueError("RAGFLOW_API_KEY is required (set env var or pass in litellm_params)")
headers.update(
{
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
}
)
return headers
def get_complete_url(
self,
api_base: Optional[str],
litellm_params: dict,
) -> str:
"""
Get the complete URL for RAGFlow datasets API.
Supports:
- RAGFLOW_API_BASE env var
- api_base in litellm_params
- Default: http://localhost:9380
"""
api_base = (
api_base
or litellm_params.get("api_base")
or get_secret_str("RAGFLOW_API_BASE")
or "http://localhost:9380"
)
# Remove trailing slashes
api_base = api_base.rstrip("/")
# RAGFlow datasets API endpoint
return f"{api_base}/api/v1/datasets"
def transform_search_vector_store_request(
self,
vector_store_id: str,
query: Union[str, List[str]],
vector_store_search_optional_params: VectorStoreSearchOptionalRequestParams,
api_base: str,
litellm_logging_obj: LiteLLMLoggingObj,
litellm_params: dict,
) -> Tuple[str, Dict]:
"""RAGFlow vector stores are management-only, search is not supported."""
raise NotImplementedError(
"RAGFlow vector stores support dataset management only, not search/retrieval"
)
def transform_search_vector_store_response(
self, response: httpx.Response, litellm_logging_obj: LiteLLMLoggingObj
) -> VectorStoreSearchResponse:
"""RAGFlow vector stores are management-only, search is not supported."""
raise NotImplementedError(
"RAGFlow vector stores support dataset management only, not search/retrieval"
)
def transform_create_vector_store_request(
self,
vector_store_create_optional_params: VectorStoreCreateOptionalRequestParams,
api_base: str,
) -> Tuple[str, Dict]:
"""
Transform create request to RAGFlow POST /api/v1/datasets format.
Maps LiteLLM params to RAGFlow dataset creation parameters.
RAGFlow-specific fields can be passed via metadata.
"""
url = api_base # Already includes /api/v1/datasets from get_complete_url
# Extract name (required by RAGFlow)
name = vector_store_create_optional_params.get("name")
if not name:
raise ValueError("name is required for RAGFlow dataset creation")
# Build request body
request_body: Dict[str, Any] = {
"name": name,
}
# Extract RAGFlow-specific fields from metadata
metadata = vector_store_create_optional_params.get("metadata")
if metadata:
# RAGFlow-specific fields that can be in metadata
ragflow_fields = [
"avatar",
"description",
"embedding_model",
"permission",
"chunk_method",
"parser_config",
"parse_type",
"pipeline_id",
]
for field in ragflow_fields:
if field in metadata:
request_body[field] = metadata[field]
# Validate: chunk_method and pipeline_id are mutually exclusive
if "chunk_method" in request_body and "pipeline_id" in request_body:
raise ValueError(
"chunk_method and pipeline_id are mutually exclusive. "
"Specify either chunk_method or pipeline_id, not both."
)
# If neither chunk_method nor pipeline_id is specified, default to naive
if "chunk_method" not in request_body and "pipeline_id" not in request_body:
request_body["chunk_method"] = "naive"
return url, request_body
def transform_create_vector_store_response(
self, response: httpx.Response
) -> VectorStoreCreateResponse:
"""
Transform RAGFlow response to VectorStoreCreateResponse format.
RAGFlow response format:
{
"code": 0,
"data": {
"id": "...",
"name": "...",
"create_time": 1745836841611, # milliseconds
...
}
}
"""
try:
response_json = response.json()
# Check for RAGFlow error response
if response_json.get("code") != 0:
error_message = response_json.get("message", "Unknown error")
raise self.get_error_class(
error_message=error_message,
status_code=response.status_code,
headers=response.headers,
)
data = response_json.get("data", {})
# Extract dataset ID
dataset_id = data.get("id")
if not dataset_id:
raise ValueError("RAGFlow response missing dataset id")
# Extract name
name = data.get("name")
# Convert create_time from milliseconds to seconds (Unix timestamp)
create_time_ms = data.get("create_time", 0)
created_at = int(create_time_ms / 1000) if create_time_ms else None
# Build VectorStoreCreateResponse
return VectorStoreCreateResponse(
id=dataset_id,
object="vector_store",
created_at=created_at or 0,
name=name,
bytes=0, # RAGFlow doesn't provide bytes in response
file_counts=VectorStoreFileCounts(
in_progress=0,
completed=0,
failed=0,
cancelled=0,
total=0,
),
status="completed",
expires_after=None,
expires_at=None,
last_active_at=None,
metadata=None,
)
except Exception as e:
# If it's already a ValueError we raised, re-raise it
if isinstance(e, ValueError) and "RAGFlow response" in str(e):
raise
# If it's already our error class (has status_code), re-raise
if hasattr(e, "status_code"):
raise
# Otherwise, wrap in our error class
raise self.get_error_class(
error_message=str(e),
status_code=response.status_code,
headers=response.headers,
)

3
litellm/llms/together_ai/chat.py
View file

@ -8,7 +8,8 @@ Docs: https://docs.together.ai/reference/completions-1
from typing import Optional
from litellm import get_model_info, verbose_logger
from litellm.utils import get_model_info
from litellm._logging import verbose_logger
from ..openai.chat.gpt_transformation import OpenAIGPTConfig

3
litellm/llms/vertex_ai/common_utils.py
View file

@ -5,7 +5,8 @@ from typing import Any, Dict, List, Literal, Optional, Set, Tuple, Union, get_ty
import httpx
import litellm
from litellm import supports_response_schema, supports_system_messages, verbose_logger
from litellm.utils import supports_response_schema, supports_system_messages
from litellm._logging import verbose_logger
from litellm.constants import DEFAULT_MAX_RECURSE_DEPTH
from litellm.litellm_core_utils.prompt_templates.common_utils import unpack_defs
from litellm.llms.base_llm.base_utils import BaseLLMModelInfo, BaseTokenCounter

17
litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
View file

@ -1091,6 +1091,11 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
if "thoughtSignature" in part:
part_copy = part.copy()
part_copy.pop("thoughtSignature")
text_content = part_copy.get("text")
if isinstance(text_content, str) and text_content.strip() == "":
continue
thinking_blocks.append(
ChatCompletionThinkingBlock(
type="thinking",
@ -1205,14 +1210,16 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
}
# Embed thought signature in ID for OpenAI client compatibility
if thought_signature:
_tool_response_chunk[
"id"
] = _encode_tool_call_id_with_signature(
_tool_response_chunk["id"] or "", thought_signature
)
_tool_response_chunk["provider_specific_fields"] = { # type: ignore
"thought_signature": thought_signature
}
# Only embed in ID if preview features are enabled
if litellm.enable_preview_features:
_tool_response_chunk[
"id"
] = _encode_tool_call_id_with_signature(
_tool_response_chunk["id"] or "", thought_signature
)
_tools.append(_tool_response_chunk)
cumulative_tool_call_idx += 1
if len(_tools) == 0:

2
litellm/llms/vertex_ai/gemini_embeddings/batch_embed_content_handler.py
View file

@ -8,7 +8,7 @@ from typing import Any, Literal, Optional, Union
import httpx
import litellm
from litellm import EmbeddingResponse
from litellm.types.utils import EmbeddingResponse
from litellm.llms.custom_httpx.http_handler import (
AsyncHTTPHandler,
HTTPHandler,

2
litellm/llms/vertex_ai/gemini_embeddings/batch_embed_content_transformation.py
View file

@ -6,7 +6,7 @@ Why separate file? Make it easy to see how transformation works
from typing import List
from litellm import EmbeddingResponse
from litellm.types.utils import EmbeddingResponse
from litellm.types.llms.openai import EmbeddingInput
from litellm.types.llms.vertex_ai import (
ContentType,

2
litellm/llms/vertex_ai/image_generation/image_generation_handler.py
View file

@ -176,7 +176,7 @@ class VertexImageGeneration(VertexLLM):
vertex_project: Optional[str],
vertex_location: Optional[str],
vertex_credentials: Optional[VERTEX_CREDENTIALS_TYPES],
model_response: litellm.ImageResponse,
model_response: ImageResponse,
logging_obj: Any,
model: str = "imagegeneration", # vertex ai uses imagegeneration as the default model
client: Optional[AsyncHTTPHandler] = None,

4
litellm/llms/vertex_ai/multimodal_embeddings/embedding_handler.py
View file

@ -147,13 +147,13 @@ class VertexMultimodalEmbedding(VertexLLM):
optional_params: dict,
litellm_params: dict,
data: dict,
model_response: litellm.EmbeddingResponse,
model_response: EmbeddingResponse,
timeout: Optional[Union[float, httpx.Timeout]],
logging_obj: LiteLLMLoggingObj,
headers={},
client: Optional[AsyncHTTPHandler] = None,
api_key: Optional[str] = None,
) -> litellm.EmbeddingResponse:
) -> EmbeddingResponse:
if client is None:
_params = {}
if timeout is not None:

4
litellm/llms/vertex_ai/vertex_embeddings/embedding_handler.py
View file

@ -137,7 +137,7 @@ class VertexEmbedding(VertexBase):
self,
model: str,
input: Union[list, str],
model_response: litellm.EmbeddingResponse,
model_response: EmbeddingResponse,
logging_obj: LiteLLMLoggingObject,
optional_params: dict,
custom_llm_provider: Literal[
@ -152,7 +152,7 @@ class VertexEmbedding(VertexBase):
gemini_api_key: Optional[str] = None,
extra_headers: Optional[dict] = None,
encoding=None,
) -> litellm.EmbeddingResponse:
) -> EmbeddingResponse:
"""
Async embedding implementation
"""

41
litellm/main.py
View file

@ -52,13 +52,10 @@ from pydantic import BaseModel
from typing_extensions import overload
import litellm
from litellm import ( # type: ignore
client,
exception_type,
get_litellm_params,
get_optional_params,
)
# client must be imported from litellm as it's a decorator used at function definition time
from litellm import client
# Other utils are imported directly to avoid circular imports
from litellm.utils import exception_type, get_litellm_params, get_optional_params
# Logging is imported lazily when needed to avoid loading litellm_logging at import time
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging
@ -1988,6 +1985,36 @@ def completion( # type: ignore # noqa: PLR0915
)
raise e
elif custom_llm_provider == "ragflow":
## COMPLETION CALL - RAGFlow uses HTTP handler to support custom URL paths
try:
response = base_llm_http_handler.completion(
model=model,
messages=messages,
headers=headers,
model_response=model_response,
api_key=api_key,
api_base=api_base,
acompletion=acompletion,
logging_obj=logging,
optional_params=optional_params,
litellm_params=litellm_params,
shared_session=shared_session,
timeout=timeout,
client=client,
custom_llm_provider=custom_llm_provider,
encoding=encoding,
stream=stream,
provider_config=provider_config,
)
except Exception as e:
logging.post_call(
input=messages,
api_key=api_key,
original_response=str(e),
additional_args={"headers": headers},
)
raise e
elif custom_llm_provider == "xai":
## COMPLETION CALL
try:

209
litellm/model_prices_and_context_window_backup.json
View file

@ -269,6 +269,71 @@
"supports_response_schema": true,
"supports_vision": true
},
"amazon.nova-2-lite-v1:0": {
"input_cost_per_token": 3e-07,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 2.5e-06,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_video_input": true,
"supports_vision": true
},
"apac.amazon.nova-2-lite-v1:0": {
"input_cost_per_token": 6e-08,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 2.75e-06,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_video_input": true,
"supports_vision": true
},
"eu.amazon.nova-2-lite-v1:0": {
"input_cost_per_token": 6e-08,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 2.75e-06,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_video_input": true,
"supports_vision": true
},
"us.amazon.nova-2-lite-v1:0": {
"input_cost_per_token": 6e-08,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 2.75e-06,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_video_input": true,
"supports_vision": true
},
"amazon.nova-micro-v1:0": {
"input_cost_per_token": 3.5e-08,
"litellm_provider": "bedrock_converse",
@ -9564,6 +9629,21 @@
"supports_prompt_caching": true,
"supports_tool_choice": true
},
"deepseek/deepseek-v3.2": {
"input_cost_per_token": 2.8e-07,
"input_cost_per_token_cache_hit": 2.8e-08,
"litellm_provider": "deepseek",
"max_input_tokens": 163840,
"max_output_tokens": 163840,
"max_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 4e-07,
"supports_assistant_prefill": true,
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_tool_choice": true
},
"deepseek.v3-v1:0": {
"input_cost_per_token": 5.8e-07,
"litellm_provider": "bedrock_converse",
@ -10772,25 +10852,25 @@
"supports_tool_choice": true
},
"ft:babbage-002": {
"input_cost_per_token": 4e-07,
"input_cost_per_token": 1.6e-06,
"input_cost_per_token_batches": 2e-07,
"litellm_provider": "text-completion-openai",
"max_input_tokens": 16384,
"max_output_tokens": 4096,
"max_tokens": 16384,
"mode": "completion",
"output_cost_per_token": 4e-07,
"output_cost_per_token": 1.6e-06,
"output_cost_per_token_batches": 2e-07
},
"ft:davinci-002": {
"input_cost_per_token": 2e-06,
"input_cost_per_token": 1.2e-05,
"input_cost_per_token_batches": 1e-06,
"litellm_provider": "text-completion-openai",
"max_input_tokens": 16384,
"max_output_tokens": 4096,
"max_tokens": 16384,
"mode": "completion",
"output_cost_per_token": 2e-06,
"output_cost_per_token": 1.2e-05,
"output_cost_per_token_batches": 1e-06
},
"ft:gpt-3.5-turbo": {
@ -10853,6 +10933,7 @@
"supports_tool_choice": true
},
"ft:gpt-4o-2024-08-06": {
"cache_read_input_token_cost": 1.875e-06,
"input_cost_per_token": 3.75e-06,
"input_cost_per_token_batches": 1.875e-06,
"litellm_provider": "openai",
@ -10865,6 +10946,7 @@
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
@ -10885,8 +10967,7 @@
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true
"supports_tool_choice": true
},
"ft:gpt-4o-mini-2024-07-18": {
"cache_read_input_token_cost": 1.5e-07,
@ -10905,8 +10986,79 @@
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true
"supports_tool_choice": true
},
"ft:gpt-4.1-2025-04-14": {
"cache_read_input_token_cost": 7.5e-07,
"input_cost_per_token": 3e-06,
"input_cost_per_token_batches": 1.5e-06,
"litellm_provider": "openai",
"max_input_tokens": 1047576,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "chat",
"output_cost_per_token": 1.2e-05,
"output_cost_per_token_batches": 6e-06,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"ft:gpt-4.1-mini-2025-04-14": {
"cache_read_input_token_cost": 2e-07,
"input_cost_per_token": 8e-07,
"input_cost_per_token_batches": 4e-07,
"litellm_provider": "openai",
"max_input_tokens": 1047576,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "chat",
"output_cost_per_token": 3.2e-06,
"output_cost_per_token_batches": 1.6e-06,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"ft:gpt-4.1-nano-2025-04-14": {
"cache_read_input_token_cost": 5e-08,
"input_cost_per_token": 2e-07,
"input_cost_per_token_batches": 1e-07,
"litellm_provider": "openai",
"max_input_tokens": 1047576,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "chat",
"output_cost_per_token": 8e-07,
"output_cost_per_token_batches": 4e-07,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"ft:o4-mini-2025-04-16": {
"cache_read_input_token_cost": 1e-06,
"input_cost_per_token": 4e-06,
"input_cost_per_token_batches": 2e-06,
"litellm_provider": "openai",
"max_input_tokens": 200000,
"max_output_tokens": 100000,
"max_tokens": 100000,
"mode": "chat",
"output_cost_per_token": 1.6e-05,
"output_cost_per_token_batches": 8e-06,
"supports_function_calling": true,
"supports_parallel_function_calling": false,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true
},
"gemini-1.0-pro": {
"input_cost_per_character": 1.25e-07,
@ -20500,6 +20652,21 @@
"supports_reasoning": true,
"supports_tool_choice": true
},
"openrouter/deepseek/deepseek-v3.2": {
"input_cost_per_token": 2.8e-07,
"input_cost_per_token_cache_hit": 2.8e-08,
"litellm_provider": "openrouter",
"max_input_tokens": 163840,
"max_output_tokens": 163840,
"max_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 4e-07,
"supports_assistant_prefill": true,
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_tool_choice": true
},
"openrouter/deepseek/deepseek-v3.2-exp": {
"input_cost_per_token": 2e-07,
"input_cost_per_token_cache_hit": 2e-08,
@ -23755,6 +23922,32 @@
"supports_vision": true,
"tool_use_system_prompt_tokens": 159
},
"global.anthropic.claude-opus-4-5-20251101-v1:0": {
"cache_creation_input_token_cost": 6.25e-06,
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 2.5e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_assistant_prefill": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true,
"tool_use_system_prompt_tokens": 159
},
"us.anthropic.claude-sonnet-4-20250514-v1:0": {
"cache_creation_input_token_cost": 3.75e-06,
"cache_read_input_token_cost": 3e-07,

1
litellm/proxy/_types.py
View file

@ -2444,6 +2444,7 @@ class CallInfo(LiteLLMPydanticObjectBase):
user_id: Optional[str] = None
team_id: Optional[str] = None
team_alias: Optional[str] = None
organization_id: Optional[str] = None
user_email: Optional[str] = None
key_alias: Optional[str] = None
projected_exceeded_date: Optional[str] = None

83
litellm/proxy/auth/auth_checks.py
View file

@ -143,6 +143,14 @@ async def common_checks(
valid_token=valid_token,
)
# 3.1. If organization is in budget
await _organization_max_budget_check(
valid_token=valid_token,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
proxy_logging_obj=proxy_logging_obj,
)
await _tag_max_budget_check(
request_body=request_body,
prisma_client=prisma_client,
@ -182,7 +190,18 @@ async def common_checks(
general_settings.get("enforce_user_param", None) is not None
and general_settings["enforce_user_param"] is True
):
if RouteChecks.is_llm_api_route(route=route) and "user" not in request_body:
# Get HTTP method from request
http_method = request.method if hasattr(request, 'method') else None
# Check if it's a POST request and if it's an OpenAI route but not MCP
is_post_method = http_method and http_method.upper() == "POST"
is_openai_route = RouteChecks.is_llm_api_route(route=route)
is_mcp_route = route in LiteLLMRoutes.mcp_routes.value or RouteChecks.check_route_access(
route=route, allowed_routes=LiteLLMRoutes.mcp_routes.value
)
# Enforce user param only for POST requests on OpenAI routes (excluding MCP routes)
if is_post_method and is_openai_route and not is_mcp_route and "user" not in request_body:
raise Exception(
f"'user' param not passed in. 'enforce_user_param'={general_settings['enforce_user_param']}"
)
@ -1893,6 +1912,7 @@ async def _virtual_key_max_budget_check(
max_budget=valid_token.max_budget,
user_id=valid_token.user_id,
team_id=valid_token.team_id,
organization_id=valid_token.org_id,
user_email=user_email,
key_alias=valid_token.key_alias,
event_group=Litellm_EntityType.KEY,
@ -1939,6 +1959,7 @@ async def _virtual_key_soft_budget_check(
user_id=valid_token.user_id,
team_id=valid_token.team_id,
team_alias=valid_token.team_alias,
organization_id=valid_token.org_id,
user_email=None,
key_alias=valid_token.key_alias,
event_group=Litellm_EntityType.KEY,
@ -1977,6 +1998,7 @@ async def _team_max_budget_check(
user_id=valid_token.user_id,
team_id=valid_token.team_id,
team_alias=valid_token.team_alias,
organization_id=valid_token.org_id,
event_group=Litellm_EntityType.TEAM,
)
asyncio.create_task(
@ -1993,6 +2015,65 @@ async def _team_max_budget_check(
)
async def _organization_max_budget_check(
valid_token: Optional[UserAPIKeyAuth],
prisma_client: Optional[PrismaClient],
user_api_key_cache: DualCache,
proxy_logging_obj: ProxyLogging,
):
"""
Check if the organization is over its max budget.
Raises:
BudgetExceededError if the organization is over its max budget.
Triggers a budget alert if the organization is over its max budget.
"""
# Only check if token has organization info and organization_max_budget is set
if (
valid_token is None
or valid_token.org_id is None
or valid_token.organization_max_budget is None
or valid_token.organization_max_budget <= 0
):
return
# Get organization object to check current spend
if prisma_client is not None:
org_table = await get_org_object(
org_id=valid_token.org_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
)
if (
org_table is not None
and org_table.spend >= valid_token.organization_max_budget
):
# Trigger budget alert
call_info = CallInfo(
token=valid_token.token,
spend=org_table.spend,
max_budget=valid_token.organization_max_budget,
user_id=valid_token.user_id,
team_id=valid_token.team_id,
team_alias=valid_token.team_alias,
organization_id=valid_token.org_id,
event_group=Litellm_EntityType.ORGANIZATION,
)
asyncio.create_task(
proxy_logging_obj.budget_alerts(
type="organization_budget",
user_info=call_info,
)
)
raise litellm.BudgetExceededError(
current_cost=org_table.spend,
max_budget=valid_token.organization_max_budget,
message=f"Budget has been exceeded! Organization={valid_token.org_id} Current cost: {org_table.spend}, Max budget: {valid_token.organization_max_budget}",
)
async def _tag_max_budget_check(
request_body: dict,
prisma_client: Optional[PrismaClient],

337
litellm/proxy/auth/login_utils.py Normal file
View file

@ -0,0 +1,337 @@
"""
Login utilities for handling user authentication in the proxy server.
This module contains the core login logic that can be reused across different
login endpoints (e.g., /login and /v2/login).
"""
import os
import secrets
from typing import Literal, Optional, cast
import litellm
from fastapi import HTTPException
from litellm.constants import LITELLM_PROXY_ADMIN_NAME
from litellm.proxy._types import (
LiteLLM_UserTable,
LitellmUserRoles,
ProxyErrorTypes,
ProxyException,
UpdateUserRequest,
UserAPIKeyAuth,
hash_token,
)
from litellm.proxy.management_endpoints.internal_user_endpoints import user_update
from litellm.proxy.management_endpoints.key_management_endpoints import (
generate_key_helper_fn,
)
from litellm.proxy.management_endpoints.ui_sso import (
get_disabled_non_admin_personal_key_creation,
)
from litellm.proxy.utils import PrismaClient, get_server_root_path
from litellm.secret_managers.main import get_secret_bool
from litellm.types.proxy.ui_sso import ReturnedUITokenObject
def get_ui_credentials(master_key: Optional[str]) -> tuple[str, str]:
"""
Get UI username and password from environment variables or master key.
Args:
master_key: Master key for the proxy (used as fallback for password)
Returns:
tuple[str, str]: A tuple containing (ui_username, ui_password)
Raises:
ProxyException: If neither UI_PASSWORD nor master_key is available
"""
ui_username = os.getenv("UI_USERNAME", "admin")
ui_password = os.getenv("UI_PASSWORD", None)
if ui_password is None:
ui_password = str(master_key) if master_key is not None else None
if ui_password is None:
raise ProxyException(
message="set Proxy master key to use UI. https://docs.litellm.ai/docs/proxy/virtual_keys. If set, use `--detailed_debug` to debug issue.",
type=ProxyErrorTypes.auth_error,
param="UI_PASSWORD",
code=500,
)
return ui_username, ui_password
class LoginResult:
"""Result object containing authentication data from login."""
def __init__(
self,
user_id: str,
key: str,
user_email: Optional[str],
user_role: str,
login_method: str = "username_password",
):
self.user_id = user_id
self.key = key
self.user_email = user_email
self.user_role = user_role
self.login_method = login_method
async def authenticate_user(
username: str,
password: str,
master_key: Optional[str],
prisma_client: Optional[PrismaClient],
) -> LoginResult:
"""
Authenticate a user and generate an API key for UI access.
This function handles two login scenarios:
1. Admin login using UI_USERNAME and UI_PASSWORD
2. User login using email and password from database
Args:
username: Username or email from the login form
password: Password from the login form
master_key: Master key for the proxy (required)
prisma_client: Prisma database client (optional)
Returns:
LoginResult: Object containing authentication data
Raises:
ProxyException: If authentication fails or required configuration is missing
"""
if master_key is None:
raise ProxyException(
message="Master Key not set for Proxy. Please set Master Key to use Admin UI. Set `LITELLM_MASTER_KEY` in .env or set general_settings:master_key in config.yaml. https://docs.litellm.ai/docs/proxy/virtual_keys. If set, use `--detailed_debug` to debug issue.",
type=ProxyErrorTypes.auth_error,
param="master_key",
code=500,
)
ui_username, ui_password = get_ui_credentials(master_key)
# Check if we can find the `username` in the db. On the UI, users can enter username=their email
_user_row: Optional[LiteLLM_UserTable] = None
user_role: Optional[
Literal[
LitellmUserRoles.PROXY_ADMIN,
LitellmUserRoles.PROXY_ADMIN_VIEW_ONLY,
LitellmUserRoles.INTERNAL_USER,
LitellmUserRoles.INTERNAL_USER_VIEW_ONLY,
]
] = None
if prisma_client is not None:
_user_row = cast(
Optional[LiteLLM_UserTable],
await prisma_client.db.litellm_usertable.find_first(
where={"user_email": {"equals": username}}
),
)
"""
To login to Admin UI, we support the following
- Login with UI_USERNAME and UI_PASSWORD
- Login with Invite Link `user_email` and `password` combination
"""
if secrets.compare_digest(username, ui_username) and secrets.compare_digest(
password, ui_password
):
# Non SSO -> If user is using UI_USERNAME and UI_PASSWORD they are Proxy admin
user_role = LitellmUserRoles.PROXY_ADMIN
user_id = LITELLM_PROXY_ADMIN_NAME
# we want the key created to have PROXY_ADMIN_PERMISSIONS
key_user_id = LITELLM_PROXY_ADMIN_NAME
if (
os.getenv("PROXY_ADMIN_ID", None) is not None
and os.environ["PROXY_ADMIN_ID"] == user_id
) or user_id == LITELLM_PROXY_ADMIN_NAME:
# checks if user is admin
key_user_id = os.getenv("PROXY_ADMIN_ID", LITELLM_PROXY_ADMIN_NAME)
# Admin is Authe'd in - generate key for the UI to access Proxy
# ensure this user is set as the proxy admin, in this route there is no sso, we can assume this user is only the admin
await user_update(
data=UpdateUserRequest(
user_id=key_user_id,
user_role=user_role,
),
user_api_key_dict=UserAPIKeyAuth(
user_role=LitellmUserRoles.PROXY_ADMIN,
),
)
if os.getenv("DATABASE_URL") is not None:
response = await generate_key_helper_fn(
request_type="key",
**{
"user_role": LitellmUserRoles.PROXY_ADMIN,
"duration": "24hr",
"key_max_budget": litellm.max_ui_session_budget,
"models": [],
"aliases": {},
"config": {},
"spend": 0,
"user_id": key_user_id,
"team_id": "litellm-dashboard",
}, # type: ignore
)
else:
raise ProxyException(
message="No Database connected. Set DATABASE_URL in .env. If set, use `--detailed_debug` to debug issue.",
type=ProxyErrorTypes.auth_error,
param="DATABASE_URL",
code=500,
)
key = response["token"] # type: ignore
if get_secret_bool("EXPERIMENTAL_UI_LOGIN"):
user_info: Optional[LiteLLM_UserTable] = None
if _user_row is not None:
user_info = _user_row
elif (
user_id is not None
): # if user_id is not None, we are using the UI_USERNAME and UI_PASSWORD
from litellm.proxy.auth.auth_checks import ExperimentalUIJWTToken
user_info = LiteLLM_UserTable(
user_id=user_id,
user_role=user_role,
models=[],
max_budget=litellm.max_ui_session_budget,
)
if user_info is None:
raise HTTPException(
status_code=401,
detail={
"error": "User Information is required for experimental UI login"
},
)
key = ExperimentalUIJWTToken.get_experimental_ui_login_jwt_auth_token(
user_info
)
return LoginResult(
user_id=user_id,
key=key,
user_email=None,
user_role=user_role,
login_method="username_password",
)
elif _user_row is not None:
"""
When sharing invite links
-> if the user has no role in the DB assume they are only a viewer
"""
user_id = getattr(_user_row, "user_id", "unknown")
user_role = getattr(
_user_row, "user_role", LitellmUserRoles.INTERNAL_USER_VIEW_ONLY
)
user_email = getattr(_user_row, "user_email", "unknown")
_password = getattr(_user_row, "password", "unknown")
if _password is None:
raise ProxyException(
message="User has no password set. Please set a password for the user via `/user/update`.",
type=ProxyErrorTypes.auth_error,
param="password",
code=401,
)
# check if password == _user_row.password
hash_password = hash_token(token=password)
if secrets.compare_digest(password, _password) or secrets.compare_digest(
hash_password, _password
):
if os.getenv("DATABASE_URL") is not None:
response = await generate_key_helper_fn(
request_type="key",
**{ # type: ignore
"user_role": user_role,
"duration": "24hr",
"key_max_budget": litellm.max_ui_session_budget,
"models": [],
"aliases": {},
"config": {},
"spend": 0,
"user_id": user_id,
"team_id": "litellm-dashboard",
},
)
else:
raise ProxyException(
message="No Database connected. Set DATABASE_URL in .env. If set, use `--detailed_debug` to debug issue.",
type=ProxyErrorTypes.auth_error,
param="DATABASE_URL",
code=500,
)
key = response["token"] # type: ignore
return LoginResult(
user_id=user_id,
key=key,
user_email=user_email,
user_role=cast(str, user_role),
login_method="username_password",
)
else:
raise ProxyException(
message=f"Invalid credentials used to access UI.\nNot valid credentials for {username}",
type=ProxyErrorTypes.auth_error,
param="invalid_credentials",
code=401,
)
else:
raise ProxyException(
message="Invalid credentials used to access UI.\nCheck 'UI_USERNAME', 'UI_PASSWORD' in .env file",
type=ProxyErrorTypes.auth_error,
param="invalid_credentials",
code=401,
)
def create_ui_token_object(
login_result: LoginResult,
general_settings: dict,
premium_user: bool,
) -> ReturnedUITokenObject:
"""
Create a ReturnedUITokenObject from a LoginResult.
Args:
login_result: The result from authenticate_user
general_settings: General proxy settings dictionary
premium_user: Whether premium features are enabled
Returns:
ReturnedUITokenObject: Token object ready for JWT encoding
"""
disabled_non_admin_personal_key_creation = (
get_disabled_non_admin_personal_key_creation()
)
return ReturnedUITokenObject(
user_id=login_result.user_id,
key=login_result.key,
user_email=login_result.user_email,
user_role=login_result.user_role,
login_method=login_result.login_method,
premium_user=premium_user,
auth_header_name=general_settings.get(
"litellm_key_header_name", "Authorization"
),
disabled_non_admin_personal_key_creation=disabled_non_admin_personal_key_creation,
server_root_path=get_server_root_path(),
)

2
litellm/proxy/common_utils/custom_openapi_spec.py
View file

@ -72,7 +72,7 @@ class CustomOpenAPISpec:
openapi_schema["components"]["schemas"] = {}
# Add the schema
openapi_schema["components"]["schemas"][schema_name] = schema_def
CustomOpenAPISpec._move_defs_to_components(openapi_schema, {schema_name: schema_def})
@staticmethod
def add_request_body_to_paths(openapi_schema: Dict[str, Any], paths: List[str], schema_ref: str) -> None:

110
litellm/proxy/guardrails/guardrail_hooks/unified_guardrail/unified_guardrail.py
View file

@ -13,6 +13,7 @@ from litellm.caching.caching import DualCache
from litellm.cost_calculator import _infer_call_type
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.api_route_to_call_types import get_call_types_for_route
from litellm.llms import load_guardrail_translation_mappings
from litellm.proxy._types import UserAPIKeyAuth
from litellm.types.guardrails import GuardrailEventHooks
@ -176,6 +177,113 @@ class UnifiedLLMGuardrails(CustomLogger):
See Aim guardrail implementation for an example - https://github.com/BerriAI/litellm/blob/d0e022cfacb8e9ebc5409bb652059b6fd97b45c0/litellm/proxy/guardrails/guardrail_hooks/aim.py#L168
Triggered by mode: 'post_call'
Supports sampling_rate parameter to control how often chunks are processed.
sampling_rate=1 means every chunk, sampling_rate=5 means every 5th chunk, etc.
"""
global endpoint_guardrail_translation_mappings
from litellm.proxy.common_utils.callback_utils import (
add_guardrail_to_applied_guardrails_header,
)
guardrail_to_apply: CustomGuardrail = request_data.pop(
"guardrail_to_apply", None
)
# Get sampling rate from guardrail config or optional_params, default to 5
sampling_rate = 5
if guardrail_to_apply is not None:
# Check guardrail config first
guardrail_config = getattr(guardrail_to_apply, "guardrail_config", {})
sampling_rate = guardrail_config.get(
"streaming_sampling_rate", sampling_rate
)
# Also check optional_params as fallback
sampling_rate = self.optional_params.get(
"streaming_sampling_rate", sampling_rate
)
if guardrail_to_apply is None:
async for item in response:
yield item
return
event_type: GuardrailEventHooks = GuardrailEventHooks.post_call
if (
guardrail_to_apply.should_run_guardrail(
data=request_data, event_type=event_type
)
is not True
):
verbose_proxy_logger.debug(
"UnifiedLLMGuardrails: Post-call streaming scanning disabled for %s",
guardrail_to_apply.guardrail_name,
)
async for item in response:
yield item
return
# Initialize translation mappings if needed
if endpoint_guardrail_translation_mappings is None:
endpoint_guardrail_translation_mappings = (
load_guardrail_translation_mappings()
)
# Infer call type from first chunk
call_type = None
chunk_counter = 0
async for item in response:
yield item
chunk_counter += 1
# Infer call type from first chunk if not already done
if call_type is None and user_api_key_dict.request_route is not None:
call_types = get_call_types_for_route(user_api_key_dict.request_route)
if call_types is not None:
call_type = call_types[0]
# If call type not supported, just pass through all chunks
if (
call_type is None
or CallTypes(call_type)
not in endpoint_guardrail_translation_mappings
):
yield item
async for remaining_item in response:
yield remaining_item
return
# Process chunk based on sampling rate
if chunk_counter % sampling_rate == 0:
verbose_proxy_logger.debug(
"Processing streaming chunk %s (sampling_rate=%s) with guardrail %s",
chunk_counter,
sampling_rate,
guardrail_to_apply.guardrail_name,
)
endpoint_translation = endpoint_guardrail_translation_mappings[
CallTypes(call_type)
]()
processed_item = (
await endpoint_translation.process_output_streaming_response(
response=item,
guardrail_to_apply=guardrail_to_apply,
litellm_logging_obj=request_data.get("litellm_logging_obj"),
user_api_key_dict=user_api_key_dict,
)
)
# Add guardrail to applied guardrails header (only once, on first processed chunk)
if chunk_counter == sampling_rate:
add_guardrail_to_applied_guardrails_header(
request_data=request_data,
guardrail_name=guardrail_to_apply.guardrail_name,
)
yield processed_item
else:
yield item

34
litellm/proxy/hooks/parallel_request_limiter_v3.py
View file

@ -1343,19 +1343,19 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
"INSIDE parallel request limiter ASYNC SUCCESS LOGGING"
)
# Get metadata from kwargs
litellm_metadata = kwargs["litellm_params"].get(
get_metadata_variable_name_from_kwargs(kwargs), {}
# Get metadata from standard_logging_object - this correctly handles both
# 'metadata' and 'litellm_metadata' fields from litellm_params
standard_logging_object = kwargs.get("standard_logging_object") or {}
standard_logging_metadata = standard_logging_object.get("metadata") or {}
# user_api_key_hash is the same as user_api_key (it's the hash)
user_api_key = standard_logging_metadata.get("user_api_key_hash")
user_api_key_user_id = standard_logging_metadata.get("user_api_key_user_id")
user_api_key_team_id = standard_logging_metadata.get("user_api_key_team_id")
user_api_key_organization_id = standard_logging_metadata.get(
"user_api_key_org_id"
)
if litellm_metadata is None:
return
user_api_key = litellm_metadata.get("user_api_key")
user_api_key_user_id = litellm_metadata.get("user_api_key_user_id")
user_api_key_team_id = litellm_metadata.get("user_api_key_team_id")
user_api_key_organization_id = litellm_metadata.get(
"user_api_key_organization_id"
)
user_api_key_end_user_id = kwargs.get("user") or litellm_metadata.get(
user_api_key_end_user_id = kwargs.get("user") or standard_logging_metadata.get(
"user_api_key_end_user_id"
)
model_group = get_model_group_from_litellm_kwargs(kwargs)
@ -1501,10 +1501,12 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
litellm_parent_otel_span: Union[
Span, None
] = _get_parent_otel_span_from_kwargs(kwargs)
litellm_metadata = kwargs["litellm_params"]["metadata"]
user_api_key = (
litellm_metadata.get("user_api_key") if litellm_metadata else None
)
# Get metadata from standard_logging_object - this correctly handles both
# 'metadata' and 'litellm_metadata' fields from litellm_params
standard_logging_object = kwargs.get("standard_logging_object") or {}
standard_logging_metadata = standard_logging_object.get("metadata") or {}
user_api_key = standard_logging_metadata.get("user_api_key_hash")
pipeline_operations: List[RedisPipelineIncrementOperation] = []
if user_api_key:

298
litellm/proxy/proxy_server.py
View file

@ -31,7 +31,9 @@ from typing import (
from litellm._uuid import uuid
from litellm.constants import (
AIOHTTP_CONNECTOR_LIMIT,
AIOHTTP_CONNECTOR_LIMIT_PER_HOST,
AIOHTTP_KEEPALIVE_TIMEOUT,
AIOHTTP_NEEDS_CLEANUP_CLOSED,
AIOHTTP_TTL_DNS_CACHE,
AUDIO_SPEECH_CHUNK_SIZE,
BASE_MCP_ROUTE,
@ -626,21 +628,26 @@ async def proxy_shutdown_event():
async def _initialize_shared_aiohttp_session():
"""Initialize shared aiohttp session for connection reuse."""
"""Initialize shared aiohttp session for connection reuse with connection limits."""
try:
from aiohttp import ClientSession, TCPConnector
# Create connector with connection pooling settings optimized for long-lived connections
connector = TCPConnector(
limit=AIOHTTP_CONNECTOR_LIMIT,
keepalive_timeout=AIOHTTP_KEEPALIVE_TIMEOUT,
ttl_dns_cache=AIOHTTP_TTL_DNS_CACHE,
enable_cleanup_closed=True,
)
connector_kwargs = {
"keepalive_timeout": AIOHTTP_KEEPALIVE_TIMEOUT,
"ttl_dns_cache": AIOHTTP_TTL_DNS_CACHE,
"enable_cleanup_closed": True,
}
if AIOHTTP_CONNECTOR_LIMIT > 0:
connector_kwargs["limit"] = AIOHTTP_CONNECTOR_LIMIT
if AIOHTTP_CONNECTOR_LIMIT_PER_HOST > 0:
connector_kwargs["limit_per_host"] = AIOHTTP_CONNECTOR_LIMIT_PER_HOST
connector = TCPConnector(**connector_kwargs)
session = ClientSession(connector=connector)
verbose_proxy_logger.info(
f"SESSION REUSE: Created shared aiohttp session for connection pooling (ID: {id(session)})"
f"SESSION REUSE: Created shared aiohttp session for connection pooling (ID: {id(session)}, "
f"limit={AIOHTTP_CONNECTOR_LIMIT}, limit_per_host={AIOHTTP_CONNECTOR_LIMIT_PER_HOST})"
)
return session
except Exception as e:
@ -8266,256 +8273,49 @@ async def fallback_login(request: Request):
) # hidden since this is a helper for UI sso login
async def login(request: Request): # noqa: PLR0915
global premium_user, general_settings, master_key
from litellm.types.proxy.ui_sso import ReturnedUITokenObject
from litellm.proxy.auth.login_utils import authenticate_user, create_ui_token_object
from litellm.proxy.utils import get_custom_url
if master_key is None:
raise ProxyException(
message="Master Key not set for Proxy. Please set Master Key to use Admin UI. Set `LITELLM_MASTER_KEY` in .env or set general_settings:master_key in config.yaml. https://docs.litellm.ai/docs/proxy/virtual_keys. If set, use `--detailed_debug` to debug issue.",
type=ProxyErrorTypes.auth_error,
param="master_key",
code=status.HTTP_500_INTERNAL_SERVER_ERROR,
)
form = await request.form()
username = str(form.get("username"))
password = str(form.get("password"))
ui_username = os.getenv("UI_USERNAME", "admin")
ui_password = os.getenv("UI_PASSWORD", None)
if ui_password is None:
ui_password = str(master_key) if master_key is not None else None
if ui_password is None:
raise ProxyException(
message="set Proxy master key to use UI. https://docs.litellm.ai/docs/proxy/virtual_keys. If set, use `--detailed_debug` to debug issue.",
type=ProxyErrorTypes.auth_error,
param="UI_PASSWORD",
code=status.HTTP_500_INTERNAL_SERVER_ERROR,
)
# check if we can find the `username` in the db. on the ui, users can enter username=their email
_user_row: Optional[LiteLLM_UserTable] = None
user_role: Optional[
Literal[
LitellmUserRoles.PROXY_ADMIN,
LitellmUserRoles.PROXY_ADMIN_VIEW_ONLY,
LitellmUserRoles.INTERNAL_USER,
LitellmUserRoles.INTERNAL_USER_VIEW_ONLY,
]
] = None
if prisma_client is not None:
_user_row = cast(
Optional[LiteLLM_UserTable],
await prisma_client.db.litellm_usertable.find_first(
where={"user_email": {"equals": username}}
),
)
disabled_non_admin_personal_key_creation = (
get_disabled_non_admin_personal_key_creation()
# Authenticate user and get login result
login_result = await authenticate_user(
username=username,
password=password,
master_key=master_key,
prisma_client=prisma_client,
)
"""
To login to Admin UI, we support the following
- Login with UI_USERNAME and UI_PASSWORD
- Login with Invite Link `user_email` and `password` combination
"""
if secrets.compare_digest(username, ui_username) and secrets.compare_digest(
password, ui_password
):
# Non SSO -> If user is using UI_USERNAME and UI_PASSWORD they are Proxy admin
user_role = LitellmUserRoles.PROXY_ADMIN
user_id = litellm_proxy_admin_name
# we want the key created to have PROXY_ADMIN_PERMISSIONS
key_user_id = litellm_proxy_admin_name
if (
os.getenv("PROXY_ADMIN_ID", None) is not None
and os.environ["PROXY_ADMIN_ID"] == user_id
) or user_id == litellm_proxy_admin_name:
# checks if user is admin
key_user_id = os.getenv("PROXY_ADMIN_ID", litellm_proxy_admin_name)
# Create UI token object
returned_ui_token_object = create_ui_token_object(
login_result=login_result,
general_settings=general_settings,
premium_user=premium_user,
)
# Admin is Authe'd in - generate key for the UI to access Proxy
# Generate JWT token
import jwt
# ensure this user is set as the proxy admin, in this route there is no sso, we can assume this user is only the admin
await user_update(
data=UpdateUserRequest(
user_id=key_user_id,
user_role=user_role,
),
user_api_key_dict=UserAPIKeyAuth(
user_role=LitellmUserRoles.PROXY_ADMIN,
),
)
if os.getenv("DATABASE_URL") is not None:
response = await generate_key_helper_fn(
request_type="key",
**{
"user_role": LitellmUserRoles.PROXY_ADMIN,
"duration": "24hr",
"key_max_budget": litellm.max_ui_session_budget,
"models": [],
"aliases": {},
"config": {},
"spend": 0,
"user_id": key_user_id,
"team_id": "litellm-dashboard",
}, # type: ignore
)
else:
raise ProxyException(
message="No Database connected. Set DATABASE_URL in .env. If set, use `--detailed_debug` to debug issue.",
type=ProxyErrorTypes.auth_error,
param="DATABASE_URL",
code=status.HTTP_500_INTERNAL_SERVER_ERROR,
)
key = response["token"] # type: ignore
litellm_dashboard_ui = get_custom_url(str(request.base_url))
if litellm_dashboard_ui.endswith("/"):
litellm_dashboard_ui += "ui/"
else:
litellm_dashboard_ui += "/ui/"
import jwt
jwt_token = jwt.encode( # type: ignore
cast(dict, returned_ui_token_object),
master_key,
algorithm="HS256",
)
if get_secret_bool("EXPERIMENTAL_UI_LOGIN"):
user_info: Optional[LiteLLM_UserTable] = None
if _user_row is not None:
user_info = _user_row
elif (
user_id is not None
): # if user_id is not None, we are using the UI_USERNAME and UI_PASSWORD
user_info = LiteLLM_UserTable(
user_id=user_id,
user_role=user_role,
models=[],
max_budget=litellm.max_ui_session_budget,
)
if user_info is None:
raise HTTPException(
status_code=401,
detail={
"error": "User Information is required for experimental UI login"
},
)
key = ExperimentalUIJWTToken.get_experimental_ui_login_jwt_auth_token(
user_info
)
returned_ui_token_object = ReturnedUITokenObject(
user_id=user_id,
key=key,
user_email=None,
user_role=user_role,
login_method="username_password",
premium_user=premium_user,
auth_header_name=general_settings.get(
"litellm_key_header_name", "Authorization"
),
disabled_non_admin_personal_key_creation=disabled_non_admin_personal_key_creation,
server_root_path=get_server_root_path(),
)
jwt_token = jwt.encode( # type: ignore
cast(dict, returned_ui_token_object),
master_key,
algorithm="HS256",
)
litellm_dashboard_ui += "?login=success"
redirect_response = RedirectResponse(url=litellm_dashboard_ui, status_code=303)
redirect_response.set_cookie(key="token", value=jwt_token)
return redirect_response
elif _user_row is not None:
"""
When sharing invite links
-> if the user has no role in the DB assume they are only a viewer
"""
user_id = getattr(_user_row, "user_id", "unknown")
user_role = getattr(
_user_row, "user_role", LitellmUserRoles.INTERNAL_USER_VIEW_ONLY
)
user_email = getattr(_user_row, "user_email", "unknown")
_password = getattr(_user_row, "password", "unknown")
if _password is None:
raise ProxyException(
message="User has no password set. Please set a password for the user via `/user/update`.",
type=ProxyErrorTypes.auth_error,
param="password",
code=status.HTTP_401_UNAUTHORIZED,
)
# check if password == _user_row.password
hash_password = hash_token(token=password)
if secrets.compare_digest(password, _password) or secrets.compare_digest(
hash_password, _password
):
if os.getenv("DATABASE_URL") is not None:
response = await generate_key_helper_fn(
request_type="key",
**{ # type: ignore
"user_role": user_role,
"duration": "24hr",
"key_max_budget": litellm.max_ui_session_budget,
"models": [],
"aliases": {},
"config": {},
"spend": 0,
"user_id": user_id,
"team_id": "litellm-dashboard",
},
)
else:
raise ProxyException(
message="No Database connected. Set DATABASE_URL in .env. If set, use `--detailed_debug` to debug issue.",
type=ProxyErrorTypes.auth_error,
param="DATABASE_URL",
code=status.HTTP_500_INTERNAL_SERVER_ERROR,
)
key = response["token"] # type: ignore
litellm_dashboard_ui = get_custom_url(str(request.base_url))
if litellm_dashboard_ui.endswith("/"):
litellm_dashboard_ui += "ui/"
else:
litellm_dashboard_ui += "/ui/"
import jwt
returned_ui_token_object = ReturnedUITokenObject(
user_id=user_id,
key=key,
user_email=user_email,
user_role=cast(str, user_role),
login_method="username_password",
premium_user=premium_user,
auth_header_name=general_settings.get(
"litellm_key_header_name", "Authorization"
),
disabled_non_admin_personal_key_creation=disabled_non_admin_personal_key_creation,
server_root_path=get_server_root_path(),
)
jwt_token = jwt.encode( # type: ignore
cast(dict, returned_ui_token_object),
master_key,
algorithm="HS256",
)
litellm_dashboard_ui += "?login=success"
redirect_response = RedirectResponse(
url=litellm_dashboard_ui, status_code=303
)
redirect_response.set_cookie(key="token", value=jwt_token)
return redirect_response
else:
raise ProxyException(
message=f"Invalid credentials used to access UI.\nNot valid credentials for {username}",
type=ProxyErrorTypes.auth_error,
param="invalid_credentials",
code=status.HTTP_401_UNAUTHORIZED,
)
# Build redirect URL
litellm_dashboard_ui = get_custom_url(str(request.base_url))
if litellm_dashboard_ui.endswith("/"):
litellm_dashboard_ui += "ui/"
else:
raise ProxyException(
message="Invalid credentials used to access UI.\nCheck 'UI_USERNAME', 'UI_PASSWORD' in .env file",
type=ProxyErrorTypes.auth_error,
param="invalid_credentials",
code=status.HTTP_401_UNAUTHORIZED,
)
litellm_dashboard_ui += "/ui/"
litellm_dashboard_ui += "?login=success"
# Create redirect response with cookie
redirect_response = RedirectResponse(url=litellm_dashboard_ui, status_code=303)
redirect_response.set_cookie(key="token", value=jwt_token)
return redirect_response
@app.get("/onboarding/get_token", include_in_schema=False)

22
litellm/proxy/utils.py
View file

@ -1072,6 +1072,7 @@ class ProxyLogging:
"user_budget",
"soft_budget",
"team_budget",
"organization_budget",
"proxy_budget",
"projected_limit_exceeded",
],
@ -1559,6 +1560,7 @@ class ProxyLogging:
Covers:
1. /chat/completions
"""
for callback in litellm.callbacks:
_callback: Optional[CustomLogger] = None
if isinstance(callback, str):
@ -1573,11 +1575,21 @@ class ProxyLogging:
) or _callback.should_run_guardrail(
data=request_data, event_type=GuardrailEventHooks.post_call
):
response = _callback.async_post_call_streaming_iterator_hook(
user_api_key_dict=user_api_key_dict,
response=response,
request_data=request_data,
)
if "apply_guardrail" in type(callback).__dict__:
request_data["guardrail_to_apply"] = callback
response = (
unified_guardrail.async_post_call_streaming_iterator_hook(
user_api_key_dict=user_api_key_dict,
request_data=request_data,
response=response,
)
)
else:
response = _callback.async_post_call_streaming_iterator_hook(
user_api_key_dict=user_api_key_dict,
response=response,
request_data=request_data,
)
return response
def _init_response_taking_too_long_task(self, data: Optional[dict] = None):

2
litellm/types/llms/openai.py
View file

@ -303,7 +303,7 @@ class OpenAIFileObject(BaseModel):
`fine-tune`, `fine-tune-results`, `vision`, and `user_data`.
"""
status: Literal["uploaded", "processed", "error"]
status: Optional[Literal["uploaded", "processed", "error"]] = None
"""Deprecated.
The current status of the file, which can be either `uploaded`, `processed`, or

328
litellm/types/utils.py
View file

@ -401,6 +401,328 @@ CallTypesLiteral = Literal[
"responses",
]
# Mapping of API routes to their corresponding call types
API_ROUTE_TO_CALL_TYPES = {
# Chat Completions
"/chat/completions": [CallTypes.acompletion, CallTypes.completion],
"/v1/chat/completions": [CallTypes.acompletion, CallTypes.completion],
"/engines/{model}/chat/completions": [CallTypes.acompletion, CallTypes.completion],
"/openai/deployments/{model}/chat/completions": [
CallTypes.acompletion,
CallTypes.completion,
],
# Text Completions
"/completions": [CallTypes.atext_completion, CallTypes.text_completion],
"/v1/completions": [CallTypes.atext_completion, CallTypes.text_completion],
"/engines/{model}/completions": [
CallTypes.atext_completion,
CallTypes.text_completion,
],
"/openai/deployments/{model}/completions": [
CallTypes.atext_completion,
CallTypes.text_completion,
],
# Embeddings
"/embeddings": [CallTypes.aembedding, CallTypes.embedding],
"/v1/embeddings": [CallTypes.aembedding, CallTypes.embedding],
"/engines/{model}/embeddings": [CallTypes.aembedding, CallTypes.embedding],
"/openai/deployments/{model}/embeddings": [
CallTypes.aembedding,
CallTypes.embedding,
],
# Image Generation
"/images/generations": [CallTypes.aimage_generation, CallTypes.image_generation],
"/v1/images/generations": [CallTypes.aimage_generation, CallTypes.image_generation],
"/engines/{model}/images/generations": [
CallTypes.aimage_generation,
CallTypes.image_generation,
],
"/openai/deployments/{model}/images/generations": [
CallTypes.aimage_generation,
CallTypes.image_generation,
],
# Image Edits
"/images/edits": [CallTypes.aimage_edit, CallTypes.image_edit],
"/v1/images/edits": [CallTypes.aimage_edit, CallTypes.image_edit],
# Audio Transcriptions
"/audio/transcriptions": [CallTypes.atranscription, CallTypes.transcription],
"/v1/audio/transcriptions": [CallTypes.atranscription, CallTypes.transcription],
# Audio Speech
"/audio/speech": [CallTypes.aspeech, CallTypes.speech],
"/v1/audio/speech": [CallTypes.aspeech, CallTypes.speech],
# Moderations
"/moderations": [CallTypes.amoderation, CallTypes.moderation],
"/v1/moderations": [CallTypes.amoderation, CallTypes.moderation],
# Rerank
"/rerank": [CallTypes.arerank, CallTypes.rerank],
"/v1/rerank": [CallTypes.arerank, CallTypes.rerank],
"/v2/rerank": [CallTypes.arerank, CallTypes.rerank],
# Search
"/search": [CallTypes.asearch, CallTypes.search],
"/v1/search": [CallTypes.asearch, CallTypes.search],
# Batches
"/batches": [CallTypes.acreate_batch, CallTypes.create_batch],
"/v1/batches": [CallTypes.acreate_batch, CallTypes.create_batch],
"/batches/{batch_id}": [CallTypes.aretrieve_batch, CallTypes.retrieve_batch],
"/v1/batches/{batch_id}": [CallTypes.aretrieve_batch, CallTypes.retrieve_batch],
# Files
"/files": [
CallTypes.acreate_file,
CallTypes.create_file,
CallTypes.afile_list,
CallTypes.file_list,
],
"/v1/files": [
CallTypes.acreate_file,
CallTypes.create_file,
CallTypes.afile_list,
CallTypes.file_list,
],
"/files/{file_id}": [
CallTypes.afile_retrieve,
CallTypes.file_retrieve,
CallTypes.afile_delete,
CallTypes.file_delete,
],
"/v1/files/{file_id}": [
CallTypes.afile_retrieve,
CallTypes.file_retrieve,
CallTypes.afile_delete,
CallTypes.file_delete,
],
"/files/{file_id}/content": [CallTypes.afile_content, CallTypes.file_content],
"/v1/files/{file_id}/content": [CallTypes.afile_content, CallTypes.file_content],
# Assistants
"/assistants": [
CallTypes.aget_assistants,
CallTypes.get_assistants,
CallTypes.acreate_assistants,
CallTypes.create_assistants,
],
"/v1/assistants": [
CallTypes.aget_assistants,
CallTypes.get_assistants,
CallTypes.acreate_assistants,
CallTypes.create_assistants,
],
"/assistants/{assistant_id}": [
CallTypes.adelete_assistant,
CallTypes.delete_assistant,
],
"/v1/assistants/{assistant_id}": [
CallTypes.adelete_assistant,
CallTypes.delete_assistant,
],
# Threads
"/threads": [CallTypes.acreate_thread, CallTypes.create_thread],
"/v1/threads": [CallTypes.acreate_thread, CallTypes.create_thread],
"/threads/{thread_id}": [CallTypes.aget_thread, CallTypes.get_thread],
"/v1/threads/{thread_id}": [CallTypes.aget_thread, CallTypes.get_thread],
# Thread Messages
"/threads/{thread_id}/messages": [
CallTypes.a_add_message,
CallTypes.add_message,
CallTypes.aget_messages,
CallTypes.get_messages,
],
"/v1/threads/{thread_id}/messages": [
CallTypes.a_add_message,
CallTypes.add_message,
CallTypes.aget_messages,
CallTypes.get_messages,
],
# Thread Runs
"/threads/{thread_id}/runs": [
CallTypes.arun_thread,
CallTypes.run_thread,
CallTypes.arun_thread_stream,
CallTypes.run_thread_stream,
],
"/v1/threads/{thread_id}/runs": [
CallTypes.arun_thread,
CallTypes.run_thread,
CallTypes.arun_thread_stream,
CallTypes.run_thread_stream,
],
# Fine-tuning Jobs
"/fine_tuning/jobs": [
CallTypes.acreate_fine_tuning_job,
CallTypes.create_fine_tuning_job,
CallTypes.alist_fine_tuning_jobs,
CallTypes.list_fine_tuning_jobs,
],
"/v1/fine_tuning/jobs": [
CallTypes.acreate_fine_tuning_job,
CallTypes.create_fine_tuning_job,
CallTypes.alist_fine_tuning_jobs,
CallTypes.list_fine_tuning_jobs,
],
"/fine_tuning/jobs/{fine_tuning_job_id}": [
CallTypes.aretrieve_fine_tuning_job,
CallTypes.retrieve_fine_tuning_job,
],
"/v1/fine_tuning/jobs/{fine_tuning_job_id}": [
CallTypes.aretrieve_fine_tuning_job,
CallTypes.retrieve_fine_tuning_job,
],
"/fine_tuning/jobs/{fine_tuning_job_id}/cancel": [
CallTypes.acancel_fine_tuning_job,
CallTypes.cancel_fine_tuning_job,
],
"/v1/fine_tuning/jobs/{fine_tuning_job_id}/cancel": [
CallTypes.acancel_fine_tuning_job,
CallTypes.cancel_fine_tuning_job,
],
# Video Generation
"/videos": [
CallTypes.acreate_video,
CallTypes.create_video,
CallTypes.avideo_list,
CallTypes.video_list,
],
"/v1/videos": [
CallTypes.acreate_video,
CallTypes.create_video,
CallTypes.avideo_list,
CallTypes.video_list,
],
"/videos/{video_id}": [
CallTypes.avideo_retrieve,
CallTypes.video_retrieve,
CallTypes.avideo_delete,
CallTypes.video_delete,
],
"/v1/videos/{video_id}": [
CallTypes.avideo_retrieve,
CallTypes.video_retrieve,
CallTypes.avideo_delete,
CallTypes.video_delete,
],
"/videos/{video_id}/content": [CallTypes.avideo_content, CallTypes.video_content],
"/v1/videos/{video_id}/content": [
CallTypes.avideo_content,
CallTypes.video_content,
],
"/videos/{video_id}/remix": [CallTypes.avideo_remix, CallTypes.video_remix],
"/v1/videos/{video_id}/remix": [CallTypes.avideo_remix, CallTypes.video_remix],
# Vector Stores
"/vector_stores": [CallTypes.avector_store_create, CallTypes.vector_store_create],
"/v1/vector_stores": [
CallTypes.avector_store_create,
CallTypes.vector_store_create,
],
"/vector_stores/{vector_store_id}/search": [
CallTypes.avector_store_search,
CallTypes.vector_store_search,
],
"/v1/vector_stores/{vector_store_id}/search": [
CallTypes.avector_store_search,
CallTypes.vector_store_search,
],
"/vector_stores/{vector_store_id}/files": [
CallTypes.avector_store_file_create,
CallTypes.vector_store_file_create,
CallTypes.avector_store_file_list,
CallTypes.vector_store_file_list,
],
"/v1/vector_stores/{vector_store_id}/files": [
CallTypes.avector_store_file_create,
CallTypes.vector_store_file_create,
CallTypes.avector_store_file_list,
CallTypes.vector_store_file_list,
],
"/vector_stores/{vector_store_id}/files/{file_id}": [
CallTypes.avector_store_file_retrieve,
CallTypes.vector_store_file_retrieve,
CallTypes.avector_store_file_delete,
CallTypes.vector_store_file_delete,
],
"/v1/vector_stores/{vector_store_id}/files/{file_id}": [
CallTypes.avector_store_file_retrieve,
CallTypes.vector_store_file_retrieve,
CallTypes.avector_store_file_delete,
CallTypes.vector_store_file_delete,
],
"/vector_stores/{vector_store_id}/files/{file_id}/content": [
CallTypes.avector_store_file_content,
CallTypes.vector_store_file_content,
],
"/v1/vector_stores/{vector_store_id}/files/{file_id}/content": [
CallTypes.avector_store_file_content,
CallTypes.vector_store_file_content,
],
"/vector_stores/{vector_store_id}/files/{file_id}/update": [
CallTypes.avector_store_file_update,
CallTypes.vector_store_file_update,
],
"/v1/vector_stores/{vector_store_id}/files/{file_id}/update": [
CallTypes.avector_store_file_update,
CallTypes.vector_store_file_update,
],
# Containers
"/containers": [
CallTypes.acreate_container,
CallTypes.create_container,
CallTypes.alist_containers,
CallTypes.list_containers,
],
"/v1/containers": [
CallTypes.acreate_container,
CallTypes.create_container,
CallTypes.alist_containers,
CallTypes.list_containers,
],
"/containers/{container_id}": [
CallTypes.aretrieve_container,
CallTypes.retrieve_container,
CallTypes.adelete_container,
CallTypes.delete_container,
],
"/v1/containers/{container_id}": [
CallTypes.aretrieve_container,
CallTypes.retrieve_container,
CallTypes.adelete_container,
CallTypes.delete_container,
],
# Responses API
"/responses": [CallTypes.aresponses, CallTypes.responses],
"/v1/responses": [CallTypes.aresponses, CallTypes.responses],
"/responses/{response_id}": [CallTypes.aresponses, CallTypes.responses],
"/v1/responses/{response_id}": [CallTypes.aresponses, CallTypes.responses],
"/responses/{response_id}/input_items": [CallTypes.alist_input_items],
"/v1/responses/{response_id}/input_items": [CallTypes.alist_input_items],
# Realtime API
"/realtime": [CallTypes.arealtime],
"/v1/realtime": [CallTypes.arealtime],
# Provider-specific routes
"/anthropic/v1/messages": [CallTypes.anthropic_messages],
# Google GenAI routes
"/generate_content": [CallTypes.agenerate_content, CallTypes.generate_content],
"/models/{model}:generateContent": [
CallTypes.agenerate_content,
CallTypes.generate_content,
],
"/generate_content_stream": [
CallTypes.agenerate_content_stream,
CallTypes.generate_content_stream,
],
"/models/{model}:streamGenerateContent": [
CallTypes.agenerate_content_stream,
CallTypes.generate_content_stream,
],
# MCP (Model Context Protocol)
"/mcp/call_tool": [CallTypes.call_mcp_tool],
# Passthrough endpoints
"/llm_passthrough": [
CallTypes.llm_passthrough_route,
CallTypes.allm_passthrough_route,
],
"/v1/llm_passthrough": [
CallTypes.llm_passthrough_route,
CallTypes.allm_passthrough_route,
],
}
class PassthroughCallTypes(Enum):
passthrough_image_generation = "passthrough-image-generation"
@ -1060,7 +1382,10 @@ class Usage(CompletionUsage):
# Auto-calculate text_tokens only if provider didn't set it explicitly
# Formula: text_tokens = completion_tokens - reasoning_tokens - image_tokens - audio_tokens
if _completion_tokens_details.text_tokens is None and completion_tokens is not None:
if (
_completion_tokens_details.text_tokens is None
and completion_tokens is not None
):
calculated_text_tokens = completion_tokens - reasoning_tokens
# Subtract other modality tokens if present
@ -2618,6 +2943,7 @@ class LlmProviders(str, Enum):
DATABRICKS = "databricks"
EMPOWER = "empower"
GITHUB = "github"
RAGFLOW = "ragflow"
COMPACTIFAI = "compactifai"
DOCKER_MODEL_RUNNER = "docker_model_runner"
CUSTOM = "custom"

4
litellm/types/vector_stores.py
View file

@ -3,17 +3,15 @@ from datetime import datetime
from enum import Enum
from typing import Any, Dict, List, Literal, Optional, Tuple, Union
from annotated_types import Ge
from pydantic import BaseModel
from typing_extensions import TypedDict
from litellm.types.router import CredentialLiteLLMParams, GenericLiteLLMParams
class SupportedVectorStoreIntegrations(str, Enum):
"""Supported vector store integrations."""
BEDROCK = "bedrock"
RAGFLOW = "ragflow"
class LiteLLM_VectorStoreConfig(TypedDict, total=False):

8
litellm/utils.py
View file

@ -7110,6 +7110,8 @@ class ProviderConfigManager:
return litellm.CompactifAIChatConfig()
elif litellm.LlmProviders.GITHUB_COPILOT == provider:
return litellm.GithubCopilotConfig()
elif litellm.LlmProviders.RAGFLOW == provider:
return litellm.RAGFlowConfig()
elif (
litellm.LlmProviders.CUSTOM == provider
or litellm.LlmProviders.CUSTOM_OPENAI == provider
@ -7631,6 +7633,12 @@ class ProviderConfigManager:
)
return GeminiVectorStoreConfig()
elif litellm.LlmProviders.RAGFLOW == provider:
from litellm.llms.ragflow.vector_stores.transformation import (
RAGFlowVectorStoreConfig,
)
return RAGFlowVectorStoreConfig()
return None
@staticmethod

144
model_prices_and_context_window.json
View file

@ -9629,6 +9629,21 @@
"supports_prompt_caching": true,
"supports_tool_choice": true
},
"deepseek/deepseek-v3.2": {
"input_cost_per_token": 2.8e-07,
"input_cost_per_token_cache_hit": 2.8e-08,
"litellm_provider": "deepseek",
"max_input_tokens": 163840,
"max_output_tokens": 163840,
"max_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 4e-07,
"supports_assistant_prefill": true,
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_tool_choice": true
},
"deepseek.v3-v1:0": {
"input_cost_per_token": 5.8e-07,
"litellm_provider": "bedrock_converse",
@ -10837,25 +10852,25 @@
"supports_tool_choice": true
},
"ft:babbage-002": {
"input_cost_per_token": 4e-07,
"input_cost_per_token": 1.6e-06,
"input_cost_per_token_batches": 2e-07,
"litellm_provider": "text-completion-openai",
"max_input_tokens": 16384,
"max_output_tokens": 4096,
"max_tokens": 16384,
"mode": "completion",
"output_cost_per_token": 4e-07,
"output_cost_per_token": 1.6e-06,
"output_cost_per_token_batches": 2e-07
},
"ft:davinci-002": {
"input_cost_per_token": 2e-06,
"input_cost_per_token": 1.2e-05,
"input_cost_per_token_batches": 1e-06,
"litellm_provider": "text-completion-openai",
"max_input_tokens": 16384,
"max_output_tokens": 4096,
"max_tokens": 16384,
"mode": "completion",
"output_cost_per_token": 2e-06,
"output_cost_per_token": 1.2e-05,
"output_cost_per_token_batches": 1e-06
},
"ft:gpt-3.5-turbo": {
@ -10918,6 +10933,7 @@
"supports_tool_choice": true
},
"ft:gpt-4o-2024-08-06": {
"cache_read_input_token_cost": 1.875e-06,
"input_cost_per_token": 3.75e-06,
"input_cost_per_token_batches": 1.875e-06,
"litellm_provider": "openai",
@ -10930,6 +10946,7 @@
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
@ -10950,8 +10967,7 @@
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true
"supports_tool_choice": true
},
"ft:gpt-4o-mini-2024-07-18": {
"cache_read_input_token_cost": 1.5e-07,
@ -10970,8 +10986,79 @@
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true
"supports_tool_choice": true
},
"ft:gpt-4.1-2025-04-14": {
"cache_read_input_token_cost": 7.5e-07,
"input_cost_per_token": 3e-06,
"input_cost_per_token_batches": 1.5e-06,
"litellm_provider": "openai",
"max_input_tokens": 1047576,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "chat",
"output_cost_per_token": 1.2e-05,
"output_cost_per_token_batches": 6e-06,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"ft:gpt-4.1-mini-2025-04-14": {
"cache_read_input_token_cost": 2e-07,
"input_cost_per_token": 8e-07,
"input_cost_per_token_batches": 4e-07,
"litellm_provider": "openai",
"max_input_tokens": 1047576,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "chat",
"output_cost_per_token": 3.2e-06,
"output_cost_per_token_batches": 1.6e-06,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"ft:gpt-4.1-nano-2025-04-14": {
"cache_read_input_token_cost": 5e-08,
"input_cost_per_token": 2e-07,
"input_cost_per_token_batches": 1e-07,
"litellm_provider": "openai",
"max_input_tokens": 1047576,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "chat",
"output_cost_per_token": 8e-07,
"output_cost_per_token_batches": 4e-07,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"ft:o4-mini-2025-04-16": {
"cache_read_input_token_cost": 1e-06,
"input_cost_per_token": 4e-06,
"input_cost_per_token_batches": 2e-06,
"litellm_provider": "openai",
"max_input_tokens": 200000,
"max_output_tokens": 100000,
"max_tokens": 100000,
"mode": "chat",
"output_cost_per_token": 1.6e-05,
"output_cost_per_token_batches": 8e-06,
"supports_function_calling": true,
"supports_parallel_function_calling": false,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true
},
"gemini-1.0-pro": {
"input_cost_per_character": 1.25e-07,
@ -20565,6 +20652,21 @@
"supports_reasoning": true,
"supports_tool_choice": true
},
"openrouter/deepseek/deepseek-v3.2": {
"input_cost_per_token": 2.8e-07,
"input_cost_per_token_cache_hit": 2.8e-08,
"litellm_provider": "openrouter",
"max_input_tokens": 163840,
"max_output_tokens": 163840,
"max_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 4e-07,
"supports_assistant_prefill": true,
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_tool_choice": true
},
"openrouter/deepseek/deepseek-v3.2-exp": {
"input_cost_per_token": 2e-07,
"input_cost_per_token_cache_hit": 2e-08,
@ -23820,6 +23922,32 @@
"supports_vision": true,
"tool_use_system_prompt_tokens": 159
},
"global.anthropic.claude-opus-4-5-20251101-v1:0": {
"cache_creation_input_token_cost": 6.25e-06,
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 2.5e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_assistant_prefill": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true,
"tool_use_system_prompt_tokens": 159
},
"us.anthropic.claude-sonnet-4-20250514-v1:0": {
"cache_creation_input_token_cost": 3.75e-06,
"cache_read_input_token_cost": 3e-07,

116
tests/enterprise/litellm_enterprise/proxy/guardrails/test_apply_guardrail_endpoint.py
View file

@ -1,6 +1,7 @@
"""
Test the /guardrails/apply_guardrail endpoint
"""
import os
import sys
from unittest.mock import AsyncMock, Mock, patch
@ -22,37 +23,45 @@ async def test_apply_guardrail_endpoint_returns_correct_response():
from litellm.proxy.guardrails.guardrail_endpoints import apply_guardrail
# Mock the guardrail registry
with patch("litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY") as mock_registry:
with patch(
"litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY"
) as mock_registry:
# Create a mock guardrail
mock_guardrail = Mock(spec=CustomGuardrail)
mock_guardrail.apply_guardrail = AsyncMock(return_value="Redacted text: [REDACTED] and [REDACTED]")
# Apply guardrail now returns a tuple (List[str], Optional[List[str]])
mock_guardrail.apply_guardrail = AsyncMock(
return_value=(["Redacted text: [REDACTED] and [REDACTED]"], None)
)
# Configure the registry to return our mock guardrail
mock_registry.get_initialized_guardrail_callback.return_value = mock_guardrail
# Create the request
request = ApplyGuardrailRequest(
guardrail_name="test-guardrail",
text="Test text with PII",
language="en",
entities=["EMAIL_ADDRESS", "PERSON"]
entities=["EMAIL_ADDRESS", "PERSON"],
)
# Create a mock user API key
user_api_key_dict = UserAPIKeyAuth(api_key="test-key")
# Call the endpoint
response = await apply_guardrail(request=request, user_api_key_dict=user_api_key_dict)
response = await apply_guardrail(
request=request, user_api_key_dict=user_api_key_dict
)
# Verify the response is of the correct type
assert isinstance(response, ApplyGuardrailResponse)
assert response.response_text == "Redacted text: [REDACTED] and [REDACTED]"
# Verify the guardrail was called with correct parameters
# Verify the guardrail was called with correct parameters (new signature)
mock_guardrail.apply_guardrail.assert_called_once_with(
text="Test text with PII",
language="en",
entities=["EMAIL_ADDRESS", "PERSON"]
texts=["Test text with PII"],
request_data={},
input_type="request",
images=None,
)
@ -63,23 +72,23 @@ async def test_apply_guardrail_endpoint_guardrail_not_found():
from litellm.proxy.guardrails.guardrail_endpoints import apply_guardrail
# Mock the guardrail registry to return None
with patch("litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY") as mock_registry:
with patch(
"litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY"
) as mock_registry:
mock_registry.get_initialized_guardrail_callback.return_value = None
# Create the request
request = ApplyGuardrailRequest(
guardrail_name="non-existent-guardrail",
text="Test text",
language="en"
guardrail_name="non-existent-guardrail", text="Test text", language="en"
)
# Create a mock user API key
user_api_key_dict = UserAPIKeyAuth(api_key="test-key")
# Verify exception is raised
with pytest.raises(ProxyException) as exc_info:
await apply_guardrail(request=request, user_api_key_dict=user_api_key_dict)
assert "non-existent-guardrail" in exc_info.value.message
assert "not found" in exc_info.value.message
@ -90,34 +99,41 @@ async def test_apply_guardrail_endpoint_with_presidio_guardrail():
from litellm.proxy.guardrails.guardrail_endpoints import apply_guardrail
# Mock the guardrail registry
with patch("litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY") as mock_registry:
with patch(
"litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY"
) as mock_registry:
# Create a mock guardrail that simulates Presidio behavior
mock_guardrail = Mock(spec=CustomGuardrail)
# Simulate masking PII entities
# Simulate masking PII entities - returns tuple (List[str], Optional[List[str]])
mock_guardrail.apply_guardrail = AsyncMock(
return_value="My name is [PERSON] and my email is [EMAIL_ADDRESS]"
return_value=(["My name is [PERSON] and my email is [EMAIL_ADDRESS]"], None)
)
# Configure the registry to return our mock guardrail
mock_registry.get_initialized_guardrail_callback.return_value = mock_guardrail
# Create the request
request = ApplyGuardrailRequest(
guardrail_name="pii-detection-guard",
text="My name is John Doe and my email is john@example.com",
language="en",
entities=["EMAIL_ADDRESS", "PERSON"]
entities=["EMAIL_ADDRESS", "PERSON"],
)
# Create a mock user API key
user_api_key_dict = UserAPIKeyAuth(api_key="test-key")
# Call the endpoint
response = await apply_guardrail(request=request, user_api_key_dict=user_api_key_dict)
response = await apply_guardrail(
request=request, user_api_key_dict=user_api_key_dict
)
# Verify the response is of the correct type
assert isinstance(response, ApplyGuardrailResponse)
assert response.response_text == "My name is [PERSON] and my email is [EMAIL_ADDRESS]"
assert (
response.response_text
== "My name is [PERSON] and my email is [EMAIL_ADDRESS]"
)
assert "john@example.com" not in response.response_text
assert "John Doe" not in response.response_text
@ -128,33 +144,37 @@ async def test_apply_guardrail_endpoint_without_optional_params():
from litellm.proxy.guardrails.guardrail_endpoints import apply_guardrail
# Mock the guardrail registry
with patch("litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY") as mock_registry:
with patch(
"litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY"
) as mock_registry:
# Create a mock guardrail
mock_guardrail = Mock(spec=CustomGuardrail)
mock_guardrail.apply_guardrail = AsyncMock(return_value="Processed text")
# Returns tuple (List[str], Optional[List[str]])
mock_guardrail.apply_guardrail = AsyncMock(
return_value=(["Processed text"], None)
)
# Configure the registry to return our mock guardrail
mock_registry.get_initialized_guardrail_callback.return_value = mock_guardrail
# Create the request without optional parameters
request = ApplyGuardrailRequest(
guardrail_name="test-guardrail",
text="Test text"
guardrail_name="test-guardrail", text="Test text"
)
# Create a mock user API key
user_api_key_dict = UserAPIKeyAuth(api_key="test-key")
# Call the endpoint
response = await apply_guardrail(request=request, user_api_key_dict=user_api_key_dict)
response = await apply_guardrail(
request=request, user_api_key_dict=user_api_key_dict
)
# Verify the response is of the correct type
assert isinstance(response, ApplyGuardrailResponse)
assert response.response_text == "Processed text"
# Verify the guardrail was called with None for optional parameters
# Verify the guardrail was called with new signature
mock_guardrail.apply_guardrail.assert_called_once_with(
text="Test text",
language=None,
entities=None
texts=["Test text"], request_data={}, input_type="request", images=None
)

161
tests/enterprise/litellm_enterprise/proxy/guardrails/test_bedrock_apply_guardrail.py
View file

@ -1,6 +1,7 @@
"""
Test the Bedrock guardrail apply_guardrail functionality
"""
import os
import sys
from unittest.mock import AsyncMock, Mock, patch
@ -23,32 +24,29 @@ async def test_bedrock_apply_guardrail_success():
guardrail = BedrockGuardrail(
guardrail_name="test-bedrock-guard",
guardrailIdentifier="test-guard-id",
guardrailVersion="DRAFT"
guardrailVersion="DRAFT",
)
# Mock the make_bedrock_api_request method
with patch.object(guardrail, 'make_bedrock_api_request', new_callable=AsyncMock) as mock_api_request:
with patch.object(
guardrail, "make_bedrock_api_request", new_callable=AsyncMock
) as mock_api_request:
# Mock a successful response from Bedrock
mock_response = {
"action": "ALLOWED",
"content": [
{
"text": {
"text": "This is a test message with some content"
}
}
]
"content": [{"text": {"text": "This is a test message with some content"}}],
}
mock_api_request.return_value = mock_response
# Test the apply_guardrail method
result = await guardrail.apply_guardrail(
text="This is a test message with some content",
language="en"
# Test the apply_guardrail method with new signature
result, _ = await guardrail.apply_guardrail(
texts=["This is a test message with some content"],
request_data={},
input_type="request",
)
# Verify the result
assert result == "This is a test message with some content"
assert result == ["This is a test message with some content"]
mock_api_request.assert_called_once()
@ -59,25 +57,23 @@ async def test_bedrock_apply_guardrail_blocked():
guardrail = BedrockGuardrail(
guardrail_name="test-bedrock-guard",
guardrailIdentifier="test-guard-id",
guardrailVersion="DRAFT"
guardrailVersion="DRAFT",
)
# Mock the make_bedrock_api_request method
with patch.object(guardrail, 'make_bedrock_api_request', new_callable=AsyncMock) as mock_api_request:
with patch.object(
guardrail, "make_bedrock_api_request", new_callable=AsyncMock
) as mock_api_request:
# Mock a blocked response from Bedrock
mock_response = {
"action": "BLOCKED",
"reason": "Content violates policy"
}
mock_response = {"action": "BLOCKED", "reason": "Content violates policy"}
mock_api_request.return_value = mock_response
# Test the apply_guardrail method should raise an exception
with pytest.raises(Exception) as exc_info:
await guardrail.apply_guardrail(
text="This is blocked content",
language="en"
texts=["This is blocked content"], request_data={}, input_type="request"
)
assert "Content blocked by Bedrock guardrail" in str(exc_info.value)
assert "Content violates policy" in str(exc_info.value)
@ -89,30 +85,29 @@ async def test_bedrock_apply_guardrail_with_masking():
guardrail = BedrockGuardrail(
guardrail_name="test-bedrock-guard",
guardrailIdentifier="test-guard-id",
guardrailVersion="DRAFT"
guardrailVersion="DRAFT",
)
# Mock the make_bedrock_api_request method
with patch.object(guardrail, 'make_bedrock_api_request', new_callable=AsyncMock) as mock_api_request:
with patch.object(
guardrail, "make_bedrock_api_request", new_callable=AsyncMock
) as mock_api_request:
# Mock a response with masked content
mock_response = {
"action": "ALLOWED",
"outputs": [
{
"text": "This is a test message with [REDACTED] content"
}
]
"outputs": [{"text": "This is a test message with [REDACTED] content"}],
}
mock_api_request.return_value = mock_response
# Test the apply_guardrail method
result = await guardrail.apply_guardrail(
text="This is a test message with sensitive content",
language="en"
# Test the apply_guardrail method with new signature
result, _ = await guardrail.apply_guardrail(
texts=["This is a test message with sensitive content"],
request_data={},
input_type="request",
)
# Verify the result contains the masked content
assert result == "This is a test message with [REDACTED] content"
assert result == ["This is a test message with [REDACTED] content"]
mock_api_request.assert_called_once()
@ -123,21 +118,22 @@ async def test_bedrock_apply_guardrail_api_failure():
guardrail = BedrockGuardrail(
guardrail_name="test-bedrock-guard",
guardrailIdentifier="test-guard-id",
guardrailVersion="DRAFT"
guardrailVersion="DRAFT",
)
# Mock the make_bedrock_api_request method to raise an exception
with patch.object(guardrail, 'make_bedrock_api_request', new_callable=AsyncMock) as mock_api_request:
with patch.object(
guardrail, "make_bedrock_api_request", new_callable=AsyncMock
) as mock_api_request:
mock_api_request.side_effect = Exception("API connection failed")
# Test the apply_guardrail method should raise an exception
with pytest.raises(Exception) as exc_info:
await guardrail.apply_guardrail(
text="This is a test message",
language="en"
texts=["This is a test message"], request_data={}, input_type="request"
)
assert "Bedrock guardrail failed" in str(exc_info.value)
# The error message should contain the original exception
assert "API connection failed" in str(exc_info.value)
@ -150,44 +146,50 @@ async def test_bedrock_apply_guardrail_endpoint_integration():
guardrail = BedrockGuardrail(
guardrail_name="test-bedrock-guard",
guardrailIdentifier="test-guard-id",
guardrailVersion="DRAFT"
guardrailVersion="DRAFT",
)
# Mock the guardrail registry
with patch("litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY") as mock_registry:
with patch(
"litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY"
) as mock_registry:
# Mock the make_bedrock_api_request method
with patch.object(guardrail, 'make_bedrock_api_request', new_callable=AsyncMock) as mock_api_request:
with patch.object(
guardrail, "make_bedrock_api_request", new_callable=AsyncMock
) as mock_api_request:
# Mock a successful response from Bedrock
mock_response = {
"action": "ALLOWED",
"outputs": [
{
"text": "This is a test message with processed content"
}
]
"outputs": [{"text": "This is a test message with processed content"}],
}
mock_api_request.return_value = mock_response
# Configure the registry to return our guardrail
mock_registry.get_initialized_guardrail_callback.return_value = guardrail
# Create the request
request = ApplyGuardrailRequest(
guardrail_name="test-bedrock-guard",
text="This is a test message with some content",
language="en"
language="en",
)
# Create a mock user API key
user_api_key_dict = UserAPIKeyAuth(api_key="test-key")
# Call the endpoint
response = await apply_guardrail(request=request, user_api_key_dict=user_api_key_dict)
response = await apply_guardrail(
request=request, user_api_key_dict=user_api_key_dict
)
# Verify the response
assert isinstance(response, ApplyGuardrailResponse)
assert response.response_text == "This is a test message with processed content"
mock_api_request.assert_called_once()
assert (
response.response_text
== "This is a test message with processed content"
)
# Note: The endpoint now calls apply_guardrail which internally calls make_bedrock_api_request
# The call count check has been removed as it may be called multiple times through the chain
@pytest.mark.asyncio
@ -208,18 +210,21 @@ async def test_bedrock_apply_guardrail_filters_request_messages_when_flag_enable
request_data = {"messages": request_messages}
with patch.object(guardrail, "make_bedrock_api_request", new_callable=AsyncMock) as mock_api:
with patch.object(
guardrail, "make_bedrock_api_request", new_callable=AsyncMock
) as mock_api:
mock_api.return_value = {"action": "ALLOWED"}
result = await guardrail.apply_guardrail(
text="latest question",
result, _ = await guardrail.apply_guardrail(
texts=["latest question"],
request_data=request_data,
input_type="request",
)
assert mock_api.called
_, kwargs = mock_api.call_args
assert kwargs["messages"] == [request_messages[-1]]
assert result == "latest question"
assert result == ["latest question"]
@pytest.mark.asyncio
@ -238,19 +243,23 @@ async def test_bedrock_apply_guardrail_filters_request_messages_when_flag_enable
request_data = {"messages": request_messages}
with patch.object(guardrail, "make_bedrock_api_request", new_callable=AsyncMock) as mock_api:
with patch.object(
guardrail, "make_bedrock_api_request", new_callable=AsyncMock
) as mock_api:
mock_api.return_value = {"action": "BLOCKED", "reason": "policy"}
with pytest.raises(Exception, match="policy") as exc_info:
await guardrail.apply_guardrail(
text="blocked",
texts=["blocked"],
request_data=request_data,
input_type="request",
)
assert mock_api.called
_, kwargs = mock_api.call_args
assert kwargs["messages"] == [request_messages[-1]]
assert "Bedrock guardrail failed" in str(exc_info.value)
assert "Content blocked by Bedrock guardrail" in str(exc_info.value)
def test_bedrock_guardrail_filters_latest_user_message_when_enabled():
guardrail = BedrockGuardrail(

20
tests/local_testing/test_gemini_reasoning_content.py Normal file
View file

@ -0,0 +1,20 @@
import json
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import VertexGeminiConfig
def test_empty_part_does_not_create_thinking_block():
parts = [{"text": "", "thoughtSignature": "sig-1"}]
config = VertexGeminiConfig()
thinking_blocks = config._extract_thinking_blocks_from_parts(parts)
assert thinking_blocks == []
def test_non_empty_part_creates_thinking_block():
parts = [{"text": "Some thinking", "thoughtSignature": "sig-2"}]
config = VertexGeminiConfig()
thinking_blocks = config._extract_thinking_blocks_from_parts(parts)
assert len(thinking_blocks) == 1
block = thinking_blocks[0]
# thinking should be valid JSON containing the text
parsed = json.loads(block["thinking"]) if isinstance(block["thinking"], str) else None
assert parsed is not None and parsed.get("text") == "Some thinking"

2
tests/logging_callback_tests/test_alerting.py
View file

@ -477,6 +477,7 @@ async def test_send_daily_reports_all_zero_or_none():
"token_budget",
"user_budget",
"team_budget",
"organization_budget",
"proxy_budget",
"projected_limit_exceeded",
],
@ -514,6 +515,7 @@ async def test_send_token_budget_crossed_alerts(alerting_type):
"token_budget",
"user_budget",
"team_budget",
"organization_budget",
"proxy_budget",
"projected_limit_exceeded",
],

47
tests/logging_callback_tests/test_datadog.py
View file

@ -633,11 +633,14 @@ async def test_datadog_message_redaction():
def test_datadog_agent_configuration():
"""
Test that DataDog logger correctly configures agent endpoint when DD_AGENT_HOST is set
Test that DataDog logger correctly configures agent endpoint when LITELLM_DD_AGENT_HOST is set.
Note: We use LITELLM_DD_AGENT_HOST instead of DD_AGENT_HOST to avoid conflicts
with ddtrace which automatically sets DD_AGENT_HOST for APM tracing.
"""
test_env = {
"DD_AGENT_HOST": "localhost",
"DD_AGENT_PORT": "10518",
"LITELLM_DD_AGENT_HOST": "localhost",
"LITELLM_DD_AGENT_PORT": "10518",
}
# Remove DD_SITE and DD_API_KEY to verify they're not required for agent mode
@ -654,4 +657,40 @@ def test_datadog_agent_configuration():
assert dd_logger.intake_url == "http://localhost:10518/api/v2/logs", f"Expected agent URL, got {dd_logger.intake_url}"
# Verify DD_API_KEY is optional (can be None)
assert dd_logger.DD_API_KEY is None or isinstance(dd_logger.DD_API_KEY, str)
assert dd_logger.DD_API_KEY is None or isinstance(dd_logger.DD_API_KEY, str)
def test_datadog_ignores_ddtrace_agent_host():
"""
Regression test: Ensure DD_AGENT_HOST set by ddtrace doesn't interfere with LiteLLM logging.
When users have ddtrace installed for APM tracing, it automatically sets DD_AGENT_HOST.
LiteLLM should ignore DD_AGENT_HOST and only use LITELLM_DD_AGENT_HOST for agent mode.
This prevents the 404 error when ddtrace's DD_AGENT_HOST points to an APM endpoint
that doesn't support /api/v2/logs.
Regression test for: https://github.com/BerriAI/litellm/issues/16379
"""
test_env = {
# User's explicit config for LiteLLM logging (direct API)
"DD_API_KEY": "fake-api-key",
"DD_SITE": "us5.datadoghq.com",
# ddtrace automatically sets these for APM tracing
"DD_AGENT_HOST": "10.176.100.40",
"DD_AGENT_PORT": "8126",
}
with patch.dict(os.environ, test_env, clear=False):
with patch("asyncio.create_task"):
dd_logger = DataDogLogger()
# Verify direct API endpoint is used (DD_AGENT_HOST should be ignored)
expected_url = "https://http-intake.logs.us5.datadoghq.com/api/v2/logs"
assert dd_logger.intake_url == expected_url, (
f"Expected direct API URL '{expected_url}', got '{dd_logger.intake_url}'. "
"DD_AGENT_HOST (set by ddtrace) should be ignored - only LITELLM_DD_AGENT_HOST should trigger agent mode."
)
# Verify API key is set correctly
assert dd_logger.DD_API_KEY == "fake-api-key"

8
tests/proxy_admin_ui_tests/e2e_ui_tests/view_internal_user.spec.ts
View file

@ -34,10 +34,10 @@ test("view internal user page", async ({ page }) => {
// The UI renders badges in each row - we just verify the column structure exists
const rowCount = await page.locator("tbody tr").count();
expect(rowCount).toBeGreaterThan(0);
// Verify table headers are present (including API Keys column)
const apiKeysHeader = page.locator("th", { hasText: "API Keys" });
await expect(apiKeysHeader).toBeVisible();
// Verify table headers are present (including Virtual Keys column)
const virtualKeysHeader = page.locator("th", { hasText: "Virtual Keys" });
await expect(virtualKeysHeader).toBeVisible();
// test pagination
// Wait for pagination controls to be visible

72
tests/test_litellm/llms/cohere/rerank/test_rerank_guardrail_handler.py
View file

@ -5,6 +5,7 @@ Unit tests for Cohere Rerank Guardrail Translation Handler
import asyncio
import os
import sys
from typing import List, Optional, Tuple
import pytest
@ -20,8 +21,10 @@ from litellm.types.utils import CallTypes
class MockGuardrail(CustomGuardrail):
"""Mock guardrail for testing"""
async def apply_guardrail(self, text: str, language=None, entities=None) -> str:
return f"{text} [GUARDRAILED]"
async def apply_guardrail(
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
return ([f"{text} [GUARDRAILED]" for text in texts], None)
class TestHandlerDiscovery:
@ -183,17 +186,20 @@ class TestPIIMaskingScenario:
"""Mock PII masking guardrail"""
async def apply_guardrail(
self, text: str, language=None, entities=None
) -> str:
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
import re
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
masked = masked.replace("John Doe", "[NAME_REDACTED]")
return masked
masked_texts = []
for text in texts:
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
masked = masked.replace("John Doe", "[NAME_REDACTED]")
masked_texts.append(masked)
return (masked_texts, None)
handler = CohereRerankHandler()
guardrail = PIIMaskingGuardrail(guardrail_name="mask_pii")
@ -231,21 +237,24 @@ class TestPIIMaskingScenario:
"""Mock PII masking guardrail"""
async def apply_guardrail(
self, text: str, language=None, entities=None
) -> str:
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
import re
# Mask emails
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
# Mask phone numbers
masked = re.sub(r"\d{3}-\d{3}-\d{4}", "[PHONE_REDACTED]", masked)
# Mask names
masked = masked.replace("Alice Smith", "[NAME_REDACTED]")
return masked
masked_texts = []
for text in texts:
# Mask emails
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
# Mask phone numbers
masked = re.sub(r"\d{3}-\d{3}-\d{4}", "[PHONE_REDACTED]", masked)
# Mask names
masked = masked.replace("Alice Smith", "[NAME_REDACTED]")
masked_texts.append(masked)
return (masked_texts, None)
handler = CohereRerankHandler()
guardrail = PIIMaskingGuardrail(guardrail_name="mask_pii")
@ -340,13 +349,16 @@ class TestContentFilteringScenario:
"""Mock content filter guardrail"""
async def apply_guardrail(
self, text: str, language=None, entities=None
) -> str:
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
bad_words = ["inappropriate", "offensive"]
filtered = text
for word in bad_words:
filtered = filtered.replace(word, "[FILTERED]")
return filtered
filtered_texts = []
for text in texts:
filtered = text
for word in bad_words:
filtered = filtered.replace(word, "[FILTERED]")
filtered_texts.append(filtered)
return (filtered_texts, None)
handler = CohereRerankHandler()
guardrail = ContentFilterGuardrail(guardrail_name="content_filter")

48
tests/test_litellm/llms/openai/completion/test_text_completion_guardrail_handler.py
View file

@ -4,6 +4,7 @@ Unit tests for OpenAI Text Completion Guardrail Translation Handler
import os
import sys
from typing import List, Optional, Tuple
from unittest.mock import MagicMock
import pytest
@ -21,8 +22,10 @@ from litellm.types.utils import CallTypes, TextChoices, TextCompletionResponse
class MockGuardrail(CustomGuardrail):
"""Mock guardrail for testing"""
async def apply_guardrail(self, text: str, language=None, entities=None) -> str:
return f"{text} [GUARDRAILED]"
async def apply_guardrail(
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
return ([f"{text} [GUARDRAILED]" for text in texts], None)
class TestHandlerDiscovery:
@ -243,19 +246,22 @@ class TestPIIMaskingScenario:
"""Mock PII masking guardrail"""
async def apply_guardrail(
self, text: str, language=None, entities=None
) -> str:
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
# Simple mock: replace email-like patterns
import re
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
# Replace names (simple mock)
masked = masked.replace("John Doe", "[NAME_REDACTED]")
return masked
masked_texts = []
for text in texts:
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
# Replace names (simple mock)
masked = masked.replace("John Doe", "[NAME_REDACTED]")
masked_texts.append(masked)
return (masked_texts, None)
handler = OpenAITextCompletionHandler()
guardrail = PIIMaskingGuardrail(guardrail_name="mask_pii")
@ -303,15 +309,19 @@ class TestPIIMaskingScenario:
"""Mock PII masking guardrail"""
async def apply_guardrail(
self, text: str, language=None, entities=None
) -> str:
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
import re
return re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
masked_texts = []
for text in texts:
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
masked_texts.append(masked)
return (masked_texts, None)
handler = OpenAITextCompletionHandler()
guardrail = PIIMaskingGuardrail(guardrail_name="mask_pii")

30
tests/test_litellm/llms/openai/image_generation/test_image_generation_guardrail_handler.py
View file

@ -4,6 +4,7 @@ Unit tests for OpenAI Image Generation Guardrail Translation Handler
import os
import sys
from typing import List, Optional, Tuple
import pytest
@ -20,8 +21,10 @@ from litellm.types.utils import CallTypes, ImageObject, ImageResponse
class MockGuardrail(CustomGuardrail):
"""Mock guardrail for testing"""
async def apply_guardrail(self, text: str, language=None, entities=None) -> str:
return f"{text} [GUARDRAILED]"
async def apply_guardrail(
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
return ([f"{text} [GUARDRAILED]" for text in texts], None)
class TestHandlerDiscovery:
@ -141,19 +144,22 @@ class TestPIIMaskingScenario:
"""Mock PII masking guardrail"""
async def apply_guardrail(
self, text: str, language=None, entities=None
) -> str:
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
# Simple mock: replace email-like patterns
import re
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
# Replace names (simple mock)
masked = masked.replace("John Doe", "[NAME_REDACTED]")
return masked
masked_texts = []
for text in texts:
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
# Replace names (simple mock)
masked = masked.replace("John Doe", "[NAME_REDACTED]")
masked_texts.append(masked)
return (masked_texts, None)
handler = OpenAIImageGenerationHandler()
guardrail = PIIMaskingGuardrail(guardrail_name="mask_pii")

Some files were not shown because too many files have changed in this diff Show more