scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-07 02:59:05 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 103

Merge remote-tracking branch 'origin' into litellm_login_route_refactor

Browse source
This commit is contained in:
yuneng-jiang 2025-12-03 10:40:11 -08:00
commit b3c0ea5414
139 changed files with 8217 additions and 2118 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

15
Dockerfile
View file

@ -1,8 +1,8 @@
# Base image for building
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/python:latest-dev
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base
# Runtime image
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/python:latest-dev
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base
# Builder stage
FROM $LITELLM_BUILD_IMAGE AS builder
@ -12,11 +12,9 @@ WORKDIR /app
USER root
# Install build dependencies
RUN apk add --no-cache gcc python3-dev openssl openssl-dev
RUN apk add --no-cache bash gcc py3-pip python3 python3-dev openssl openssl-dev
RUN pip install --upgrade pip>=24.3.1 && \
pip install build
RUN python -m pip install build
# Copy the current directory contents into the container at /app
COPY . .
@ -48,10 +46,7 @@ FROM $LITELLM_RUNTIME_IMAGE AS runtime
USER root
# Install runtime dependencies
RUN apk add --no-cache openssl tzdata nodejs npm
# Upgrade pip to fix CVE-2025-8869
RUN pip install --upgrade pip>=24.3.1
RUN apk add --no-cache bash openssl tzdata nodejs npm python3 py3-pip
WORKDIR /app
# Copy the current directory contents into the container at /app

4
deploy/charts/litellm-helm/Chart.yaml
View file

@ -18,7 +18,7 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.4.8
version: 0.4.9
# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
@ -33,5 +33,5 @@ dependencies:
condition: db.deployStandalone
- name: redis
version: ">=18.0.0"
repository: oci://registry-1.docker.io/bitnamicharts
repository: oci://registry-1.docker.io/bitnamicharts
condition: redis.enabled

125
deploy/charts/litellm-helm/README.md
View file

@ -10,46 +10,48 @@
- Helm 3.8.0+
If `db.deployStandalone` is used:
- PV provisioner support in the underlying infrastructure
If `db.useStackgresOperator` is used (not yet implemented):
- The Stackgres Operator must already be installed in the Kubernetes Cluster. This chart will **not** install the operator if it is missing.
- The Stackgres Operator must already be installed in the Kubernetes Cluster. This chart will **not** install the operator if it is missing.
## Parameters
### LiteLLM Proxy Deployment Settings
| Name | Description | Value |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
| `replicaCount` | The number of LiteLLM Proxy pods to be deployed | `1` |
| `masterkeySecretName` | The name of the Kubernetes Secret that contains the Master API Key for LiteLLM. If not specified, use the generated secret name. | N/A |
| `masterkeySecretKey` | The key within the Kubernetes Secret that contains the Master API Key for LiteLLM. If not specified, use `masterkey` as the key. | N/A |
| `masterkey` | The Master API Key for LiteLLM. If not specified, a random key in the `sk-...` format is generated. | N/A |
| `environmentSecrets` | An optional array of Secret object names. The keys and values in these secrets will be presented to the LiteLLM proxy pod as environment variables. See below for an example Secret object. | `[]` |
| `environmentConfigMaps` | An optional array of ConfigMap object names. The keys and values in these configmaps will be presented to the LiteLLM proxy pod as environment variables. See below for an example Secret object. | `[]` |
| `image.repository` | LiteLLM Proxy image repository | `ghcr.io/berriai/litellm` |
| `image.pullPolicy` | LiteLLM Proxy image pull policy | `IfNotPresent` |
| `image.tag` | Overrides the image tag whose default the latest version of LiteLLM at the time this chart was published. | `""` |
| `imagePullSecrets` | Registry credentials for the LiteLLM and initContainer images. | `[]` |
| `serviceAccount.create` | Whether or not to create a Kubernetes Service Account for this deployment. The default is `false` because LiteLLM has no need to access the Kubernetes API. | `false` |
| `service.type` | Kubernetes Service type (e.g. `LoadBalancer`, `ClusterIP`, etc.) | `ClusterIP` |
| `service.port` | TCP port that the Kubernetes Service will listen on. Also the TCP port within the Pod that the proxy will listen on. | `4000` |
| `service.loadBalancerClass` | Optional LoadBalancer implementation class (only used when `service.type` is `LoadBalancer`) | `""` |
| `ingress.*` | See [values.yaml](./values.yaml) for example settings | N/A |
| `proxyConfigMap.create` | When `true`, render a ConfigMap from `.Values.proxy_config` and mount it. | `true` |
| `proxyConfigMap.name` | When `create=false`, name of the existing ConfigMap to mount. | `""` |
| `proxyConfigMap.key` | Key in the ConfigMap that contains the proxy config file. | `"config.yaml"` |
| `proxy_config.*` | See [values.yaml](./values.yaml) for default settings. Rendered into the ConfigMap’s `config.yaml` only when `proxyConfigMap.create=true`. See [example_config_yaml](../../../litellm/proxy/example_config_yaml/) for configuration examples. | `N/A` |
| `extraContainers[]` | An array of additional containers to be deployed as sidecars alongside the LiteLLM Proxy.
| `pdb.enabled` | Enable a PodDisruptionBudget for the LiteLLM proxy Deployment | `false` |
| `pdb.minAvailable` | Minimum number/percentage of pods that must be available during **voluntary** disruptions (choose **one** of minAvailable/maxUnavailable) | `null` |
| `pdb.maxUnavailable` | Maximum number/percentage of pods that can be unavailable during **voluntary** disruptions (choose **one** of minAvailable/maxUnavailable) | `null` |
| `pdb.annotations` | Extra metadata annotations to add to the PDB | `{}` |
| `pdb.labels` | Extra metadata labels to add to the PDB | `{}` |
| Name | Description | Value |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| `replicaCount` | The number of LiteLLM Proxy pods to be deployed | `1` |
| `masterkeySecretName` | The name of the Kubernetes Secret that contains the Master API Key for LiteLLM. If not specified, use the generated secret name. | N/A |
| `masterkeySecretKey` | The key within the Kubernetes Secret that contains the Master API Key for LiteLLM. If not specified, use `masterkey` as the key. | N/A |
| `masterkey` | The Master API Key for LiteLLM. If not specified, a random key in the `sk-...` format is generated. | N/A |
| `environmentSecrets` | An optional array of Secret object names. The keys and values in these secrets will be presented to the LiteLLM proxy pod as environment variables. See below for an example Secret object. | `[]` |
| `environmentConfigMaps` | An optional array of ConfigMap object names. The keys and values in these configmaps will be presented to the LiteLLM proxy pod as environment variables. See below for an example Secret object. | `[]` |
| `image.repository` | LiteLLM Proxy image repository | `ghcr.io/berriai/litellm` |
| `image.pullPolicy` | LiteLLM Proxy image pull policy | `IfNotPresent` |
| `image.tag` | Overrides the image tag whose default the latest version of LiteLLM at the time this chart was published. | `""` |
| `imagePullSecrets` | Registry credentials for the LiteLLM and initContainer images. | `[]` |
| `serviceAccount.create` | Whether or not to create a Kubernetes Service Account for this deployment. The default is `false` because LiteLLM has no need to access the Kubernetes API. | `false` |
| `service.type` | Kubernetes Service type (e.g. `LoadBalancer`, `ClusterIP`, etc.) | `ClusterIP` |
| `service.port` | TCP port that the Kubernetes Service will listen on. Also the TCP port within the Pod that the proxy will listen on. | `4000` |
| `service.loadBalancerClass` | Optional LoadBalancer implementation class (only used when `service.type` is `LoadBalancer`) | `""` |
| `ingress.labels` | Additional labels for the Ingress resource | `{}` |
| `ingress.*` | See [values.yaml](./values.yaml) for example settings | N/A |
| `proxyConfigMap.create` | When `true`, render a ConfigMap from `.Values.proxy_config` and mount it. | `true` |
| `proxyConfigMap.name` | When `create=false`, name of the existing ConfigMap to mount. | `""` |
| `proxyConfigMap.key` | Key in the ConfigMap that contains the proxy config file. | `"config.yaml"` |
| `proxy_config.*` | See [values.yaml](./values.yaml) for default settings. Rendered into the ConfigMap’s `config.yaml` only when `proxyConfigMap.create=true`. See [example_config_yaml](../../../litellm/proxy/example_config_yaml/) for configuration examples. | `N/A` |
| `extraContainers[]` | An array of additional containers to be deployed as sidecars alongside the LiteLLM Proxy. |
| `pdb.enabled` | Enable a PodDisruptionBudget for the LiteLLM proxy Deployment | `false` |
| `pdb.minAvailable` | Minimum number/percentage of pods that must be available during **voluntary** disruptions (choose **one** of minAvailable/maxUnavailable) | `null` |
| `pdb.maxUnavailable` | Maximum number/percentage of pods that can be unavailable during **voluntary** disruptions (choose **one** of minAvailable/maxUnavailable) | `null` |
| `pdb.annotations` | Extra metadata annotations to add to the PDB | `{}` |
| `pdb.labels` | Extra metadata labels to add to the PDB | `{}` |
#### Example `proxy_config` ConfigMap from values (default):
```
proxyConfigMap:
create: true
@ -67,7 +69,6 @@ proxy_config:
#### Example using existing `proxyConfigMap` instead of creating it:
```
proxyConfigMap:
create: false
@ -77,8 +78,7 @@ proxyConfigMap:
# proxy_config is ignored in this mode
```
#### Example `environmentSecrets` Secret
#### Example `environmentSecrets` Secret
```
apiVersion: v1
@ -91,21 +91,23 @@ type: Opaque
```
### Database Settings
| Name | Description | Value |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
| `db.useExisting` | Use an existing Postgres database. A Kubernetes Secret object must exist that contains credentials for connecting to the database. An example secret object definition is provided below. | `false` |
| `db.endpoint` | If `db.useExisting` is `true`, this is the IP, Hostname or Service Name of the Postgres server to connect to. | `localhost` |
| `db.database` | If `db.useExisting` is `true`, the name of the existing database to connect to. | `litellm` |
| `db.url` | If `db.useExisting` is `true`, the connection url of the existing database to connect to can be overwritten with this value. | `postgresql://$(DATABASE_USERNAME):$(DATABASE_PASSWORD)@$(DATABASE_HOST)/$(DATABASE_NAME)` |
| `db.secret.name` | If `db.useExisting` is `true`, the name of the Kubernetes Secret that contains credentials. | `postgres` |
| `db.secret.usernameKey` | If `db.useExisting` is `true`, the name of the key within the Kubernetes Secret that holds the username for authenticating with the Postgres instance. | `username` |
| `db.secret.passwordKey` | If `db.useExisting` is `true`, the name of the key within the Kubernetes Secret that holds the password associates with the above user. | `password` |
| `db.useStackgresOperator` | Not yet implemented. | `false` |
| `db.deployStandalone` | Deploy a standalone, single instance deployment of Postgres, using the Bitnami postgresql chart. This is useful for getting started but doesn't provide HA or (by default) data backups. | `true` |
| `postgresql.*` | If `db.deployStandalone` is `true`, configuration passed to the Bitnami postgresql chart. See the [Bitnami Documentation](https://github.com/bitnami/charts/tree/main/bitnami/postgresql) for full configuration details. See [values.yaml](./values.yaml) for the default configuration. | See [values.yaml](./values.yaml) |
| `postgresql.auth.*` | If `db.deployStandalone` is `true`, care should be taken to ensure the default `password` and `postgres-password` values are **NOT** used. | `NoTaGrEaTpAsSwOrD` |
| Name | Description | Value |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| `db.useExisting` | Use an existing Postgres database. A Kubernetes Secret object must exist that contains credentials for connecting to the database. An example secret object definition is provided below. | `false` |
| `db.endpoint` | If `db.useExisting` is `true`, this is the IP, Hostname or Service Name of the Postgres server to connect to. | `localhost` |
| `db.database` | If `db.useExisting` is `true`, the name of the existing database to connect to. | `litellm` |
| `db.url` | If `db.useExisting` is `true`, the connection url of the existing database to connect to can be overwritten with this value. | `postgresql://$(DATABASE_USERNAME):$(DATABASE_PASSWORD)@$(DATABASE_HOST)/$(DATABASE_NAME)` |
| `db.secret.name` | If `db.useExisting` is `true`, the name of the Kubernetes Secret that contains credentials. | `postgres` |
| `db.secret.usernameKey` | If `db.useExisting` is `true`, the name of the key within the Kubernetes Secret that holds the username for authenticating with the Postgres instance. | `username` |
| `db.secret.passwordKey` | If `db.useExisting` is `true`, the name of the key within the Kubernetes Secret that holds the password associates with the above user. | `password` |
| `db.useStackgresOperator` | Not yet implemented. | `false` |
| `db.deployStandalone` | Deploy a standalone, single instance deployment of Postgres, using the Bitnami postgresql chart. This is useful for getting started but doesn't provide HA or (by default) data backups. | `true` |
| `postgresql.*` | If `db.deployStandalone` is `true`, configuration passed to the Bitnami postgresql chart. See the [Bitnami Documentation](https://github.com/bitnami/charts/tree/main/bitnami/postgresql) for full configuration details. See [values.yaml](./values.yaml) for the default configuration. | See [values.yaml](./values.yaml) |
| `postgresql.auth.*` | If `db.deployStandalone` is `true`, care should be taken to ensure the default `password` and `postgres-password` values are **NOT** used. | `NoTaGrEaTpAsSwOrD` |
#### Example Postgres `db.useExisting` Secret
```yaml
apiVersion: v1
kind: Secret
@ -143,7 +145,7 @@ metadata:
name: litellm-env-secret
type: Opaque
data:
SOME_PASSWORD: cDZbUGVXeU5e0ZW # base64 encoded
SOME_PASSWORD: cDZbUGVXeU5e0ZW # base64 encoded
ANOTHER_PASSWORD: AAZbUGVXeU5e0ZB # base64 encoded
```
@ -153,23 +155,23 @@ Source: [GitHub Gist from troyharvey](https://gist.github.com/troyharvey/4506472
The migration job supports both ArgoCD and Helm hooks to ensure database migrations run at the appropriate time during deployments.
| Name | Description | Value |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
| `migrationJob.enabled` | Enable or disable the schema migration Job | `true` |
| `migrationJob.backoffLimit` | Backoff limit for Job restarts | `4` |
| `migrationJob.ttlSecondsAfterFinished` | TTL for completed migration jobs | `120` |
| `migrationJob.annotations` | Additional annotations for the migration job pod | `{}` |
| `migrationJob.extraContainers` | Additional containers to run alongside the migration job | `[]` |
| `migrationJob.hooks.argocd.enabled` | Enable ArgoCD hooks for the migration job (uses PreSync hook with BeforeHookCreation delete policy) | `true` |
| `migrationJob.hooks.helm.enabled` | Enable Helm hooks for the migration job (uses pre-install,pre-upgrade hooks with before-hook-creation delete policy) | `false` |
| `migrationJob.hooks.helm.weight` | Helm hook execution order (lower weights executed first). Optional - defaults to "1" if not specified. | N/A |
| Name | Description | Value |
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ------- |
| `migrationJob.enabled` | Enable or disable the schema migration Job | `true` |
| `migrationJob.backoffLimit` | Backoff limit for Job restarts | `4` |
| `migrationJob.ttlSecondsAfterFinished` | TTL for completed migration jobs | `120` |
| `migrationJob.annotations` | Additional annotations for the migration job pod | `{}` |
| `migrationJob.extraContainers` | Additional containers to run alongside the migration job | `[]` |
| `migrationJob.hooks.argocd.enabled` | Enable ArgoCD hooks for the migration job (uses PreSync hook with BeforeHookCreation delete policy) | `true` |
| `migrationJob.hooks.helm.enabled` | Enable Helm hooks for the migration job (uses pre-install,pre-upgrade hooks with before-hook-creation delete policy) | `false` |
| `migrationJob.hooks.helm.weight` | Helm hook execution order (lower weights executed first). Optional - defaults to "1" if not specified. | N/A |
## Accessing the Admin UI
When browsing to the URL published per the settings in `ingress.*`, you will
be prompted for **Admin Configuration**. The **Proxy Endpoint** is the internal
be prompted for **Admin Configuration**. The **Proxy Endpoint** is the internal
(from the `litellm` pod's perspective) URL published by the `<RELEASE>-litellm`
Kubernetes Service. If the deployment uses the default settings for this
Kubernetes Service. If the deployment uses the default settings for this
service, the **Proxy Endpoint** should be set to `http://<RELEASE>-litellm:4000`.
The **Proxy Key** is the value specified for `masterkey` or, if a `masterkey`
@ -181,7 +183,8 @@ kubectl -n litellm get secret <RELEASE>-litellm-masterkey -o jsonpath="{.data.ma
```
## Admin UI Limitations
At the time of writing, the Admin UI is unable to add models. This is because
At the time of writing, the Admin UI is unable to add models. This is because
it would need to update the `config.yaml` file which is a exposed ConfigMap, and
therefore, read-only. This is a limitation of this helm chart, not the Admin UI
therefore, read-only. This is a limitation of this helm chart, not the Admin UI
itself.

3
deploy/charts/litellm-helm/templates/ingress.yaml
View file

@ -18,6 +18,9 @@ metadata:
name: {{ $fullName }}
labels:
{{- include "litellm.labels" . | nindent 4 }}
{{- with .Values.ingress.labels }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}

45
deploy/charts/litellm-helm/tests/ingress_tests.yaml Normal file
View file

@ -0,0 +1,45 @@
suite: Ingress Configuration Tests
templates:
- ingress.yaml
tests:
- it: should not create Ingress by default
asserts:
- hasDocuments:
count: 0
- it: should create Ingress when enabled
set:
ingress.enabled: true
asserts:
- hasDocuments:
count: 1
- isKind:
of: Ingress
- it: should add custom labels
set:
ingress.enabled: true
ingress.labels:
custom-label: "true"
another-label: "value"
asserts:
- isKind:
of: Ingress
- equal:
path: metadata.labels.custom-label
value: "true"
- equal:
path: metadata.labels.another-label
value: "value"
- it: should add annotations
set:
ingress.enabled: true
ingress.annotations:
kubernetes.io/ingress.class: "nginx"
asserts:
- isKind:
of: Ingress
- equal:
path: metadata.annotations["kubernetes.io/ingress.class"]
value: "nginx"

45
deploy/charts/litellm-helm/values.yaml
View file

@ -35,7 +35,8 @@ podAnnotations: {}
podLabels: {}
terminationGracePeriodSeconds: 90
topologySpreadConstraints: []
topologySpreadConstraints:
[]
# - maxSkew: 1
# topologyKey: kubernetes.io/hostname
# whenUnsatisfiable: DoNotSchedule
@ -46,7 +47,8 @@ topologySpreadConstraints: []
# At the time of writing, the litellm docker image requires write access to the
# filesystem on startup so that prisma can install some dependencies.
podSecurityContext: {}
securityContext: {}
securityContext:
{}
# capabilities:
# drop:
# - ALL
@ -57,13 +59,15 @@ securityContext: {}
# A list of Kubernetes Secret objects that will be exported to the LiteLLM proxy
# pod as environment variables. These secrets can then be referenced in the
# configuration file (or "litellm" ConfigMap) with `os.environ/<Env Var Name>`
environmentSecrets: []
environmentSecrets:
[]
# - litellm-env-secret
# A list of Kubernetes ConfigMap objects that will be exported to the LiteLLM proxy
# pod as environment variables. The ConfigMap kv-pairs can then be referenced in the
# configuration file (or "litellm" ConfigMap) with `os.environ/<Env Var Name>`
environmentConfigMaps: []
environmentConfigMaps:
[]
# - litellm-env-configmap
service:
@ -82,7 +86,9 @@ separateHealthPort: 8081
ingress:
enabled: false
className: "nginx"
annotations: {}
labels: {}
annotations:
{}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts:
@ -129,7 +135,8 @@ proxy_config:
general_settings:
master_key: os.environ/PROXY_MASTER_KEY
resources: {}
resources:
{}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
@ -231,7 +238,7 @@ migrationJob:
# cpu: 100m
# memory: 100Mi
extraContainers: []
# Hook configuration
hooks:
argocd:
@ -240,30 +247,30 @@ migrationJob:
enabled: false
# Additional environment variables to be added to the deployment as a map of key-value pairs
envVars: {
# USE_DDTRACE: "true"
}
envVars: {}
# USE_DDTRACE: "true"
# Additional environment variables to be added to the deployment as a list of k8s env vars
extraEnvVars: {
# - name: EXTRA_ENV_VAR
# value: EXTRA_ENV_VAR_VALUE
}
extraEnvVars: {}
# - name: EXTRA_ENV_VAR
# value: EXTRA_ENV_VAR_VALUE
# Pod Disruption Budget
pdb:
enabled: false
# Set exactly one of the following. If both are set, minAvailable takes precedence.
minAvailable: null # e.g. "50%" or 1
maxUnavailable: null # e.g. 1 or "20%"
minAvailable: null # e.g. "50%" or 1
maxUnavailable: null # e.g. 1 or "20%"
annotations: {}
labels: {}
serviceMonitor:
enabled: false
labels: {}
labels:
{}
# test: test
annotations: {}
annotations:
{}
# kubernetes.io/test: test
interval: 15s
scrapeTimeout: 10s
@ -273,4 +280,4 @@ serviceMonitor:
# action: replace
namespaceSelector:
matchNames: []
# - test-namespace
# - test-namespace

16
docker/Dockerfile.database
View file

@ -1,8 +1,8 @@
# Base image for building
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/python:latest-dev
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base
# Runtime image
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/python:latest-dev
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base
# Builder stage
FROM $LITELLM_BUILD_IMAGE AS builder
@ -13,13 +13,15 @@ USER root
# Install build dependencies
RUN apk add --no-cache \
build-base \
bash \
gcc \
py3-pip \
python3 \
python3-dev \
openssl \
openssl-dev
RUN pip install --upgrade pip && \
pip install build
RUN python -m pip install build
# Copy the current directory contents into the container at /app
COPY . .
@ -46,7 +48,7 @@ FROM $LITELLM_RUNTIME_IMAGE AS runtime
USER root
# Install runtime dependencies
RUN apk add --no-cache openssl
RUN apk add --no-cache bash openssl tzdata nodejs npm python3 py3-pip
WORKDIR /app
# Copy the current directory contents into the container at /app

19
docker/Dockerfile.non_root
View file

@ -1,6 +1,6 @@
# Base images
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/python:latest-dev
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/python:latest-dev
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base
# -----------------
# Builder Stage
@ -10,7 +10,18 @@ WORKDIR /app
# Install build dependencies including Node.js for UI build
USER root
RUN apk add --no-cache build-base bash nodejs npm \
RUN apk add --no-cache \
python3 \
py3-pip \
clang \
llvm \
lld \
gcc \
linux-headers \
build-base \
bash \
nodejs \
npm \
&& pip install --no-cache-dir --upgrade pip build
# Copy project files
@ -62,7 +73,7 @@ WORKDIR /app
# Install runtime dependencies
USER root
RUN apk upgrade --no-cache && \
apk add --no-cache bash libstdc++ ca-certificates openssl supervisor
apk add --no-cache python3 py3-pip bash openssl tzdata nodejs npm supervisor
# Copy only necessary artifacts from builder stage for runtime
COPY . .

1
docs/my-website/docs/completion/knowledgebase.md
View file

@ -21,6 +21,7 @@ LiteLLM integrates with vector stores, allowing your models to access your organ
- [Azure Vector Stores](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/file-search?tabs=python#vector-stores) (Cannot be directly queried. Only available for calling in Assistants messages. We will be adding Azure AI Search Vector Store API support soon.)
- [Vertex AI RAG API](https://cloud.google.com/vertex-ai/generative-ai/docs/rag-overview)
- [Gemini File Search](https://ai.google.dev/gemini-api/docs/file-search)
- [RAGFlow Datasets](/docs/providers/ragflow_vector_store.md) (Dataset management only, search not supported)
## Quick Start

108
docs/my-website/docs/getting_started.md
View file

@ -1,108 +0,0 @@
# Getting Started
import QuickStart from '../src/components/QuickStart.js'
LiteLLM simplifies LLM API calls by mapping them all to the [OpenAI ChatCompletion format](https://platform.openai.com/docs/api-reference/chat).
## basic usage
By default we provide a free $10 community-key to try all providers supported on LiteLLM.
```python
from litellm import completion
## set ENV variables
os.environ["OPENAI_API_KEY"] = "your-api-key"
os.environ["COHERE_API_KEY"] = "your-api-key"
messages = [{ "content": "Hello, how are you?","role": "user"}]
# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)
# cohere call
response = completion("command-nightly", messages)
```
**Need a dedicated key?**
Email us @ krrish@berri.ai
Next Steps 👉 [Call all supported models - e.g. Claude-2, Llama2-70b, etc.](./proxy_api.md#supported-models)
More details 👉
- [Completion() function details](./completion/)
- [Overview of supported models / providers on LiteLLM](./providers/)
- [Search all models / providers](https://models.litellm.ai/)
- [Build your own OpenAI proxy](https://github.com/BerriAI/liteLLM-proxy/tree/main)
## streaming
Same example from before. Just pass in `stream=True` in the completion args.
```python
from litellm import completion
## set ENV variables
os.environ["OPENAI_API_KEY"] = "openai key"
os.environ["COHERE_API_KEY"] = "cohere key"
messages = [{ "content": "Hello, how are you?","role": "user"}]
# openai call
response = completion(model="gpt-3.5-turbo", messages=messages, stream=True)
# cohere call
response = completion("command-nightly", messages, stream=True)
print(response)
```
More details 👉
- [streaming + async](./completion/stream.md)
- [tutorial for streaming Llama2 on TogetherAI](./tutorials/TogetherAI_liteLLM.md)
## exception handling
LiteLLM maps exceptions across all supported providers to the OpenAI exceptions. All our exceptions inherit from OpenAI's exception types, so any error-handling you have for that, should work out of the box with LiteLLM.
```python
from openai.error import OpenAIError
from litellm import completion
os.environ["ANTHROPIC_API_KEY"] = "bad-key"
try:
# some code
completion(model="claude-instant-1", messages=[{"role": "user", "content": "Hey, how's it going?"}])
except OpenAIError as e:
print(e)
```
## Logging Observability - Log LLM Input/Output ([Docs](https://docs.litellm.ai/docs/observability/callbacks))
LiteLLM exposes pre defined callbacks to send data to MLflow, Lunary, Langfuse, Helicone, Promptlayer, Traceloop, Slack
```python
from litellm import completion
## set env variables for logging tools (API key set up is not required when using MLflow)
os.environ["LUNARY_PUBLIC_KEY"] = "your-lunary-public-key" # get your public key at https://app.lunary.ai/settings
os.environ["HELICONE_API_KEY"] = "your-helicone-key"
os.environ["LANGFUSE_PUBLIC_KEY"] = ""
os.environ["LANGFUSE_SECRET_KEY"] = ""
os.environ["OPENAI_API_KEY"]
# set callbacks
litellm.success_callback = ["lunary", "mlflow", "langfuse", "helicone"] # log input/output to MLflow, langfuse, lunary, helicone
#openai call
response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hi 👋 - i'm openai"}])
```
More details 👉
- [exception mapping](./exception_mapping.md)
- [retries + model fallbacks for completion()](./completion/reliable_completions.md)
- [tutorial for model fallbacks with completion()](./tutorials/fallbacks.md)

18
docs/my-website/docs/observability/datadog.md
View file

@ -71,17 +71,19 @@ DD_SOURCE="litellm_dev" # [OPTIONAL] your datadog source. use to different
Send logs through a local DataDog agent (useful for containerized environments):
```shell
DD_AGENT_HOST="localhost" # hostname or IP of DataDog agent
DD_AGENT_PORT="10518" # [OPTIONAL] port of DataDog agent (default: 10518)
DD_API_KEY="5f2d0f310***********" # [OPTIONAL] your datadog API Key (agent handles auth)
DD_SOURCE="litellm_dev" # [OPTIONAL] your datadog source
LITELLM_DD_AGENT_HOST="localhost" # hostname or IP of DataDog agent
LITELLM_DD_AGENT_PORT="10518" # [OPTIONAL] port of DataDog agent (default: 10518)
DD_API_KEY="5f2d0f310***********" # [OPTIONAL] your datadog API Key (agent handles auth)
DD_SOURCE="litellm_dev" # [OPTIONAL] your datadog source
```
When `DD_AGENT_HOST` is set, logs are sent to the agent instead of directly to DataDog API. This is useful for:
When `LITELLM_DD_AGENT_HOST` is set, logs are sent to the agent instead of directly to DataDog API. This is useful for:
- Centralized log shipping in containerized environments
- Reducing direct API calls from multiple services
- Leveraging agent-side processing and filtering
**Note:** We use `LITELLM_DD_AGENT_HOST` instead of `DD_AGENT_HOST` to avoid conflicts with `ddtrace` which automatically sets `DD_AGENT_HOST` for APM tracing.
**Step 3**: Start the proxy, make a test request
Start proxy
@ -191,8 +193,8 @@ LiteLLM supports customizing the following Datadog environment variables
|---------------------|-------------|---------------|----------|
| `DD_API_KEY` | Your Datadog API key for authentication (required for direct API, optional for agent) | None | Conditional* |
| `DD_SITE` | Your Datadog site (e.g., "us5.datadoghq.com") (required for direct API) | None | Conditional* |
| `DD_AGENT_HOST` | Hostname or IP of DataDog agent (e.g., "localhost"). When set, logs are sent to agent instead of direct API | None | ❌ No |
| `DD_AGENT_PORT` | Port of DataDog agent for log intake | "10518" | ❌ No |
| `LITELLM_DD_AGENT_HOST` | Hostname or IP of DataDog agent (e.g., "localhost"). When set, logs are sent to agent instead of direct API | None | ❌ No |
| `LITELLM_DD_AGENT_PORT` | Port of DataDog agent for log intake | "10518" | ❌ No |
| `DD_ENV` | Environment tag for your logs (e.g., "production", "staging") | "unknown" | ❌ No |
| `DD_SERVICE` | Service name for your logs | "litellm-server" | ❌ No |
| `DD_SOURCE` | Source name for your logs | "litellm" | ❌ No |
@ -201,5 +203,5 @@ LiteLLM supports customizing the following Datadog environment variables
| `POD_NAME` | Pod name tag (useful for Kubernetes deployments) | "unknown" | ❌ No |
\* **Required when using Direct API** (default): `DD_API_KEY` and `DD_SITE` are required
\* **Optional when using DataDog Agent**: Set `DD_AGENT_HOST` to use agent mode; `DD_API_KEY` and `DD_SITE` are not required
\* **Optional when using DataDog Agent**: Set `LITELLM_DD_AGENT_HOST` to use agent mode; `DD_API_KEY` and `DD_SITE` are not required

79
docs/my-website/docs/observability/phoenix_integration.md
View file

@ -6,7 +6,7 @@ Open source tracing and evaluation platform
:::tip
This is community maintained, Please make an issue if you run into a bug
This is community maintained. Please make an issue if you run into a bug:
https://github.com/BerriAI/litellm
:::
@ -31,19 +31,16 @@ litellm.callbacks = ["arize_phoenix"]
import litellm
import os
os.environ["PHOENIX_API_KEY"] = "" # Necessary only using Phoenix Cloud
os.environ["PHOENIX_COLLECTOR_HTTP_ENDPOINT"] = "" # The URL of your Phoenix OSS instance e.g. http://localhost:6006/v1/traces
os.environ["PHOENIX_PROJECT_NAME"]="litellm" # OPTIONAL: you can configure project names, otherwise traces would go to "default" project
# Set env variables
os.environ["PHOENIX_API_KEY"] = "d0*****" # Set the Phoenix API key here. It is necessary only when using Phoenix Cloud.
os.environ["PHOENIX_COLLECTOR_HTTP_ENDPOINT"] = "https://app.phoenix.arize.com/s/<space-name>/v1/traces" # Set the URL of your Phoenix OSS instance, otherwise tracer would use https://app.phoenix.arize.com/v1/traces for Phoenix Cloud.
os.environ["PHOENIX_PROJECT_NAME"] = "litellm" # Configure the project name, otherwise traces would go to "default" project.
os.environ['OPENAI_API_KEY'] = "fake-key" # Set the OpenAI API key here.
# This defaults to https://app.phoenix.arize.com/v1/traces for Phoenix Cloud
# LLM API Keys
os.environ['OPENAI_API_KEY']=""
# set arize as a callback, litellm will send the data to arize
# Set arize_phoenix as a callback & LiteLLM will send the data to Phoenix.
litellm.callbacks = ["arize_phoenix"]
# openai call
# OpenAI call
response = litellm.completion(
model="gpt-3.5-turbo",
messages=[
@ -52,8 +49,9 @@ response = litellm.completion(
)
```
### Using with LiteLLM Proxy
## Using with LiteLLM Proxy
1. Setup config.yaml
```yaml
model_list:
@ -66,12 +64,63 @@ model_list:
litellm_settings:
callbacks: ["arize_phoenix"]
general_settings:
master_key: "sk-1234"
environment_variables:
PHOENIX_API_KEY: "d0*****"
PHOENIX_COLLECTOR_ENDPOINT: "https://app.phoenix.arize.com/v1/traces" # OPTIONAL, for setting the GRPC endpoint
PHOENIX_COLLECTOR_HTTP_ENDPOINT: "https://app.phoenix.arize.com/v1/traces" # OPTIONAL, for setting the HTTP endpoint
PHOENIX_COLLECTOR_ENDPOINT: "https://app.phoenix.arize.com/s/<space-name>/v1/traces" # OPTIONAL - For setting the gRPC endpoint
PHOENIX_COLLECTOR_HTTP_ENDPOINT: "https://app.phoenix.arize.com/s/<space-name>/v1/traces" # OPTIONAL - For setting the HTTP endpoint
```
2. Start the proxy
```bash
litellm --config config.yaml
```
3. Test it!
```bash
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Hi 👋 - i'm openai"}]}'
```
## Supported Phoenix Endpoints
Phoenix now supports multiple deployment types. The correct endpoint depends on which version of Phoenix Cloud you are using.
**Phoenix Cloud (With Spaces - New Version)**
Use this if your Phoenix URL contains `/s/<space-name>` path.
```bash
https://app.phoenix.arize.com/s/<space-name>/v1/traces
```
**Phoenix Cloud (Legacy - Deprecated)**
Use this only if your deployment still shows the `/legacy` pattern.
```bash
https://app.phoenix.arize.com/legacy/v1/traces
```
**Phoenix Cloud (Without Spaces - Old Version)**
Use this if your Phoenix Cloud URL does not contain `/s/<space-name>` or `/legacy` path.
```bash
https://app.phoenix.arize.com/v1/traces
```
**Self-Hosted Phoenix (Local Instance)**
Use this when running Phoenix on your machine or a private server.
```bash
http://localhost:6006/v1/traces
```
Depending on which Phoenix Cloud version or deployment you are using, you should set the corresponding endpoint in `PHOENIX_COLLECTOR_HTTP_ENDPOINT` or `PHOENIX_COLLECTOR_ENDPOINT`.
## Support & Talk to Founders
- [Schedule Demo 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)

10
docs/my-website/docs/projects/Agent Lightning.md Normal file
View file

@ -0,0 +1,10 @@
# Agent Lightning
[Agent Lightning](https://github.com/microsoft/agent-lightning) is Microsoft's open-source framework for training and optimizing AI agents with Reinforcement Learning, Automatic Prompt Optimization, and Supervised Fine-tuning — with almost zero code changes.
It works with any agent framework including LangChain, OpenAI Agents SDK, AutoGen, and CrewAI. Agent Lightning uses LiteLLM Proxy under the hood to route LLM requests and collect traces that power its training algorithms.
- [GitHub](https://github.com/microsoft/agent-lightning)
- [Docs](https://microsoft.github.io/agent-lightning/)
- [arXiv Paper](https://arxiv.org/abs/2508.03680)

21
docs/my-website/docs/projects/Google ADK.md Normal file
View file

@ -0,0 +1,21 @@
# Google ADK (Agent Development Kit)
[Google ADK](https://github.com/google/adk-python) is an open-source, code-first Python framework for building, evaluating, and deploying sophisticated AI agents. While optimized for Gemini, ADK is model-agnostic and supports LiteLLM for using 100+ providers.
```python
from google.adk.agents.llm_agent import Agent
from google.adk.models.lite_llm import LiteLlm
root_agent = Agent(
model=LiteLlm(model="openai/gpt-4o"), # Or any LiteLLM-supported model
name="my_agent",
description="An agent using LiteLLM",
instruction="You are a helpful assistant.",
tools=[your_tools],
)
```
- [GitHub](https://github.com/google/adk-python)
- [Documentation](https://google.github.io/adk-docs)
- [LiteLLM Samples](https://github.com/google/adk-python/tree/main/contributing/samples/hello_world_litellm)

24
docs/my-website/docs/projects/Harbor.md Normal file
View file

@ -0,0 +1,24 @@
# Harbor
[Harbor](https://github.com/laude-institute/harbor) is a framework from the creators of Terminal-Bench for evaluating and optimizing agents and language models. It uses LiteLLM to call 100+ LLM providers.
```bash
# Install
pip install harbor
# Run a benchmark with any LiteLLM-supported model
harbor run --dataset terminal-bench@2.0 \
--agent claude-code \
--model anthropic/claude-opus-4-1 \
--n-concurrent 4
```
Key features:
- Evaluate agents like Claude Code, OpenHands, Codex CLI
- Build and share benchmarks and environments
- Run experiments in parallel across cloud providers (Daytona, Modal)
- Generate rollouts for RL optimization
- [GitHub](https://github.com/laude-institute/harbor)
- [Documentation](https://harborframework.com/docs)

14
docs/my-website/docs/providers/bedrock.md
View file

@ -43,6 +43,8 @@ export AWS_BEARER_TOKEN_BEDROCK="your-api-key"
Option 2: use the api_key parameter to pass in API key for completion, embedding, image_generation API calls.
<Tabs>
<TabItem value="sdk" label="SDK">
```python
response = completion(
model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
@ -50,7 +52,17 @@ response = completion(
api_key="your-api-key"
)
```
</TabItem>
<TabItem value="proxy" label="PROXY">
```yaml
model_list:
- model_name: bedrock-claude-3-sonnet
litellm_params:
model: bedrock/anthropic.claude-3-sonnet-20240229-v1:0
api_key: os.environ/AWS_BEARER_TOKEN_BEDROCK
```
</TabItem>
</Tabs>
## Usage

244
docs/my-website/docs/providers/ragflow.md Normal file
View file

@ -0,0 +1,244 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# RAGFlow
Litellm supports Ragflow's chat completions APIs
## Supported Features
- ✅ Chat completions
- ✅ Streaming responses
- ✅ Both chat and agent endpoints
- ✅ Multiple credential sources (params, env vars, litellm_params)
- ✅ OpenAI-compatible API format
## API Key
```python
# env variable
os.environ['RAGFLOW_API_KEY']
```
## API Base
```python
# env variable
os.environ['RAGFLOW_API_BASE']
```
## Overview
RAGFlow provides OpenAI-compatible APIs with unique path structures that include chat and agent IDs:
- **Chat endpoint**: `/api/v1/chats_openai/{chat_id}/chat/completions`
- **Agent endpoint**: `/api/v1/agents_openai/{agent_id}/chat/completions`
The model name format embeds the endpoint type and ID:
- Chat: `ragflow/chat/{chat_id}/{model_name}`
- Agent: `ragflow/agent/{agent_id}/{model_name}`
## Sample Usage - Chat Endpoint
```python
from litellm import completion
import os
os.environ['RAGFLOW_API_KEY'] = "your-ragflow-api-key"
os.environ['RAGFLOW_API_BASE'] = "http://localhost:9380" # or your hosted URL
response = completion(
model="ragflow/chat/my-chat-id/gpt-4o-mini",
messages=[{"role": "user", "content": "How does the deep doc understanding work?"}]
)
print(response)
```
## Sample Usage - Agent Endpoint
```python
from litellm import completion
import os
os.environ['RAGFLOW_API_KEY'] = "your-ragflow-api-key"
os.environ['RAGFLOW_API_BASE'] = "http://localhost:9380" # or your hosted URL
response = completion(
model="ragflow/agent/my-agent-id/gpt-4o-mini",
messages=[{"role": "user", "content": "What are the key features?"}]
)
print(response)
```
## Sample Usage - With Parameters
You can also pass `api_key` and `api_base` directly as parameters:
```python
from litellm import completion
response = completion(
model="ragflow/chat/my-chat-id/gpt-4o-mini",
messages=[{"role": "user", "content": "Hello!"}],
api_key="your-ragflow-api-key",
api_base="http://localhost:9380"
)
print(response)
```
## Sample Usage - Streaming
```python
from litellm import completion
import os
os.environ['RAGFLOW_API_KEY'] = "your-ragflow-api-key"
os.environ['RAGFLOW_API_BASE'] = "http://localhost:9380"
response = completion(
model="ragflow/agent/my-agent-id/gpt-4o-mini",
messages=[{"role": "user", "content": "Explain RAGFlow"}],
stream=True
)
for chunk in response:
print(chunk)
```
## Model Name Format
The model name must follow one of these formats:
### Chat Endpoint
```
ragflow/chat/{chat_id}/{model_name}
```
Example: `ragflow/chat/my-chat-id/gpt-4o-mini`
### Agent Endpoint
```
ragflow/agent/{agent_id}/{model_name}
```
Example: `ragflow/agent/my-agent-id/gpt-4o-mini`
Where:
- `{chat_id}` or `{agent_id}` is the ID of your chat or agent in RAGFlow
- `{model_name}` is the actual model name (e.g., `gpt-4o-mini`, `gpt-4o`, etc.)
## Configuration Sources
LiteLLM supports multiple ways to provide credentials, checked in this order:
1. **Function parameters**: `api_key="..."`, `api_base="..."`
2. **litellm_params**: `litellm_params={"api_key": "...", "api_base": "..."}`
3. **Environment variables**: `RAGFLOW_API_KEY`, `RAGFLOW_API_BASE`
4. **Global litellm settings**: `litellm.api_key`, `litellm.api_base`
## Usage - LiteLLM Proxy Server
### 1. Save key in your environment
```bash
export RAGFLOW_API_KEY="your-ragflow-api-key"
export RAGFLOW_API_BASE="http://localhost:9380"
```
### 2. Start the proxy
<Tabs>
<TabItem value="config" label="config.yaml">
```yaml
model_list:
- model_name: ragflow-chat-gpt4
litellm_params:
model: ragflow/chat/my-chat-id/gpt-4o-mini
api_key: os.environ/RAGFLOW_API_KEY
api_base: os.environ/RAGFLOW_API_BASE
- model_name: ragflow-agent-gpt4
litellm_params:
model: ragflow/agent/my-agent-id/gpt-4o-mini
api_key: os.environ/RAGFLOW_API_KEY
api_base: os.environ/RAGFLOW_API_BASE
```
</TabItem>
<TabItem value="cli" label="CLI">
```bash
$ litellm --config /path/to/config.yaml
# Server running on http://0.0.0.0:4000
```
</TabItem>
</Tabs>
### 3. Test it
<Tabs>
<TabItem value="Curl" label="Curl Request">
```bash
curl http://0.0.0.0:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "ragflow-chat-gpt4",
"messages": [
{"role": "user", "content": "How does RAGFlow work?"}
]
}'
```
</TabItem>
<TabItem value="Python" label="Python SDK">
```python
from openai import OpenAI
client = OpenAI(
api_key="sk-1234", # Your LiteLLM proxy key
base_url="http://0.0.0.0:4000"
)
response = client.chat.completions.create(
model="ragflow-chat-gpt4",
messages=[
{"role": "user", "content": "How does RAGFlow work?"}
]
)
print(response)
```
</TabItem>
</Tabs>
## API Base URL Handling
The `api_base` parameter can be provided with or without `/v1` suffix. LiteLLM will automatically handle it:
- `http://localhost:9380` → `http://localhost:9380/api/v1/chats_openai/{chat_id}/chat/completions`
- `http://localhost:9380/v1` → `http://localhost:9380/api/v1/chats_openai/{chat_id}/chat/completions`
- `http://localhost:9380/api/v1` → `http://localhost:9380/api/v1/chats_openai/{chat_id}/chat/completions`
All three formats will work correctly.
## Error Handling
If you encounter errors:
1. **Invalid model format**: Ensure your model name follows `ragflow/{chat|agent}/{id}/{model_name}` format
2. **Missing api_base**: Provide `api_base` via parameter, environment variable, or litellm_params
3. **Connection errors**: Verify your RAGFlow server is running and accessible at the provided `api_base`
:::info
For more information about passing provider-specific parameters, [go here](../completion/provider_specific_params.md)
:::

349
docs/my-website/docs/providers/ragflow_vector_store.md Normal file
View file

@ -0,0 +1,349 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import Image from '@theme/IdealImage';
# RAGFlow Vector Stores
Litellm support creation and management of datasets for document processing and knowledge base management in Ragflow.
| Property | Details |
|----------|---------|
| Description | RAGFlow datasets enable document processing, chunking, and knowledge base management for RAG applications. |
| Provider Route on LiteLLM | `ragflow` in the litellm vector_store_registry |
| Provider Doc | [RAGFlow API Documentation ↗](https://ragflow.io/docs) |
| Supported Operations | Dataset Management (Create, List, Update, Delete) |
| Search/Retrieval | ❌ Not supported (management only) |
## Quick Start
### LiteLLM Python SDK
```python showLineNumbers title="Example using LiteLLM Python SDK"
import os
import litellm
# Set RAGFlow credentials
os.environ["RAGFLOW_API_KEY"] = "your-ragflow-api-key"
os.environ["RAGFLOW_API_BASE"] = "http://localhost:9380" # Optional, defaults to localhost:9380
# Create a RAGFlow dataset
response = litellm.vector_stores.create(
name="my-dataset",
custom_llm_provider="ragflow",
metadata={
"description": "My knowledge base dataset",
"embedding_model": "BAAI/bge-large-zh-v1.5@BAAI",
"chunk_method": "naive"
}
)
print(f"Created dataset ID: {response.id}")
print(f"Dataset name: {response.name}")
```
### LiteLLM Proxy
#### 1. Configure your vector_store_registry
<Tabs>
<TabItem value="config-yaml" label="config.yaml">
```yaml
model_list:
- model_name: gpt-4o-mini
litellm_params:
model: gpt-4o-mini
api_key: os.environ/OPENAI_API_KEY
vector_store_registry:
- vector_store_name: "ragflow-knowledge-base"
litellm_params:
vector_store_id: "your-dataset-id"
custom_llm_provider: "ragflow"
api_key: os.environ/RAGFLOW_API_KEY
api_base: os.environ/RAGFLOW_API_BASE # Optional
vector_store_description: "RAGFlow dataset for knowledge base"
vector_store_metadata:
source: "Company documentation"
```
</TabItem>
<TabItem value="litellm-ui" label="LiteLLM UI">
On the LiteLLM UI, Navigate to Experimental > Vector Stores > Create Vector Store. On this page you can create a vector store with a name, vector store id and credentials.
<Image
img={require('../../img/kb_2.png')}
style={{width: '50%'}}
/>
</TabItem>
</Tabs>
#### 2. Create a dataset via Proxy
<Tabs>
<TabItem value="curl" label="Curl">
```bash
curl http://localhost:4000/v1/vector_stores \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-d '{
"name": "my-ragflow-dataset",
"custom_llm_provider": "ragflow",
"metadata": {
"description": "Test dataset",
"chunk_method": "naive"
}
}'
```
</TabItem>
<TabItem value="openai-sdk" label="OpenAI Python SDK">
```python
from openai import OpenAI
# Initialize client with your LiteLLM proxy URL
client = OpenAI(
base_url="http://localhost:4000",
api_key="your-litellm-api-key"
)
# Create a RAGFlow dataset
response = client.vector_stores.create(
name="my-ragflow-dataset",
custom_llm_provider="ragflow",
metadata={
"description": "Test dataset",
"chunk_method": "naive"
}
)
print(f"Created dataset: {response.id}")
```
</TabItem>
</Tabs>
## Configuration
### Environment Variables
RAGFlow vector stores support configuration via environment variables:
- `RAGFLOW_API_KEY` - Your RAGFlow API key (required)
- `RAGFLOW_API_BASE` - RAGFlow API base URL (optional, defaults to `http://localhost:9380`)
### Parameters
You can also pass these via `litellm_params`:
- `api_key` - RAGFlow API key (overrides `RAGFLOW_API_KEY` env var)
- `api_base` - RAGFlow API base URL (overrides `RAGFLOW_API_BASE` env var)
## Dataset Creation Options
### Basic Dataset Creation
```python
response = litellm.vector_stores.create(
name="basic-dataset",
custom_llm_provider="ragflow"
)
```
### Dataset with Chunk Method
RAGFlow supports various chunk methods for different document types:
<Tabs>
<TabItem value="naive" label="Naive (General)">
```python
response = litellm.vector_stores.create(
name="general-dataset",
custom_llm_provider="ragflow",
metadata={
"chunk_method": "naive",
"parser_config": {
"chunk_token_num": 512,
"delimiter": "\n",
"html4excel": False,
"layout_recognize": "DeepDOC"
}
}
)
```
</TabItem>
<TabItem value="book" label="Book">
```python
response = litellm.vector_stores.create(
name="book-dataset",
custom_llm_provider="ragflow",
metadata={
"chunk_method": "book",
"parser_config": {
"raptor": {
"use_raptor": False
}
}
}
)
```
</TabItem>
<TabItem value="qa" label="Q&A">
```python
response = litellm.vector_stores.create(
name="qa-dataset",
custom_llm_provider="ragflow",
metadata={
"chunk_method": "qa",
"parser_config": {
"raptor": {
"use_raptor": False
}
}
}
)
```
</TabItem>
<TabItem value="paper" label="Paper">
```python
response = litellm.vector_stores.create(
name="paper-dataset",
custom_llm_provider="ragflow",
metadata={
"chunk_method": "paper",
"parser_config": {
"raptor": {
"use_raptor": False
}
}
}
)
```
</TabItem>
</Tabs>
### Dataset with Ingestion Pipeline
Instead of using a chunk method, you can use an ingestion pipeline:
```python
response = litellm.vector_stores.create(
name="pipeline-dataset",
custom_llm_provider="ragflow",
metadata={
"parse_type": 2, # Number of parsers in your pipeline
"pipeline_id": "d0bebe30ae2211f0970942010a8e0005" # 32-character hex ID
}
)
```
**Note**: `chunk_method` and `pipeline_id` are mutually exclusive. Use one or the other.
### Advanced Parser Configuration
```python
response = litellm.vector_stores.create(
name="advanced-dataset",
custom_llm_provider="ragflow",
metadata={
"chunk_method": "naive",
"description": "Advanced dataset with custom parser config",
"embedding_model": "BAAI/bge-large-zh-v1.5@BAAI",
"permission": "me", # or "team"
"parser_config": {
"chunk_token_num": 1024,
"delimiter": "\n!?;。；！？",
"html4excel": True,
"layout_recognize": "DeepDOC",
"auto_keywords": 5,
"auto_questions": 3,
"task_page_size": 12,
"raptor": {
"use_raptor": True
},
"graphrag": {
"use_graphrag": False
}
}
}
)
```
## Supported Chunk Methods
RAGFlow supports the following chunk methods:
- `naive` - General purpose (default)
- `book` - For book documents
- `email` - For email documents
- `laws` - For legal documents
- `manual` - Manual chunking
- `one` - Single chunk
- `paper` - For academic papers
- `picture` - For image documents
- `presentation` - For presentation documents
- `qa` - Q&A format
- `table` - For table documents
- `tag` - Tag-based chunking
## RAGFlow-Specific Parameters
All RAGFlow-specific parameters should be passed via the `metadata` field:
| Parameter | Type | Description |
|-----------|------|-------------|
| `avatar` | string | Base64 encoding of the avatar (max 65535 chars) |
| `description` | string | Brief description of the dataset (max 65535 chars) |
| `embedding_model` | string | Embedding model name (e.g., "BAAI/bge-large-zh-v1.5@BAAI") |
| `permission` | string | Access permission: "me" (default) or "team" |
| `chunk_method` | string | Chunking method (see supported methods above) |
| `parser_config` | object | Parser configuration (varies by chunk_method) |
| `parse_type` | int | Number of parsers in pipeline (required with pipeline_id) |
| `pipeline_id` | string | 32-character hex pipeline ID (required with parse_type) |
## Error Handling
RAGFlow returns error responses in the following format:
```json
{
"code": 101,
"message": "Dataset name 'my-dataset' already exists"
}
```
LiteLLM automatically maps these to appropriate exceptions:
- `code != 0` → Raises exception with the error message
- Missing required fields → Raises `ValueError`
- Mutually exclusive parameters → Raises `ValueError`
## Limitations
- **Search/Retrieval**: RAGFlow vector stores support dataset management only. Search operations are not supported and will raise `NotImplementedError`.
- **List/Update/Delete**: These operations are not yet implemented through the standard vector store API. Use RAGFlow's native API endpoints directly.
## Further Reading
Vector Stores:
- [Vector Store Creation](../vector_stores/create.md)
- [Using Vector Stores with Completions](../completion/knowledgebase.md)
- [Vector Store Registry](../completion/knowledgebase.md#vectorstoreregistry)

349
docs/my-website/docs/providers/vertex.md
View file

@ -2550,355 +2550,6 @@ print(response)
</TabItem>
</Tabs>
## **Gemini TTS (Text-to-Speech) Audio Output**
:::info
LiteLLM supports Gemini TTS models on Vertex AI that can generate audio responses using the OpenAI-compatible `audio` parameter format.
:::
### Supported Models
LiteLLM supports Gemini TTS models with audio capabilities on Vertex AI (e.g. `vertex_ai/gemini-2.5-flash-preview-tts` and `vertex_ai/gemini-2.5-pro-preview-tts`). For the complete list of available TTS models and voices, see the [official Gemini TTS documentation](https://ai.google.dev/gemini-api/docs/speech-generation).
### Limitations
:::warning
**Important Limitations**:
- Gemini TTS models only support the `pcm16` audio format
- **Streaming support has not been added** to TTS models yet
- The `modalities` parameter must be set to `['audio']` for TTS requests
:::
### Quick Start
<Tabs>
<TabItem value="sdk" label="SDK">
```python
from litellm import completion
import json
## GET CREDENTIALS
file_path = 'path/to/vertex_ai_service_account.json'
# Load the JSON file
with open(file_path, 'r') as file:
vertex_credentials = json.load(file)
# Convert to JSON string
vertex_credentials_json = json.dumps(vertex_credentials)
response = completion(
model="vertex_ai/gemini-2.5-flash-preview-tts",
messages=[{"role": "user", "content": "Say hello in a friendly voice"}],
modalities=["audio"], # Required for TTS models
audio={
"voice": "Kore",
"format": "pcm16" # Required: must be "pcm16"
},
vertex_credentials=vertex_credentials_json
)
print(response)
```
</TabItem>
<TabItem value="proxy" label="PROXY">
1. Setup config.yaml
```yaml
model_list:
- model_name: gemini-tts-flash
litellm_params:
model: vertex_ai/gemini-2.5-flash-preview-tts
vertex_project: "your-project-id"
vertex_location: "us-central1"
vertex_credentials: "/path/to/service_account.json"
- model_name: gemini-tts-pro
litellm_params:
model: vertex_ai/gemini-2.5-pro-preview-tts
vertex_project: "your-project-id"
vertex_location: "us-central1"
vertex_credentials: "/path/to/service_account.json"
```
2. Start proxy
```bash
litellm --config /path/to/config.yaml
```
3. Make TTS request
```bash
curl http://0.0.0.0:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <YOUR-LITELLM-KEY>" \
-d '{
"model": "gemini-tts-flash",
"messages": [{"role": "user", "content": "Say hello in a friendly voice"}],
"modalities": ["audio"],
"audio": {
"voice": "Kore",
"format": "pcm16"
}
}'
```
</TabItem>
</Tabs>
### Advanced Usage
You can combine TTS with other Gemini features:
```python
response = completion(
model="vertex_ai/gemini-2.5-pro-preview-tts",
messages=[
{"role": "system", "content": "You are a helpful assistant that speaks clearly."},
{"role": "user", "content": "Explain quantum computing in simple terms"}
],
modalities=["audio"],
audio={
"voice": "Charon",
"format": "pcm16"
},
temperature=0.7,
max_tokens=150,
vertex_credentials=vertex_credentials_json
)
```
For more information about Gemini's TTS capabilities and available voices, see the [official Gemini TTS documentation](https://ai.google.dev/gemini-api/docs/speech-generation).
## **Text to Speech APIs**
:::info
LiteLLM supports calling [Vertex AI Text to Speech API](https://console.cloud.google.com/vertex-ai/generative/speech/text-to-speech) in the OpenAI text to speech API format
:::
### Usage - Basic
<Tabs>
<TabItem value="sdk" label="SDK">
Vertex AI does not support passing a `model` param - so passing `model=vertex_ai/` is the only required param
**Sync Usage**
```python
speech_file_path = Path(__file__).parent / "speech_vertex.mp3"
response = litellm.speech(
model="vertex_ai/",
input="hello what llm guardrail do you have",
)
response.stream_to_file(speech_file_path)
```
**Async Usage**
```python
speech_file_path = Path(__file__).parent / "speech_vertex.mp3"
response = litellm.aspeech(
model="vertex_ai/",
input="hello what llm guardrail do you have",
)
response.stream_to_file(speech_file_path)
```
</TabItem>
<TabItem value="proxy" label="LiteLLM PROXY (Unified Endpoint)">
1. Add model to config.yaml
```yaml
model_list:
- model_name: vertex-tts
litellm_params:
model: vertex_ai/ # Vertex AI does not support passing a `model` param - so passing `model=vertex_ai/` is the only required param
vertex_project: "adroit-crow-413218"
vertex_location: "us-central1"
vertex_credentials: adroit-crow-413218-a956eef1a2a8.json
litellm_settings:
drop_params: True
```
2. Start Proxy
```
$ litellm --config /path/to/config.yaml
```
3. Make Request use OpenAI Python SDK
```python
import openai
client = openai.OpenAI(api_key="sk-1234", base_url="http://0.0.0.0:4000")
# see supported values for "voice" on vertex here:
# https://console.cloud.google.com/vertex-ai/generative/speech/text-to-speech
response = client.audio.speech.create(
model = "vertex-tts",
input="the quick brown fox jumped over the lazy dogs",
voice={'languageCode': 'en-US', 'name': 'en-US-Studio-O'}
)
print("response from proxy", response)
```
</TabItem>
</Tabs>
### Usage - `ssml` as input
Pass your `ssml` as input to the `input` param, if it contains `<speak>`, it will be automatically detected and passed as `ssml` to the Vertex AI API
If you need to force your `input` to be passed as `ssml`, set `use_ssml=True`
<Tabs>
<TabItem value="sdk" label="SDK">
Vertex AI does not support passing a `model` param - so passing `model=vertex_ai/` is the only required param
```python
speech_file_path = Path(__file__).parent / "speech_vertex.mp3"
ssml = """
<speak>
<p>Hello, world!</p>
<p>This is a test of the <break strength="medium" /> text-to-speech API.</p>
</speak>
"""
response = litellm.speech(
input=ssml,
model="vertex_ai/test",
voice={
"languageCode": "en-UK",
"name": "en-UK-Studio-O",
},
audioConfig={
"audioEncoding": "LINEAR22",
"speakingRate": "10",
},
)
response.stream_to_file(speech_file_path)
```
</TabItem>
<TabItem value="proxy" label="LiteLLM PROXY (Unified Endpoint)">
```python
import openai
client = openai.OpenAI(api_key="sk-1234", base_url="http://0.0.0.0:4000")
ssml = """
<speak>
<p>Hello, world!</p>
<p>This is a test of the <break strength="medium" /> text-to-speech API.</p>
</speak>
"""
# see supported values for "voice" on vertex here:
# https://console.cloud.google.com/vertex-ai/generative/speech/text-to-speech
response = client.audio.speech.create(
model = "vertex-tts",
input=ssml,
voice={'languageCode': 'en-US', 'name': 'en-US-Studio-O'},
)
print("response from proxy", response)
```
</TabItem>
</Tabs>
### Forcing SSML Usage
You can force the use of SSML by setting the `use_ssml` parameter to `True`. This is useful when you want to ensure that your input is treated as SSML, even if it doesn't contain the `<speak>` tags.
Here are examples of how to force SSML usage:
<Tabs>
<TabItem value="sdk" label="SDK">
Vertex AI does not support passing a `model` param - so passing `model=vertex_ai/` is the only required param
```python
speech_file_path = Path(__file__).parent / "speech_vertex.mp3"
ssml = """
<speak>
<p>Hello, world!</p>
<p>This is a test of the <break strength="medium" /> text-to-speech API.</p>
</speak>
"""
response = litellm.speech(
input=ssml,
use_ssml=True,
model="vertex_ai/test",
voice={
"languageCode": "en-UK",
"name": "en-UK-Studio-O",
},
audioConfig={
"audioEncoding": "LINEAR22",
"speakingRate": "10",
},
)
response.stream_to_file(speech_file_path)
```
</TabItem>
<TabItem value="proxy" label="LiteLLM PROXY (Unified Endpoint)">
```python
import openai
client = openai.OpenAI(api_key="sk-1234", base_url="http://0.0.0.0:4000")
ssml = """
<speak>
<p>Hello, world!</p>
<p>This is a test of the <break strength="medium" /> text-to-speech API.</p>
</speak>
"""
# see supported values for "voice" on vertex here:
# https://console.cloud.google.com/vertex-ai/generative/speech/text-to-speech
response = client.audio.speech.create(
model = "vertex-tts",
input=ssml, # pass as None since OpenAI SDK requires this param
voice={'languageCode': 'en-US', 'name': 'en-US-Studio-O'},
extra_body={"use_ssml": True},
)
print("response from proxy", response)
```
</TabItem>
</Tabs>
## **Fine Tuning APIs**

423
docs/my-website/docs/providers/vertex_speech.md Normal file
View file

@ -0,0 +1,423 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Vertex AI Text to Speech
| Property | Details |
|-------|-------|
| Description | Google Cloud Text-to-Speech with Chirp3 HD voices and Gemini TTS |
| Provider Route on LiteLLM | `vertex_ai/chirp` (Chirp), `vertex_ai/gemini-*-tts` (Gemini) |
## Chirp3 HD Voices
Google Cloud Text-to-Speech API with high-quality Chirp3 HD voices.
### Quick Start
#### LiteLLM Python SDK
```python showLineNumbers title="Chirp3 Quick Start"
from litellm import speech
from pathlib import Path
speech_file_path = Path(__file__).parent / "speech.mp3"
response = speech(
model="vertex_ai/chirp",
voice="alloy", # OpenAI voice name - automatically mapped
input="Hello, this is Vertex AI Text to Speech",
vertex_project="your-project-id",
vertex_location="us-central1",
)
response.stream_to_file(speech_file_path)
```
#### LiteLLM AI Gateway
**1. Setup config.yaml**
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: vertex-tts
litellm_params:
model: vertex_ai/chirp
vertex_project: "your-project-id"
vertex_location: "us-central1"
vertex_credentials: "/path/to/service_account.json"
```
**2. Start the proxy**
```bash title="Start LiteLLM Proxy"
litellm --config /path/to/config.yaml
```
**3. Make requests**
<Tabs>
<TabItem value="curl" label="curl">
```bash showLineNumbers title="Chirp3 Quick Start"
curl http://0.0.0.0:4000/v1/audio/speech \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
"model": "vertex-tts",
"voice": "alloy",
"input": "Hello, this is Vertex AI Text to Speech"
}' \
--output speech.mp3
```
</TabItem>
<TabItem value="openai-sdk" label="OpenAI Python SDK">
```python showLineNumbers title="Chirp3 Quick Start"
import openai
client = openai.OpenAI(api_key="sk-1234", base_url="http://0.0.0.0:4000")
response = client.audio.speech.create(
model="vertex-tts",
voice="alloy",
input="Hello, this is Vertex AI Text to Speech",
)
response.stream_to_file("speech.mp3")
```
</TabItem>
</Tabs>
### Voice Mapping
LiteLLM maps OpenAI voice names to Google Cloud voices. You can use either OpenAI voices or Google Cloud voices directly.
| OpenAI Voice | Google Cloud Voice |
|-------------|-------------------|
| `alloy` | en-US-Studio-O |
| `echo` | en-US-Studio-M |
| `fable` | en-GB-Studio-B |
| `onyx` | en-US-Wavenet-D |
| `nova` | en-US-Studio-O |
| `shimmer` | en-US-Wavenet-F |
### Using Google Cloud Voices Directly
#### LiteLLM Python SDK
```python showLineNumbers title="Chirp3 HD Voice"
from litellm import speech
# Pass Chirp3 HD voice name directly
response = speech(
model="vertex_ai/chirp",
voice="en-US-Chirp3-HD-Charon",
input="Hello with a Chirp3 HD voice",
vertex_project="your-project-id",
)
response.stream_to_file("speech.mp3")
```
```python showLineNumbers title="Voice as Dict (Multilingual)"
from litellm import speech
# Pass as dict for full control over language and voice
response = speech(
model="vertex_ai/chirp",
voice={
"languageCode": "de-DE",
"name": "de-DE-Chirp3-HD-Charon",
},
input="Hallo, dies ist ein Test",
vertex_project="your-project-id",
)
response.stream_to_file("speech.mp3")
```
#### LiteLLM AI Gateway
<Tabs>
<TabItem value="curl" label="curl">
```bash showLineNumbers title="Chirp3 HD Voice"
curl http://0.0.0.0:4000/v1/audio/speech \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
"model": "vertex-tts",
"voice": "en-US-Chirp3-HD-Charon",
"input": "Hello with a Chirp3 HD voice"
}' \
--output speech.mp3
```
```bash showLineNumbers title="Voice as Dict (Multilingual)"
curl http://0.0.0.0:4000/v1/audio/speech \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
"model": "vertex-tts",
"voice": {"languageCode": "de-DE", "name": "de-DE-Chirp3-HD-Charon"},
"input": "Hallo, dies ist ein Test"
}' \
--output speech.mp3
```
</TabItem>
<TabItem value="openai-sdk" label="OpenAI Python SDK">
```python showLineNumbers title="Chirp3 HD Voice"
import openai
client = openai.OpenAI(api_key="sk-1234", base_url="http://0.0.0.0:4000")
response = client.audio.speech.create(
model="vertex-tts",
voice="en-US-Chirp3-HD-Charon",
input="Hello with a Chirp3 HD voice",
)
response.stream_to_file("speech.mp3")
```
```python showLineNumbers title="Voice as Dict (Multilingual)"
import openai
client = openai.OpenAI(api_key="sk-1234", base_url="http://0.0.0.0:4000")
response = client.audio.speech.create(
model="vertex-tts",
voice={"languageCode": "de-DE", "name": "de-DE-Chirp3-HD-Charon"},
input="Hallo, dies ist ein Test",
)
response.stream_to_file("speech.mp3")
```
</TabItem>
</Tabs>
Browse available voices: [Google Cloud Text-to-Speech Console](https://console.cloud.google.com/vertex-ai/generative/speech/text-to-speech)
### Passing Raw SSML
LiteLLM auto-detects SSML when your input contains `<speak>` tags and passes it through unchanged.
#### LiteLLM Python SDK
```python showLineNumbers title="SSML Input"
from litellm import speech
ssml = """
<speak>
<p>Hello, world!</p>
<p>This is a test of the <break strength="medium" /> text-to-speech API.</p>
</speak>
"""
response = speech(
model="vertex_ai/chirp",
voice="en-US-Studio-O",
input=ssml, # Auto-detected as SSML
vertex_project="your-project-id",
)
response.stream_to_file("speech.mp3")
```
```python showLineNumbers title="Force SSML Mode"
from litellm import speech
# Force SSML mode with use_ssml=True
response = speech(
model="vertex_ai/chirp",
voice="en-US-Studio-O",
input="<speak><prosody rate='slow'>Speaking slowly</prosody></speak>",
use_ssml=True,
vertex_project="your-project-id",
)
response.stream_to_file("speech.mp3")
```
#### LiteLLM AI Gateway
<Tabs>
<TabItem value="curl" label="curl">
```bash showLineNumbers title="SSML Input"
curl http://0.0.0.0:4000/v1/audio/speech \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
"model": "vertex-tts",
"voice": "en-US-Studio-O",
"input": "<speak><p>Hello!</p><break time=\"500ms\"/><p>How are you?</p></speak>"
}' \
--output speech.mp3
```
</TabItem>
<TabItem value="openai-sdk" label="OpenAI Python SDK">
```python showLineNumbers title="SSML Input"
import openai
client = openai.OpenAI(api_key="sk-1234", base_url="http://0.0.0.0:4000")
ssml = """<speak><p>Hello!</p><break time="500ms"/><p>How are you?</p></speak>"""
response = client.audio.speech.create(
model="vertex-tts",
voice="en-US-Studio-O",
input=ssml,
)
response.stream_to_file("speech.mp3")
```
</TabItem>
</Tabs>
### Supported Parameters
| Parameter | Description | Values |
|-----------|-------------|--------|
| `voice` | Voice selection | OpenAI voice, Google Cloud voice name, or dict |
| `input` | Text to convert | Plain text or SSML |
| `speed` | Speaking rate | 0.25 to 4.0 (default: 1.0) |
| `response_format` | Audio format | `mp3`, `opus`, `wav`, `pcm`, `flac` |
| `use_ssml` | Force SSML mode | `True` / `False` |
### Async Usage
```python showLineNumbers title="Async Speech Generation"
import asyncio
from litellm import aspeech
async def main():
response = await aspeech(
model="vertex_ai/chirp",
voice="alloy",
input="Hello from async",
vertex_project="your-project-id",
)
response.stream_to_file("speech.mp3")
asyncio.run(main())
```
---
## Gemini TTS
Gemini models with audio output capabilities using the chat completions API.
:::warning
**Limitations:**
- Only supports `pcm16` audio format
- Streaming not yet supported
- Must set `modalities: ["audio"]`
:::
### Quick Start
#### LiteLLM Python SDK
```python showLineNumbers title="Gemini TTS Quick Start"
from litellm import completion
import json
# Load credentials
with open('path/to/service_account.json', 'r') as file:
vertex_credentials = json.dumps(json.load(file))
response = completion(
model="vertex_ai/gemini-2.5-flash-preview-tts",
messages=[{"role": "user", "content": "Say hello in a friendly voice"}],
modalities=["audio"],
audio={
"voice": "Kore",
"format": "pcm16"
},
vertex_credentials=vertex_credentials
)
print(response)
```
#### LiteLLM AI Gateway
**1. Setup config.yaml**
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: gemini-tts
litellm_params:
model: vertex_ai/gemini-2.5-flash-preview-tts
vertex_project: "your-project-id"
vertex_location: "us-central1"
vertex_credentials: "/path/to/service_account.json"
```
**2. Start the proxy**
```bash title="Start LiteLLM Proxy"
litellm --config /path/to/config.yaml
```
**3. Make requests**
<Tabs>
<TabItem value="curl" label="curl">
```bash showLineNumbers title="Gemini TTS Request"
curl http://0.0.0.0:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "gemini-tts",
"messages": [{"role": "user", "content": "Say hello in a friendly voice"}],
"modalities": ["audio"],
"audio": {"voice": "Kore", "format": "pcm16"}
}'
```
</TabItem>
<TabItem value="openai-sdk" label="OpenAI Python SDK">
```python showLineNumbers title="Gemini TTS Request"
import openai
client = openai.OpenAI(api_key="sk-1234", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
model="gemini-tts",
messages=[{"role": "user", "content": "Say hello in a friendly voice"}],
modalities=["audio"],
audio={"voice": "Kore", "format": "pcm16"},
)
print(response)
```
</TabItem>
</Tabs>
### Supported Models
- `vertex_ai/gemini-2.5-flash-preview-tts`
- `vertex_ai/gemini-2.5-pro-preview-tts`
See [Gemini TTS documentation](https://ai.google.dev/gemini-api/docs/speech-generation) for available voices.
### Advanced Usage
```python showLineNumbers title="Gemini TTS with System Prompt"
from litellm import completion
response = completion(
model="vertex_ai/gemini-2.5-pro-preview-tts",
messages=[
{"role": "system", "content": "You are a helpful assistant that speaks clearly."},
{"role": "user", "content": "Explain quantum computing in simple terms"}
],
modalities=["audio"],
audio={"voice": "Charon", "format": "pcm16"},
temperature=0.7,
max_tokens=150,
vertex_credentials=vertex_credentials
)
```

90
docs/my-website/docs/proxy/error_diagnosis.md Normal file
View file

@ -0,0 +1,90 @@
# Diagnosing Errors - Provider vs Gateway
Having trouble diagnosing if an error is from the **LLM Provider** (OpenAI, Anthropic, etc.) or from the **LiteLLM AI Gateway** itself? Here's how to tell.
## Quick Rule
**If the error contains `<Provider>Exception`, it's from the provider.**
| Error Contains | Error Source |
|----------------|--------------|
| `AnthropicException` | Anthropic |
| `OpenAIException` | OpenAI |
| `AzureException` | Azure |
| `BedrockException` | AWS Bedrock |
| `VertexAIException` | Google Vertex AI |
| No provider name | LiteLLM AI Gateway |
## Examples
### Provider Error (from AWS Bedrock)
```
{
"error": {
"message": "litellm.BadRequestError: BedrockException - {\"message\":\"The model returned the following errors: messages.1.content.0.type: Expected `thinking` or `redacted_thinking`, but found `text`.\"}",
"type": "invalid_request_error",
"param": null,
"code": "400"
}
}
```
This error is from **AWS Bedrock** (notice `BedrockException`). The Bedrock API is rejecting the request due to invalid message format - this is not a LiteLLM issue.
### Provider Error (from OpenAI)
```
{
"error": {
"message": "litellm.AuthenticationError: OpenAIException - Incorrect API key provided: <my-key>. You can find your API key at https://platform.openai.com/account/api-keys.",
"type": "invalid_request_error",
"param": null,
"code": "invalid_api_key"
}
}
```
This error is from **OpenAI** (notice `OpenAIException`). The OpenAI API key configured in LiteLLM is invalid.
### Provider Error (from Anthropic)
```
{
"error": {
"message": "litellm.InternalServerError: AnthropicException - Overloaded. Handle with `litellm.InternalServerError`.",
"type": "internal_server_error",
"param": null,
"code": "500"
}
}
```
This error is from **Anthropic** (notice `AnthropicException`). The Anthropic API is overloaded - this is not a LiteLLM issue.
### Gateway Error (from LiteLLM)
```
{
"error": {
"message": "Invalid API Key. Please check your LiteLLM API key.",
"type": "auth_error",
"param": null,
"code": "401"
}
}
```
This error is from the **LiteLLM AI Gateway** (no provider name). Your LiteLLM virtual key is invalid.
## What to do?
| Error Source | Action |
|--------------|--------|
| Provider Error | Check the provider's status page, adjust rate limits, or retry later |
| Gateway Error | Check your LiteLLM configuration, API keys, or [open an issue](https://github.com/BerriAI/litellm/issues) |
## See Also
- [Debugging](/docs/proxy/debugging) - Enable debug logs to see detailed request/response info
- [Exception Mapping](/docs/exception_mapping) - Full list of LiteLLM exception types

1
docs/my-website/docs/vector_stores/create.md
View file

@ -14,6 +14,7 @@ Create a vector store which can be used to store and search document chunks for
| End-user Tracking | ✅ | |
| Support LLM Providers (OpenAI `/vector_stores` API) | **OpenAI** | Full vector stores API support across providers |
| Support LLM Providers (Passthrough API) | [**Azure AI**](/docs/providers/azure_ai/azure_ai_vector_stores_passthrough) | Full vector stores API support across providers |
| Support LLM Providers (Dataset Management) | [**RAGFlow**](/docs/providers/ragflow_vector_store.md) | Dataset creation and management (search not supported) |
## Usage

8
docs/my-website/sidebars.js
View file

@ -141,6 +141,7 @@ const sidebars = {
"proxy/quick_start",
"proxy/cli",
"proxy/debugging",
"proxy/error_diagnosis",
"proxy/deploy",
"proxy/health",
"proxy/master_key_rotations",
@ -520,6 +521,7 @@ const sidebars = {
"providers/vertex_partner",
"providers/vertex_self_deployed",
"providers/vertex_image",
"providers/vertex_speech",
"providers/vertex_batch",
"providers/vertex_ocr",
]
@ -625,6 +627,7 @@ const sidebars = {
"providers/petals",
"providers/publicai",
"providers/predibase",
"providers/ragflow",
"providers/recraft",
"providers/replicate",
{
@ -818,10 +821,13 @@ const sidebars = {
"Learn how to deploy + call models from different providers on LiteLLM",
slug: "/project",
},
items: [
items: [
"projects/smolagents",
"projects/mini-swe-agent",
"projects/openai-agents",
"projects/Google ADK",
"projects/Agent Lightning",
"projects/Harbor",
"projects/Docq.AI",
"projects/PDL",
"projects/OpenInterpreter",

47
docs/my-website/src/pages/intro.md
View file

@ -1,47 +0,0 @@
---
sidebar_position: 1
---
# Tutorial Intro
Let's discover **Docusaurus in less than 5 minutes**.
## Getting Started
Get started by **creating a new site**.
Or **try Docusaurus immediately** with **[docusaurus.new](https://docusaurus.new)**.
### What you'll need
- [Node.js](https://nodejs.org/en/download/) version 16.14 or above:
- When installing Node.js, you are recommended to check all checkboxes related to dependencies.
## Generate a new site
Generate a new Docusaurus site using the **classic template**.
The classic template will automatically be added to your project after you run the command:
```bash
npm init docusaurus@latest my-website classic
```
You can type this command into Command Prompt, Powershell, Terminal, or any other integrated terminal of your code editor.
The command also installs all necessary dependencies you need to run Docusaurus.
## Start your site
Run the development server:
```bash
cd my-website
npm run start
```
The `cd` command changes the directory you're working with. In order to work with your newly created Docusaurus site, you'll need to navigate the terminal there.
The `npm run start` command builds your website locally and serves it through a development server, ready for you to view at http://localhost:3000/.
Open `docs/intro.md` (this page) and edit some lines: the site **reloads automatically** and displays your changes.

8
docs/my-website/src/pages/tutorial-basics/_category_.json
View file

@ -1,8 +0,0 @@
{
"label": "Tutorial - Basics",
"position": 2,
"link": {
"type": "generated-index",
"description": "5 minutes to learn the most important Docusaurus concepts."
}
}

23
docs/my-website/src/pages/tutorial-basics/congratulations.md
View file

@ -1,23 +0,0 @@
---
sidebar_position: 6
---
# Congratulations!
You have just learned the **basics of Docusaurus** and made some changes to the **initial template**.
Docusaurus has **much more to offer**!
Have **5 more minutes**? Take a look at **[versioning](../tutorial-extras/manage-docs-versions.md)** and **[i18n](../tutorial-extras/translate-your-site.md)**.
Anything **unclear** or **buggy** in this tutorial? [Please report it!](https://github.com/facebook/docusaurus/discussions/4610)
## What's next?
- Read the [official documentation](https://docusaurus.io/)
- Modify your site configuration with [`docusaurus.config.js`](https://docusaurus.io/docs/api/docusaurus-config)
- Add navbar and footer items with [`themeConfig`](https://docusaurus.io/docs/api/themes/configuration)
- Add a custom [Design and Layout](https://docusaurus.io/docs/styling-layout)
- Add a [search bar](https://docusaurus.io/docs/search)
- Find inspirations in the [Docusaurus showcase](https://docusaurus.io/showcase)
- Get involved in the [Docusaurus Community](https://docusaurus.io/community/support)

34
docs/my-website/src/pages/tutorial-basics/create-a-blog-post.md
View file

@ -1,34 +0,0 @@
---
sidebar_position: 3
---
# Create a Blog Post
Docusaurus creates a **page for each blog post**, but also a **blog index page**, a **tag system**, an **RSS** feed...
## Create your first Post
Create a file at `blog/2021-02-28-greetings.md`:
```md title="blog/2021-02-28-greetings.md"
---
slug: greetings
title: Greetings!
authors:
- name: Joel Marcey
title: Co-creator of Docusaurus 1
url: https://github.com/JoelMarcey
image_url: https://github.com/JoelMarcey.png
- name: Sébastien Lorber
title: Docusaurus maintainer
url: https://sebastienlorber.com
image_url: https://github.com/slorber.png
tags: [greetings]
---
Congratulations, you have made your first post!
Feel free to play around and edit this post as much you like.
```
A new blog post is now available at [http://localhost:3000/blog/greetings](http://localhost:3000/blog/greetings).

57
docs/my-website/src/pages/tutorial-basics/create-a-document.md
View file

@ -1,57 +0,0 @@
---
sidebar_position: 2
---
# Create a Document
Documents are **groups of pages** connected through:
- a **sidebar**
- **previous/next navigation**
- **versioning**
## Create your first Doc
Create a Markdown file at `docs/hello.md`:
```md title="docs/hello.md"
# Hello
This is my **first Docusaurus document**!
```
A new document is now available at [http://localhost:3000/docs/hello](http://localhost:3000/docs/hello).
## Configure the Sidebar
Docusaurus automatically **creates a sidebar** from the `docs` folder.
Add metadata to customize the sidebar label and position:
```md title="docs/hello.md" {1-4}
---
sidebar_label: 'Hi!'
sidebar_position: 3
---
# Hello
This is my **first Docusaurus document**!
```
It is also possible to create your sidebar explicitly in `sidebars.js`:
```js title="sidebars.js"
module.exports = {
tutorialSidebar: [
'intro',
// highlight-next-line
'hello',
{
type: 'category',
label: 'Tutorial',
items: ['tutorial-basics/create-a-document'],
},
],
};
```

43
docs/my-website/src/pages/tutorial-basics/create-a-page.md
View file

@ -1,43 +0,0 @@
---
sidebar_position: 1
---
# Create a Page
Add **Markdown or React** files to `src/pages` to create a **standalone page**:
- `src/pages/index.js` → `localhost:3000/`
- `src/pages/foo.md` → `localhost:3000/foo`
- `src/pages/foo/bar.js` → `localhost:3000/foo/bar`
## Create your first React Page
Create a file at `src/pages/my-react-page.js`:
```jsx title="src/pages/my-react-page.js"
import React from 'react';
import Layout from '@theme/Layout';
export default function MyReactPage() {
return (
<Layout>
<h1>My React page</h1>
<p>This is a React page</p>
</Layout>
);
}
```
A new page is now available at [http://localhost:3000/my-react-page](http://localhost:3000/my-react-page).
## Create your first Markdown Page
Create a file at `src/pages/my-markdown-page.md`:
```mdx title="src/pages/my-markdown-page.md"
# My Markdown page
This is a Markdown page
```
A new page is now available at [http://localhost:3000/my-markdown-page](http://localhost:3000/my-markdown-page).

31
docs/my-website/src/pages/tutorial-basics/deploy-your-site.md
View file

@ -1,31 +0,0 @@
---
sidebar_position: 5
---
# Deploy your site
Docusaurus is a **static-site-generator** (also called **[Jamstack](https://jamstack.org/)**).
It builds your site as simple **static HTML, JavaScript and CSS files**.
## Build your site
Build your site **for production**:
```bash
npm run build
```
The static files are generated in the `build` folder.
## Deploy your site
Test your production build locally:
```bash
npm run serve
```
The `build` folder is now served at [http://localhost:3000/](http://localhost:3000/).
You can now deploy the `build` folder **almost anywhere** easily, **for free** or very small cost (read the **[Deployment Guide](https://docusaurus.io/docs/deployment)**).

150
docs/my-website/src/pages/tutorial-basics/markdown-features.mdx
View file

@ -1,150 +0,0 @@
---
sidebar_position: 4
---
# Markdown Features
Docusaurus supports **[Markdown](https://daringfireball.net/projects/markdown/syntax)** and a few **additional features**.
## Front Matter
Markdown documents have metadata at the top called [Front Matter](https://jekyllrb.com/docs/front-matter/):
```text title="my-doc.md"
// highlight-start
---
id: my-doc-id
title: My document title
description: My document description
slug: /my-custom-url
---
// highlight-end
## Markdown heading
Markdown text with [links](./hello.md)
```
## Links
Regular Markdown links are supported, using url paths or relative file paths.
```md
Let's see how to [Create a page](/create-a-page).
```
```md
Let's see how to [Create a page](./create-a-page.md).
```
**Result:** Let's see how to [Create a page](./create-a-page.md).
## Images
Regular Markdown images are supported.
You can use absolute paths to reference images in the static directory (`static/img/docusaurus.png`):
```md
![Docusaurus logo](/img/docusaurus.png)
```
![Docusaurus logo](/img/docusaurus.png)
You can reference images relative to the current file as well. This is particularly useful to colocate images close to the Markdown files using them:
```md
![Docusaurus logo](./img/docusaurus.png)
```
## Code Blocks
Markdown code blocks are supported with Syntax highlighting.
```jsx title="src/components/HelloDocusaurus.js"
function HelloDocusaurus() {
return (
<h1>Hello, Docusaurus!</h1>
)
}
```
```jsx title="src/components/HelloDocusaurus.js"
function HelloDocusaurus() {
return <h1>Hello, Docusaurus!</h1>;
}
```
## Admonitions
Docusaurus has a special syntax to create admonitions and callouts:
:::tip My tip
Use this awesome feature option
:::
:::danger Take care
This action is dangerous
:::
:::tip My tip
Use this awesome feature option
:::
:::danger Take care
This action is dangerous
:::
## MDX and React Components
[MDX](https://mdxjs.com/) can make your documentation more **interactive** and allows using any **React components inside Markdown**:
```jsx
export const Highlight = ({children, color}) => (
<span
style={{
backgroundColor: color,
borderRadius: '20px',
color: '#fff',
padding: '10px',
cursor: 'pointer',
}}
onClick={() => {
alert(`You clicked the color ${color} with label ${children}`)
}}>
{children}
</span>
);
This is <Highlight color="#25c2a0">Docusaurus green</Highlight> !
This is <Highlight color="#1877F2">Facebook blue</Highlight> !
```
export const Highlight = ({children, color}) => (
<span
style={{
backgroundColor: color,
borderRadius: '20px',
color: '#fff',
padding: '10px',
cursor: 'pointer',
}}
onClick={() => {
alert(`You clicked the color ${color} with label ${children}`);
}}>
{children}
</span>
);
This is <Highlight color="#25c2a0">Docusaurus green</Highlight> !
This is <Highlight color="#1877F2">Facebook blue</Highlight> !

7
docs/my-website/src/pages/tutorial-extras/_category_.json
View file

@ -1,7 +0,0 @@
{
"label": "Tutorial - Extras",
"position": 3,
"link": {
"type": "generated-index"
}
}

BIN
docs/my-website/src/pages/tutorial-extras/img/docsVersionDropdown.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 25 KiB

BIN
docs/my-website/src/pages/tutorial-extras/img/localeDropdown.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 27 KiB

55
docs/my-website/src/pages/tutorial-extras/manage-docs-versions.md
View file

@ -1,55 +0,0 @@
---
sidebar_position: 1
---
# Manage Docs Versions
Docusaurus can manage multiple versions of your docs.
## Create a docs version
Release a version 1.0 of your project:
```bash
npm run docusaurus docs:version 1.0
```
The `docs` folder is copied into `versioned_docs/version-1.0` and `versions.json` is created.
Your docs now have 2 versions:
- `1.0` at `http://localhost:3000/docs/` for the version 1.0 docs
- `current` at `http://localhost:3000/docs/next/` for the **upcoming, unreleased docs**
## Add a Version Dropdown
To navigate seamlessly across versions, add a version dropdown.
Modify the `docusaurus.config.js` file:
```js title="docusaurus.config.js"
module.exports = {
themeConfig: {
navbar: {
items: [
// highlight-start
{
type: 'docsVersionDropdown',
},
// highlight-end
],
},
},
};
```
The docs version dropdown appears in your navbar:
![Docs Version Dropdown](./img/docsVersionDropdown.png)
## Update an existing version
It is possible to edit versioned docs in their respective folder:
- `versioned_docs/version-1.0/hello.md` updates `http://localhost:3000/docs/hello`
- `docs/hello.md` updates `http://localhost:3000/docs/next/hello`

88
docs/my-website/src/pages/tutorial-extras/translate-your-site.md
View file

@ -1,88 +0,0 @@
---
sidebar_position: 2
---
# Translate your site
Let's translate `docs/intro.md` to French.
## Configure i18n
Modify `docusaurus.config.js` to add support for the `fr` locale:
```js title="docusaurus.config.js"
module.exports = {
i18n: {
defaultLocale: 'en',
locales: ['en', 'fr'],
},
};
```
## Translate a doc
Copy the `docs/intro.md` file to the `i18n/fr` folder:
```bash
mkdir -p i18n/fr/docusaurus-plugin-content-docs/current/
cp docs/intro.md i18n/fr/docusaurus-plugin-content-docs/current/intro.md
```
Translate `i18n/fr/docusaurus-plugin-content-docs/current/intro.md` in French.
## Start your localized site
Start your site on the French locale:
```bash
npm run start -- --locale fr
```
Your localized site is accessible at [http://localhost:3000/fr/](http://localhost:3000/fr/) and the `Getting Started` page is translated.
:::caution
In development, you can only use one locale at a same time.
:::
## Add a Locale Dropdown
To navigate seamlessly across languages, add a locale dropdown.
Modify the `docusaurus.config.js` file:
```js title="docusaurus.config.js"
module.exports = {
themeConfig: {
navbar: {
items: [
// highlight-start
{
type: 'localeDropdown',
},
// highlight-end
],
},
},
};
```
The locale dropdown now appears in your navbar:
![Locale Dropdown](./img/localeDropdown.png)
## Build your localized site
Build your site for a specific locale:
```bash
npm run build -- --locale fr
```
Or build your site to include all the locales at once:
```bash
npm run build
```

1
litellm/__init__.py
View file

@ -1387,6 +1387,7 @@ from .llms.docker_model_runner.chat.transformation import DockerModelRunnerChatC
from .llms.v0.chat.transformation import V0ChatConfig
from .llms.oci.chat.transformation import OCIChatConfig
from .llms.morph.chat.transformation import MorphChatConfig
from .llms.ragflow.chat.transformation import RAGFlowConfig
from .llms.lambda_ai.chat.transformation import LambdaAIChatConfig
from .llms.hyperbolic.chat.transformation import HyperbolicChatConfig
from .llms.vercel_ai_gateway.chat.transformation import VercelAIGatewayConfig

2
litellm/batches/main.py
View file

@ -18,8 +18,6 @@ from typing import Any, Coroutine, Dict, Literal, Optional, Union, cast
import httpx
from openai.types.batch import BatchRequestCounts
from openai.types.batch import Metadata
from openai.types.batch import Metadata as OpenAIBatchMetadata
import litellm
from litellm._logging import verbose_logger

29
litellm/constants.py
View file

@ -1,4 +1,5 @@
import os
import sys
from typing import List, Literal
DEFAULT_HEALTH_CHECK_PROMPT = str(
@ -103,6 +104,12 @@ _DEFAULT_TTL_FOR_HTTPX_CLIENTS = 3600 # 1 hour, re-use the same httpx client fo
AIOHTTP_CONNECTOR_LIMIT = int(os.getenv("AIOHTTP_CONNECTOR_LIMIT", 0))
AIOHTTP_KEEPALIVE_TIMEOUT = int(os.getenv("AIOHTTP_KEEPALIVE_TIMEOUT", 120))
AIOHTTP_TTL_DNS_CACHE = int(os.getenv("AIOHTTP_TTL_DNS_CACHE", 300))
# enable_cleanup_closed is only needed for Python versions with the SSL leak bug
# Fixed in Python 3.12.7+ and 3.13.1+ (see https://github.com/python/cpython/pull/118960)
# Reference: https://github.com/aio-libs/aiohttp/blob/master/aiohttp/connector.py#L74-L78
AIOHTTP_NEEDS_CLEANUP_CLOSED = (
(3, 13, 0) <= sys.version_info < (3, 13, 1) or sys.version_info < (3, 12, 7)
)
# WebSocket constants
# Default to None (unlimited) to match OpenAI's official agents SDK behavior
@ -255,7 +262,9 @@ TOGETHER_AI_EMBEDDING_350_M = int(os.getenv("TOGETHER_AI_EMBEDDING_350_M", 350))
QDRANT_SCALAR_QUANTILE = float(os.getenv("QDRANT_SCALAR_QUANTILE", 0.99))
QDRANT_VECTOR_SIZE = int(os.getenv("QDRANT_VECTOR_SIZE", 1536))
CACHED_STREAMING_CHUNK_DELAY = float(os.getenv("CACHED_STREAMING_CHUNK_DELAY", 0.02))
AUDIO_SPEECH_CHUNK_SIZE = 8192 # chunk_size for audio speech streaming. Balance between latency and memory usage
AUDIO_SPEECH_CHUNK_SIZE = int(
os.getenv("AUDIO_SPEECH_CHUNK_SIZE", 8192)
) # chunk_size for audio speech streaming. Balance between latency and memory usage
MAX_SIZE_PER_ITEM_IN_MEMORY_CACHE_IN_KB = int(
os.getenv("MAX_SIZE_PER_ITEM_IN_MEMORY_CACHE_IN_KB", 512)
)
@ -278,10 +287,16 @@ REDACTED_BY_LITELM_STRING = "REDACTED_BY_LITELM"
MAX_LANGFUSE_INITIALIZED_CLIENTS = int(
os.getenv("MAX_LANGFUSE_INITIALIZED_CLIENTS", 50)
)
LOGGING_WORKER_CONCURRENCY = int(os.getenv("LOGGING_WORKER_CONCURRENCY", 100)) # Must be above 0
LOGGING_WORKER_CONCURRENCY = int(
os.getenv("LOGGING_WORKER_CONCURRENCY", 100)
) # Must be above 0
LOGGING_WORKER_MAX_QUEUE_SIZE = int(os.getenv("LOGGING_WORKER_MAX_QUEUE_SIZE", 50_000))
LOGGING_WORKER_MAX_TIME_PER_COROUTINE = float(os.getenv("LOGGING_WORKER_MAX_TIME_PER_COROUTINE", 20.0))
LOGGING_WORKER_CLEAR_PERCENTAGE = int(os.getenv("LOGGING_WORKER_CLEAR_PERCENTAGE", 50)) # Percentage of queue to clear (default: 50%)
LOGGING_WORKER_MAX_TIME_PER_COROUTINE = float(
os.getenv("LOGGING_WORKER_MAX_TIME_PER_COROUTINE", 20.0)
)
LOGGING_WORKER_CLEAR_PERCENTAGE = int(
os.getenv("LOGGING_WORKER_CLEAR_PERCENTAGE", 50)
) # Percentage of queue to clear (default: 50%)
MAX_ITERATIONS_TO_CLEAR_QUEUE = int(os.getenv("MAX_ITERATIONS_TO_CLEAR_QUEUE", 200))
MAX_TIME_TO_CLEAR_QUEUE = float(os.getenv("MAX_TIME_TO_CLEAR_QUEUE", 5.0))
LOGGING_WORKER_AGGRESSIVE_CLEAR_COOLDOWN_SECONDS = float(
@ -586,6 +601,7 @@ openai_compatible_providers: List = [
"cometapi",
"clarifai",
"docker_model_runner",
"ragflow",
]
openai_text_completion_compatible_providers: List = (
[ # providers that support `/v1/completions`
@ -859,7 +875,7 @@ BEDROCK_INVOKE_PROVIDERS_LITERAL = Literal[
"deepseek_r1",
"qwen3",
"twelvelabs",
"openai"
"openai",
]
BEDROCK_EMBEDDING_PROVIDERS_LITERAL = Literal[
@ -909,6 +925,9 @@ BEDROCK_CONVERSE_MODELS = [
"meta.llama3-2-3b-instruct-v1:0",
"meta.llama3-2-11b-instruct-v1:0",
"meta.llama3-2-90b-instruct-v1:0",
"amazon.nova-lite-v1:0",
"amazon.nova-2-lite-v1:0",
"amazon.nova-pro-v1:0",
]

10
litellm/integrations/SlackAlerting/budget_alert_types.py
View file

@ -50,6 +50,14 @@ class TeamBudgetAlert(BaseBudgetAlertType):
return user_info.team_id or "default_id"
class OrganizationBudgetAlert(BaseBudgetAlertType):
def get_event_message(self) -> str:
return "Organization Budget: "
def get_id(self, user_info: CallInfo) -> str:
return user_info.organization_id or "default_id"
class TokenBudgetAlert(BaseBudgetAlertType):
def get_event_message(self) -> str:
return "Key Budget: "
@ -72,6 +80,7 @@ def get_budget_alert_type(
"soft_budget",
"user_budget",
"team_budget",
"organization_budget",
"proxy_budget",
"projected_limit_exceeded",
],
@ -83,6 +92,7 @@ def get_budget_alert_type(
"soft_budget": SoftBudgetAlert(),
"user_budget": UserBudgetAlert(),
"team_budget": TeamBudgetAlert(),
"organization_budget": OrganizationBudgetAlert(),
"token_budget": TokenBudgetAlert(),
"projected_limit_exceeded": ProjectedLimitExceededAlert(),
}

19
litellm/integrations/SlackAlerting/slack_alerting.py
View file

@ -134,19 +134,25 @@ class SlackAlerting(CustomBatchLogger):
if llm_router is not None:
self.llm_router = llm_router
def _prepare_outage_value_for_cache(self, outage_value: Union[dict, ProviderRegionOutageModel, OutageModel]) -> dict:
def _prepare_outage_value_for_cache(
self, outage_value: Union[dict, ProviderRegionOutageModel, OutageModel]
) -> dict:
"""
Helper method to prepare outage value for Redis caching.
Converts set objects to lists for JSON serialization.
"""
# Convert to dict for processing
cache_value = dict(outage_value)
if "deployment_ids" in cache_value and isinstance(cache_value["deployment_ids"], set):
if "deployment_ids" in cache_value and isinstance(
cache_value["deployment_ids"], set
):
cache_value["deployment_ids"] = list(cache_value["deployment_ids"])
return cache_value
def _restore_outage_value_from_cache(self, outage_value: Optional[dict]) -> Optional[dict]:
def _restore_outage_value_from_cache(
self, outage_value: Optional[dict]
) -> Optional[dict]:
"""
Helper method to restore outage value after retrieving from cache.
Converts list objects back to sets for proper handling.
@ -528,6 +534,7 @@ class SlackAlerting(CustomBatchLogger):
"soft_budget",
"user_budget",
"team_budget",
"organization_budget",
"proxy_budget",
"projected_limit_exceeded",
],
@ -1338,7 +1345,7 @@ Model Info:
subject=email_event["subject"],
html=email_event["html"],
)
if webhook_event.event_group == "team":
if webhook_event.event_group == Litellm_EntityType.TEAM:
from litellm.integrations.email_alerting import send_team_budget_alert
await send_team_budget_alert(webhook_event=webhook_event)
@ -1399,7 +1406,7 @@ Model Info:
current_time = datetime.now().strftime("%H:%M:%S")
_proxy_base_url = os.getenv("PROXY_BASE_URL", None)
# Use .name if it's an enum, otherwise use as is
alert_type_name = getattr(alert_type, 'name', alert_type)
alert_type_name = getattr(alert_type, "name", alert_type)
alert_type_formatted = f"Alert type: `{alert_type_name}`"
if alert_type == "daily_reports" or alert_type == "new_model_added":
formatted_message = alert_type_formatted + message

1
litellm/integrations/custom_guardrail.py
View file

@ -20,7 +20,6 @@ from litellm.types.guardrails import (
GuardrailEventHooks,
LitellmParams,
Mode,
PiiEntityType,
)
from litellm.types.llms.openai import AllMessageValues
from litellm.types.proxy.guardrails.guardrail_hooks.base import GuardrailConfigModel

13
litellm/integrations/datadog/datadog.py
View file

@ -65,11 +65,11 @@ class DataDogLogger(
`DD_SITE` - your datadog site, example = `"us5.datadoghq.com"`
Optional environment variables (DataDog Agent):
`DD_AGENT_HOST` - hostname or IP of DataDog agent, example = `"localhost"`
`DD_AGENT_PORT` - port of DataDog agent (default: 10518 for logs)
`LITELLM_DD_AGENT_HOST` - hostname or IP of DataDog agent, example = `"localhost"`
`LITELLM_DD_AGENT_PORT` - port of DataDog agent (default: 10518 for logs)
Note: If DD_AGENT_HOST is set, logs will be sent to the agent instead of directly to DataDog API.
In this case, DD_API_KEY and DD_SITE are not required (agent handles authentication).
Note: We use LITELLM_DD_AGENT_HOST instead of DD_AGENT_HOST to avoid conflicts
with ddtrace which automatically sets DD_AGENT_HOST for APM tracing.
"""
try:
verbose_logger.debug("Datadog: in init datadog logger")
@ -85,7 +85,8 @@ class DataDogLogger(
)
# Configure DataDog endpoint (Agent or Direct API)
dd_agent_host = os.getenv("DD_AGENT_HOST")
# Use LITELLM_DD_AGENT_HOST to avoid conflicts with ddtrace's DD_AGENT_HOST
dd_agent_host = os.getenv("LITELLM_DD_AGENT_HOST")
if dd_agent_host:
self._configure_dd_agent(dd_agent_host=dd_agent_host)
else:
@ -127,7 +128,7 @@ class DataDogLogger(
Args:
dd_agent_host: Hostname or IP of DataDog agent
"""
dd_agent_port = os.getenv("DD_AGENT_PORT", "10518") # default port for logs
dd_agent_port = os.getenv("LITELLM_DD_AGENT_PORT", "10518") # default port for logs
self.intake_url = f"http://{dd_agent_host}:{dd_agent_port}/api/v2/logs"
self.DD_API_KEY = os.getenv("DD_API_KEY") # Optional when using agent
verbose_logger.debug(f"Datadog: Using DD Agent at {self.intake_url}")

1
litellm/litellm_core_utils/README.md
View file

@ -9,4 +9,5 @@ Core files:
- `default_encoding.py`: code for loading the default encoding (tiktoken)
- `get_llm_provider_logic.py`: code for inferring the LLM provider from a given model name.
- `duration_parser.py`: code for parsing durations - e.g. "1d", "1mo", "10s"
- `api_route_to_call_types.py`: mapping of API routes to their corresponding CallTypes (e.g., `/chat/completions` -> [acompletion, completion])

38
litellm/litellm_core_utils/api_route_to_call_types.py Normal file
View file

@ -0,0 +1,38 @@
"""
Dictionary mapping API routes to their corresponding CallTypes in LiteLLM.
This dictionary maps each API endpoint to the CallTypes that can be used for that route.
Each route can have both async (prefixed with 'a') and sync call types.
"""
from litellm.types.utils import API_ROUTE_TO_CALL_TYPES, CallTypes
def get_call_types_for_route(route: str) -> list:
"""
Get the list of CallTypes for a given API route.
Args:
route: API route path (e.g., "/chat/completions")
Returns:
List of CallTypes for that route, or empty list if route not found
"""
return API_ROUTE_TO_CALL_TYPES.get(route, [])
def get_routes_for_call_type(call_type: CallTypes) -> list:
"""
Get all routes that use a specific CallType.
Args:
call_type: The CallType to search for
Returns:
List of routes that use this CallType
"""
routes = []
for route, types in API_ROUTE_TO_CALL_TYPES.items():
if call_type in types:
routes.append(route)
return routes

10
litellm/litellm_core_utils/get_llm_provider_logic.py
View file

@ -840,6 +840,16 @@ def _get_openai_compatible_provider_info( # noqa: PLR0915
) = litellm.ClarifaiConfig()._get_openai_compatible_provider_info(
api_base, api_key
)
elif custom_llm_provider == "ragflow":
full_model = f"ragflow/{model}"
(
api_base,
dynamic_api_key,
_,
) = litellm.RAGFlowConfig()._get_openai_compatible_provider_info(
full_model, api_base, api_key, "ragflow"
)
model = full_model
if api_base is not None and not isinstance(api_base, str):
raise Exception("api base needs to be a string. api_base={}".format(api_base))

53
litellm/litellm_core_utils/streaming_handler.py
View file

@ -96,9 +96,9 @@ class CustomStreamWrapper:
self.system_fingerprint: Optional[str] = None
self.received_finish_reason: Optional[str] = None
self.intermittent_finish_reason: Optional[str] = (
None # finish reasons that show up mid-stream
)
self.intermittent_finish_reason: Optional[
str
] = None # finish reasons that show up mid-stream
self.special_tokens = [
"<|assistant|>",
"<|system|>",
@ -735,7 +735,7 @@ class CustomStreamWrapper:
and completion_obj["function_call"] is not None
)
or (
"tool_calls" in model_response.choices[0].delta
"tool_calls" in model_response.choices[0].delta
and model_response.choices[0].delta["tool_calls"] is not None
)
or (
@ -889,7 +889,6 @@ class CustomStreamWrapper:
## check if openai/azure chunk
original_chunk = response_obj.get("original_chunk", None)
if original_chunk:
if len(original_chunk.choices) > 0:
choices = []
for choice in original_chunk.choices:
@ -906,7 +905,6 @@ class CustomStreamWrapper:
print_verbose(f"choices in streaming: {choices}")
setattr(model_response, "choices", choices)
else:
return
model_response.system_fingerprint = (
original_chunk.system_fingerprint
@ -1435,9 +1433,9 @@ class CustomStreamWrapper:
_json_delta = delta.model_dump()
print_verbose(f"_json_delta: {_json_delta}")
if "role" not in _json_delta or _json_delta["role"] is None:
_json_delta["role"] = (
"assistant" # mistral's api returns role as None
)
_json_delta[
"role"
] = "assistant" # mistral's api returns role as None
if "tool_calls" in _json_delta and isinstance(
_json_delta["tool_calls"], list
):
@ -1533,7 +1531,7 @@ class CustomStreamWrapper:
async def _call_post_streaming_deployment_hook(self, chunk):
"""
Call the post-call streaming deployment hook for callbacks.
This allows callbacks to modify streaming chunks before they're returned.
"""
try:
@ -1544,15 +1542,17 @@ class CustomStreamWrapper:
# Get request kwargs from logging object
request_data = self.logging_obj.model_call_details
call_type_str = self.logging_obj.call_type
try:
typed_call_type = CallTypes(call_type_str)
except ValueError:
typed_call_type = None
# Call hooks for all callbacks
for callback in litellm.callbacks:
if isinstance(callback, CustomLogger) and hasattr(callback, "async_post_call_streaming_deployment_hook"):
if isinstance(callback, CustomLogger) and hasattr(
callback, "async_post_call_streaming_deployment_hook"
):
result = await callback.async_post_call_streaming_deployment_hook(
request_data=request_data,
response_chunk=chunk,
@ -1560,11 +1560,14 @@ class CustomStreamWrapper:
)
if result is not None:
chunk = result
return chunk
except Exception as e:
from litellm._logging import verbose_logger
verbose_logger.exception(f"Error in post-call streaming deployment hook: {str(e)}")
verbose_logger.exception(
f"Error in post-call streaming deployment hook: {str(e)}"
)
return chunk
def cache_streaming_response(self, processed_chunk, cache_hit: bool):
@ -1687,7 +1690,7 @@ class CustomStreamWrapper:
response, "usage"
): # remove usage from chunk, only send on final chunk
# Convert the object to a dictionary
obj_dict = response.dict()
obj_dict = response.model_dump()
# Remove an attribute (e.g., 'attr2')
if "usage" in obj_dict:
@ -1852,7 +1855,7 @@ class CustomStreamWrapper:
processed_chunk, "usage"
): # remove usage from chunk, only send on final chunk
# Convert the object to a dictionary
obj_dict = processed_chunk.dict()
obj_dict = processed_chunk.model_dump()
# Remove an attribute (e.g., 'attr2')
if "usage" in obj_dict:
@ -1872,11 +1875,15 @@ class CustomStreamWrapper:
if self.sent_last_chunk is True and self.stream_options is None:
usage = calculate_total_usage(chunks=self.chunks)
processed_chunk._hidden_params["usage"] = usage
# Call post-call streaming deployment hook for final chunk
if self.sent_last_chunk is True:
processed_chunk = await self._call_post_streaming_deployment_hook(processed_chunk)
processed_chunk = (
await self._call_post_streaming_deployment_hook(
processed_chunk
)
)
return processed_chunk
raise StopAsyncIteration
else: # temporary patch for non-aiohttp async calls
@ -1890,9 +1897,9 @@ class CustomStreamWrapper:
chunk = next(self.completion_stream)
if chunk is not None and chunk != b"":
print_verbose(f"PROCESSED CHUNK PRE CHUNK CREATOR: {chunk}")
processed_chunk: Optional[ModelResponseStream] = (
self.chunk_creator(chunk=chunk)
)
processed_chunk: Optional[
ModelResponseStream
] = self.chunk_creator(chunk=chunk)
print_verbose(
f"PROCESSED CHUNK POST CHUNK CREATOR: {processed_chunk}"
)

14
litellm/llms/base_llm/guardrail_translation/base_translation.py
View file

@ -84,3 +84,17 @@ class BaseTranslation(ABC):
user_api_key_dict: User API key metadata (passed separately since response doesn't contain it)
"""
pass
async def process_output_streaming_response(
self,
response: Any,
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Optional["LiteLLMLoggingObj"] = None,
user_api_key_dict: Optional["UserAPIKeyAuth"] = None,
) -> Any:
"""
Process output streaming response with guardrails.
Optional to override in subclasses.
"""
return response

10
litellm/llms/bedrock/chat/converse_handler.py
View file

@ -29,6 +29,7 @@ def make_sync_call(
logging_obj: LiteLLMLoggingObject,
json_mode: Optional[bool] = False,
fake_stream: bool = False,
stream_chunk_size: int = 1024,
):
if client is None:
client = _get_httpx_client() # Create a new client if none provided
@ -66,7 +67,7 @@ def make_sync_call(
)
else:
decoder = AWSEventStreamDecoder(model=model)
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=1024))
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=stream_chunk_size))
# LOGGING
logging_obj.post_call(
@ -102,6 +103,7 @@ class BedrockConverseLLM(BaseAWSLLM):
fake_stream: bool = False,
json_mode: Optional[bool] = False,
api_key: Optional[str] = None,
stream_chunk_size: int = 1024,
) -> CustomStreamWrapper:
request_data = await litellm.AmazonConverseConfig()._async_transform_request(
model=model,
@ -143,6 +145,7 @@ class BedrockConverseLLM(BaseAWSLLM):
logging_obj=logging_obj,
fake_stream=fake_stream,
json_mode=json_mode,
stream_chunk_size=stream_chunk_size,
)
streaming_response = CustomStreamWrapper(
completion_stream=completion_stream,
@ -260,6 +263,7 @@ class BedrockConverseLLM(BaseAWSLLM):
):
## SETUP ##
stream = optional_params.pop("stream", None)
stream_chunk_size = optional_params.pop("stream_chunk_size", 1024)
unencoded_model_id = optional_params.pop("model_id", None)
fake_stream = optional_params.pop("fake_stream", False)
json_mode = optional_params.get("json_mode", False)
@ -356,7 +360,8 @@ class BedrockConverseLLM(BaseAWSLLM):
json_mode=json_mode,
fake_stream=fake_stream,
credentials=credentials,
api_key=api_key
api_key=api_key,
stream_chunk_size=stream_chunk_size,
) # type: ignore
### ASYNC COMPLETION
return self.async_completion(
@ -433,6 +438,7 @@ class BedrockConverseLLM(BaseAWSLLM):
logging_obj=logging_obj,
json_mode=json_mode,
fake_stream=fake_stream,
stream_chunk_size=stream_chunk_size,
)
streaming_response = CustomStreamWrapper(
completion_stream=completion_stream,

102
litellm/llms/bedrock/chat/converse_transformation.py
View file

@ -246,6 +246,93 @@ class AmazonConverseConfig(BaseConfig):
llm_provider="bedrock",
)
def _is_nova_lite_2_model(self, model: str) -> bool:
"""
Check if the model is a Nova Lite 2 model that supports reasoningConfig.
Nova Lite 2 models use a different reasoning configuration structure compared to
Anthropic's thinking parameter and GPT-OSS's reasoning_effort parameter.
Supported models:
- amazon.nova-2-lite-v1:0
- us.amazon.nova-2-lite-v1:0
- eu.amazon.nova-2-lite-v1:0
- apac.amazon.nova-2-lite-v1:0
Args:
model: The model identifier
Returns:
True if the model is a Nova Lite 2 model, False otherwise
Examples:
>>> config = AmazonConverseConfig()
>>> config._is_nova_lite_2_model("amazon.nova-2-lite-v1:0")
True
>>> config._is_nova_lite_2_model("us.amazon.nova-2-lite-v1:0")
True
>>> config._is_nova_lite_2_model("amazon.nova-pro-1-5-v1:0")
False
>>> config._is_nova_lite_2_model("amazon.nova-pro-v1:0")
False
"""
# Remove regional prefix if present (us., eu., apac.)
model_without_region = model
for prefix in ["us.", "eu.", "apac."]:
if model.startswith(prefix):
model_without_region = model[len(prefix) :]
break
# Check if the model is specifically Nova Lite 2
return "nova-2-lite" in model_without_region
def _transform_reasoning_effort_to_reasoning_config(
self, reasoning_effort: str
) -> dict:
"""
Transform reasoning_effort parameter to Nova 2 reasoningConfig structure.
Nova 2 models use a reasoningConfig structure in additionalModelRequestFields
that differs from both Anthropic's thinking parameter and GPT-OSS's reasoning_effort.
Args:
reasoning_effort: The reasoning effort level, must be "low" or "high"
Returns:
dict: A dictionary containing the reasoningConfig structure:
{
"reasoningConfig": {
"type": "enabled",
"maxReasoningEffort": "low" | "medium" |"high"
}
}
Raises:
BadRequestError: If reasoning_effort is not "low", "medium" or "high"
Examples:
>>> config = AmazonConverseConfig()
>>> config._transform_reasoning_effort_to_reasoning_config("high")
{'reasoningConfig': {'type': 'enabled', 'maxReasoningEffort': 'high'}}
>>> config._transform_reasoning_effort_to_reasoning_config("low")
{'reasoningConfig': {'type': 'enabled', 'maxReasoningEffort': 'low'}}
"""
valid_values = ["low", "medium", "high"]
if reasoning_effort not in valid_values:
raise litellm.exceptions.BadRequestError(
message=f"Invalid reasoning_effort value '{reasoning_effort}' for Nova 2 models. "
f"Supported values: {valid_values}",
model="amazon.nova-2-lite-v1:0",
llm_provider="bedrock_converse",
)
return {
"reasoningConfig": {
"type": "enabled",
"maxReasoningEffort": reasoning_effort,
}
}
def get_supported_openai_params(self, model: str) -> List[str]:
from litellm.utils import supports_function_calling
@ -299,6 +386,10 @@ class AmazonConverseConfig(BaseConfig):
if "gpt-oss" in model:
supported_params.append("reasoning_effort")
elif self._is_nova_lite_2_model(model):
# Nova Lite 2 models support reasoning_effort (transformed to reasoningConfig)
# These models use a different reasoning structure than Anthropic's thinking parameter
supported_params.append("reasoning_effort")
elif (
"claude-3-7" in model
or "claude-sonnet-4" in model
@ -564,6 +655,12 @@ class AmazonConverseConfig(BaseConfig):
# GPT-OSS models: keep reasoning_effort as-is
# It will be passed through to additionalModelRequestFields
optional_params["reasoning_effort"] = value
elif self._is_nova_lite_2_model(model):
# Nova Lite 2 models: transform to reasoningConfig
reasoning_config = (
self._transform_reasoning_effort_to_reasoning_config(value)
)
optional_params.update(reasoning_config)
else:
# Anthropic and other models: convert to thinking parameter
optional_params["thinking"] = AnthropicConfig._map_reasoning_effort(
@ -574,8 +671,9 @@ class AmazonConverseConfig(BaseConfig):
self._validate_request_metadata(value) # type: ignore
optional_params["requestMetadata"] = value
# Only update thinking tokens for non-GPT-OSS models
if "gpt-oss" not in model:
# Only update thinking tokens for non-GPT-OSS models and non-Nova-Lite-2 models
# Nova Lite 2 handles token budgeting differently through reasoningConfig
if "gpt-oss" not in model and not self._is_nova_lite_2_model(model):
self.update_optional_params_with_thinking_tokens(
non_default_params=non_default_params, optional_params=optional_params
)

20
litellm/llms/bedrock/chat/invoke_handler.py
View file

@ -192,6 +192,7 @@ async def make_call(
fake_stream: bool = False,
json_mode: Optional[bool] = False,
bedrock_invoke_provider: Optional[litellm.BEDROCK_INVOKE_PROVIDERS_LITERAL] = None,
stream_chunk_size: int = 1024,
):
try:
if client is None:
@ -235,7 +236,7 @@ async def make_call(
json_mode=json_mode,
)
completion_stream = decoder.aiter_bytes(
response.aiter_bytes(chunk_size=1024)
response.aiter_bytes(chunk_size=stream_chunk_size)
)
elif bedrock_invoke_provider == "deepseek_r1":
decoder = AmazonDeepSeekR1StreamDecoder(
@ -243,12 +244,12 @@ async def make_call(
sync_stream=False,
)
completion_stream = decoder.aiter_bytes(
response.aiter_bytes(chunk_size=1024)
response.aiter_bytes(chunk_size=stream_chunk_size)
)
else:
decoder = AWSEventStreamDecoder(model=model)
completion_stream = decoder.aiter_bytes(
response.aiter_bytes(chunk_size=1024)
response.aiter_bytes(chunk_size=stream_chunk_size)
)
# LOGGING
@ -281,6 +282,7 @@ def make_sync_call(
fake_stream: bool = False,
json_mode: Optional[bool] = False,
bedrock_invoke_provider: Optional[litellm.BEDROCK_INVOKE_PROVIDERS_LITERAL] = None,
stream_chunk_size: int = 1024,
):
try:
if client is None:
@ -321,16 +323,16 @@ def make_sync_call(
sync_stream=True,
json_mode=json_mode,
)
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=1024))
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=stream_chunk_size))
elif bedrock_invoke_provider == "deepseek_r1":
decoder = AmazonDeepSeekR1StreamDecoder(
model=model,
sync_stream=True,
)
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=1024))
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=stream_chunk_size))
else:
decoder = AWSEventStreamDecoder(model=model)
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=1024))
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=stream_chunk_size))
# LOGGING
logging_obj.post_call(
@ -729,6 +731,7 @@ class BedrockLLM(BaseAWSLLM):
## SETUP ##
stream = optional_params.pop("stream", None)
stream_chunk_size = optional_params.pop("stream_chunk_size", 1024)
provider = self.get_bedrock_invoke_provider(model)
modelId = self.get_bedrock_model_id(
@ -1003,6 +1006,7 @@ class BedrockLLM(BaseAWSLLM):
headers=prepped.headers,
timeout=timeout,
client=client,
stream_chunk_size=stream_chunk_size,
) # type: ignore
### ASYNC COMPLETION
return self.async_completion(
@ -1048,7 +1052,7 @@ class BedrockLLM(BaseAWSLLM):
decoder = AWSEventStreamDecoder(model=model)
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=1024))
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=stream_chunk_size))
streaming_response = CustomStreamWrapper(
completion_stream=completion_stream,
model=model,
@ -1168,6 +1172,7 @@ class BedrockLLM(BaseAWSLLM):
logger_fn=None,
headers={},
client: Optional[AsyncHTTPHandler] = None,
stream_chunk_size: int = 1024,
) -> CustomStreamWrapper:
# The call is not made here; instead, we prepare the necessary objects for the stream.
@ -1183,6 +1188,7 @@ class BedrockLLM(BaseAWSLLM):
messages=messages,
logging_obj=logging_obj,
fake_stream=True if "ai21" in api_base else False,
stream_chunk_size=stream_chunk_size,
),
model=model,
custom_llm_provider="bedrock",

67
litellm/llms/custom_httpx/aiohttp_transport.py
View file

@ -82,9 +82,7 @@ class AiohttpResponseStream(httpx.AsyncByteStream):
async def __aiter__(self) -> typing.AsyncIterator[bytes]:
try:
async for chunk in self._aiohttp_response.content.iter_chunked(
self.CHUNK_SIZE
):
async for chunk in self._aiohttp_response.content.iter_chunked(self.CHUNK_SIZE):
yield chunk
except (
aiohttp.ClientPayloadError,
@ -120,16 +118,13 @@ class AiohttpResponseStream(httpx.AsyncByteStream):
class AiohttpTransport(httpx.AsyncBaseTransport):
def __init__(
self, client: Union[ClientSession, Callable[[], ClientSession]]
) -> None:
def __init__(self, client: Union[ClientSession, Callable[[], ClientSession]]) -> None:
self.client = client
#########################################################
# Class variables for proxy settings
#########################################################
self.proxy: Optional[str] = None
self.checked_proxy_env_settings: bool = False
self.proxy_cache: Dict[str, Optional[str]] = {}
async def aclose(self) -> None:
if isinstance(self.client, ClientSession):
@ -184,11 +179,7 @@ class LiteLLMAiohttpTransport(AiohttpTransport):
current_loop = asyncio.get_running_loop()
# If session is from a different or closed loop, recreate it
if (
session_loop is None
or session_loop != current_loop
or session_loop.is_closed()
):
if session_loop is None or session_loop != current_loop or session_loop.is_closed():
# Close old session to prevent leaks
old_session = self.client
try:
@ -215,7 +206,7 @@ class LiteLLMAiohttpTransport(AiohttpTransport):
self.client = ClientSession()
return self.client
async def _make_aiohttp_request(
self,
client_session: ClientSession,
@ -226,20 +217,20 @@ class LiteLLMAiohttpTransport(AiohttpTransport):
) -> ClientResponse:
"""
Helper function to make an aiohttp request with the given parameters.
Args:
client_session: The aiohttp ClientSession to use
request: The httpx Request to send
timeout: Timeout settings dict with 'connect', 'read', 'pool' keys
proxy: Optional proxy URL
sni_hostname: Optional SNI hostname for SSL
Returns:
ClientResponse from aiohttp
"""
from aiohttp import ClientTimeout
from yarl import URL as YarlURL
try:
data = request.content
except httpx.RequestNotRead:
@ -262,9 +253,9 @@ class LiteLLMAiohttpTransport(AiohttpTransport):
proxy=proxy,
server_hostname=sni_hostname,
).__aenter__()
return response
async def handle_async_request(
self,
request: httpx.Request,
@ -297,7 +288,7 @@ class LiteLLMAiohttpTransport(AiohttpTransport):
else:
self.client = ClientSession()
client_session = self.client
# Retry the request with the new session
with map_aiohttp_exceptions():
response = await self._make_aiohttp_request(
@ -317,45 +308,41 @@ class LiteLLMAiohttpTransport(AiohttpTransport):
content=AiohttpResponseStream(response),
request=request,
)
async def _get_proxy_settings(self, request: httpx.Request):
proxy = None
if not (
litellm.disable_aiohttp_trust_env
or str_to_bool(os.getenv("DISABLE_AIOHTTP_TRUST_ENV", "False"))
):
if not (litellm.disable_aiohttp_trust_env or str_to_bool(os.getenv("DISABLE_AIOHTTP_TRUST_ENV", "False"))):
try:
proxy = self._proxy_from_env(request.url)
except Exception as e: # pragma: no cover - best effort
verbose_logger.debug(f"Error reading proxy env: {e}")
return proxy
def _proxy_from_env(self, url: httpx.URL) -> typing.Optional[str]:
"""
Return proxy URL from env for the given request URL
Only check the proxy env settings once, this is a costly operation for CPU % usage
."""
#########################################################
# Check if we've already checked the proxy env settings
#########################################################
if self.checked_proxy_env_settings is True:
return self.proxy
#########################################################
# set self.checked_proxy_env_settings to True
#########################################################
self.checked_proxy_env_settings = True
proxy_cache_key = url.host
if proxy_cache_key in self.proxy_cache:
return self.proxy_cache[proxy_cache_key]
proxies = urllib.request.getproxies()
if urllib.request.proxy_bypass(url.host):
return None
proxy_url = None
else:
proxy = proxies.get(url.scheme) or proxies.get("all")
if proxy and "://" not in proxy:
proxy = f"http://{proxy}"
proxy_url = proxy
proxy = proxies.get(url.scheme) or proxies.get("all")
if proxy and "://" not in proxy:
proxy = f"http://{proxy}"
self.proxy = proxy
return self.proxy
self.proxy_cache[proxy_cache_key] = proxy_url
return proxy_url

3
litellm/llms/custom_httpx/http_handler.py
View file

@ -17,6 +17,7 @@ from litellm.constants import (
_DEFAULT_TTL_FOR_HTTPX_CLIENTS,
AIOHTTP_CONNECTOR_LIMIT,
AIOHTTP_KEEPALIVE_TIMEOUT,
AIOHTTP_NEEDS_CLEANUP_CLOSED,
AIOHTTP_TTL_DNS_CACHE,
DEFAULT_SSL_CIPHERS,
)
@ -798,7 +799,7 @@ class AsyncHTTPHandler:
limit=AIOHTTP_CONNECTOR_LIMIT,
keepalive_timeout=AIOHTTP_KEEPALIVE_TIMEOUT,
ttl_dns_cache=AIOHTTP_TTL_DNS_CACHE,
enable_cleanup_closed=True,
enable_cleanup_closed=AIOHTTP_NEEDS_CLEANUP_CLOSED,
**connector_kwargs,
),
trust_env=trust_env,

98
litellm/llms/openai/chat/guardrail_translation/handler.py
View file

@ -14,16 +14,16 @@ Pattern Overview:
This pattern can be replicated for other message formats (e.g., Anthropic).
"""
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, cast
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union, cast
import litellm
from litellm._logging import verbose_proxy_logger
from litellm.llms.base_llm.guardrail_translation.base_translation import BaseTranslation
from litellm.types.utils import Choices
from litellm.types.utils import Choices, StreamingChoices
if TYPE_CHECKING:
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.types.utils import ModelResponse
from litellm.types.utils import ModelResponse, ModelResponseStream
class OpenAIChatCompletionsHandler(BaseTranslation):
@ -241,21 +241,79 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
return response
def _has_text_content(self, response: "ModelResponse") -> bool:
async def process_output_streaming_response(
self,
response: "ModelResponseStream",
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Optional[Any] = None,
user_api_key_dict: Optional[Any] = None,
) -> Any:
"""
Process output streaming response by applying guardrails to text content.
Args:
response: LiteLLM ModelResponseStream object
guardrail_to_apply: The guardrail instance to apply
litellm_logging_obj: Optional logging object
user_api_key_dict: User API key metadata to pass to guardrails
Returns:
Modified response with guardrail applied to content
Response Format Support:
- String content: choice.message.content = "text here"
- List content: choice.message.content = [{"type": "text", "text": "text here"}, ...]
"""
# Step 0: Check if response has any text content to process
if not self._has_text_content(response):
return response
texts_to_check: List[str] = []
images_to_check: List[str] = []
task_mappings: List[Tuple[int, Optional[int]]] = []
# Track (choice_index, content_index) for each text
# Step 1: Extract all text content and images from response choices
for choice_idx, choice in enumerate(response.choices):
self._extract_output_text_and_images(
choice=choice,
choice_idx=choice_idx,
texts_to_check=texts_to_check,
images_to_check=images_to_check,
task_mappings=task_mappings,
)
def _has_text_content(
self, response: Union["ModelResponse", "ModelResponseStream"]
) -> bool:
"""
Check if response has any text content to process.
Override this method to customize text content detection.
"""
for choice in response.choices:
if isinstance(choice, litellm.Choices):
if choice.message.content and isinstance(choice.message.content, str):
return True
from litellm.types.utils import ModelResponse, ModelResponseStream
if isinstance(response, ModelResponse):
for choice in response.choices:
if isinstance(choice, litellm.Choices):
if choice.message.content and isinstance(
choice.message.content, str
):
return True
elif isinstance(response, ModelResponseStream):
for choice in response.choices:
if isinstance(choice, litellm.Choices):
if choice.message.content and isinstance(
choice.message.content, str
):
return True
return False
def _extract_output_text_and_images(
self,
choice: Any,
choice: Union[Choices, StreamingChoices],
choice_idx: int,
texts_to_check: List[str],
images_to_check: List[str],
@ -266,21 +324,29 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
Override this method to customize text/image extraction logic.
"""
if not isinstance(choice, litellm.Choices):
return
verbose_proxy_logger.debug(
"OpenAI Chat Completions: Processing choice: %s", choice
)
if choice.message.content and isinstance(choice.message.content, str):
# Determine content source based on choice type
content = None
if isinstance(choice, litellm.Choices):
content = choice.message.content
elif isinstance(choice, litellm.StreamingChoices):
content = choice.delta.content
else:
# Unknown choice type, skip processing
return
# Process content if it exists
if content and isinstance(content, str):
# Simple string content
texts_to_check.append(choice.message.content)
texts_to_check.append(content)
task_mappings.append((choice_idx, None))
elif choice.message.content and isinstance(choice.message.content, list):
elif content and isinstance(content, list):
# List content (e.g., multimodal response)
for content_idx, content_item in enumerate(choice.message.content):
for content_idx, content_item in enumerate(content):
# Extract text
content_text = content_item.get("text")
if content_text:

15
litellm/llms/openrouter/chat/transformation.py
View file

@ -10,6 +10,7 @@ from enum import Enum
from typing import Any, AsyncIterator, Iterator, List, Optional, Tuple, Union, cast
import httpx
import litellm
from litellm.llms.base_llm.base_model_iterator import BaseModelResponseIterator
from litellm.llms.base_llm.chat.transformation import BaseLLMException
@ -28,6 +29,20 @@ class CacheControlSupportedModels(str, Enum):
class OpenrouterConfig(OpenAIGPTConfig):
def get_supported_openai_params(self, model: str) -> list:
"""
Allow reasoning parameters for models flagged as reasoning-capable.
"""
supported_params = super().get_supported_openai_params(model=model)
try:
if litellm.supports_reasoning(
model=model, custom_llm_provider="openrouter"
) or litellm.supports_reasoning(model=model):
supported_params.append("reasoning_effort")
except Exception:
pass
return list(dict.fromkeys(supported_params))
def map_openai_params(
self,
non_default_params: dict,

8
litellm/llms/ragflow/__init__.py Normal file
View file

@ -0,0 +1,8 @@
"""
RAGFlow provider for LiteLLM.
RAGFlow provides OpenAI-compatible APIs with unique path structures:
- Chat endpoint: /api/v1/chats_openai/{chat_id}/chat/completions
- Agent endpoint: /api/v1/agents_openai/{agent_id}/chat/completions
"""

4
litellm/llms/ragflow/chat/__init__.py Normal file
View file

@ -0,0 +1,4 @@
"""
RAGFlow chat completion configuration.
"""

264
litellm/llms/ragflow/chat/transformation.py Normal file
View file

@ -0,0 +1,264 @@
"""
RAGFlow provider configuration for OpenAI-compatible API.
RAGFlow provides OpenAI-compatible APIs with unique path structures:
- Chat endpoint: /api/v1/chats_openai/{chat_id}/chat/completions
- Agent endpoint: /api/v1/agents_openai/{agent_id}/chat/completions
Model name format:
- Chat: ragflow/chat/{chat_id}/{model_name}
- Agent: ragflow/agent/{agent_id}/{model_name}
"""
from typing import List, Optional, Tuple
import litellm
from litellm.llms.openai.openai import OpenAIConfig
from litellm.secret_managers.main import get_secret, get_secret_str
from litellm.types.llms.openai import AllMessageValues
class RAGFlowConfig(OpenAIConfig):
"""
Configuration for RAGFlow OpenAI-compatible API.
Handles both chat and agent endpoints by parsing the model name format:
- ragflow/chat/{chat_id}/{model_name} for chat endpoints
- ragflow/agent/{agent_id}/{model_name} for agent endpoints
"""
def _parse_ragflow_model(self, model: str) -> Tuple[str, str, str]:
"""
Parse RAGFlow model name format: ragflow/{endpoint_type}/{id}/{model_name}
Args:
model: Model name in format ragflow/chat/{chat_id}/{model} or ragflow/agent/{agent_id}/{model}
Returns:
Tuple of (endpoint_type, id, model_name)
Raises:
ValueError: If model format is invalid
"""
parts = model.split("/")
if len(parts) < 4:
raise ValueError(
f"Invalid RAGFlow model format: {model}. "
f"Expected format: ragflow/chat/{{chat_id}}/{{model}} or ragflow/agent/{{agent_id}}/{{model}}"
)
if parts[0] != "ragflow":
raise ValueError(
f"Invalid RAGFlow model format: {model}. Must start with 'ragflow/'"
)
endpoint_type = parts[1]
if endpoint_type not in ["chat", "agent"]:
raise ValueError(
f"Invalid RAGFlow endpoint type: {endpoint_type}. Must be 'chat' or 'agent'"
)
entity_id = parts[2]
model_name = "/".join(parts[3:]) # Handle model names that might contain slashes
return endpoint_type, entity_id, model_name
def get_complete_url(
self,
api_base: Optional[str],
api_key: Optional[str],
model: str,
optional_params: dict,
litellm_params: dict,
stream: Optional[bool] = None,
) -> str:
"""
Get the complete URL for the RAGFlow API call.
Constructs URL based on endpoint type:
- Chat: /api/v1/chats_openai/{chat_id}/chat/completions
- Agent: /api/v1/agents_openai/{agent_id}/chat/completions
Args:
api_base: Base API URL (e.g., http://ragflow-server:port or http://ragflow-server:port/v1)
api_key: API key (not used in URL construction)
model: Model name in format ragflow/{endpoint_type}/{id}/{model}
optional_params: Optional parameters
litellm_params: LiteLLM parameters (may contain api_base)
stream: Whether streaming is enabled
Returns:
Complete URL for the API call
"""
# Get api_base from multiple sources: input param, litellm_params, environment, or global litellm setting
if litellm_params and hasattr(litellm_params, 'api_base') and litellm_params.api_base:
api_base = api_base or litellm_params.api_base
api_base = (
api_base
or litellm.api_base
or get_secret("RAGFLOW_API_BASE")
or get_secret_str("RAGFLOW_API_BASE")
)
if api_base is None:
raise ValueError("api_base is required for RAGFlow provider. Set it via api_base parameter, RAGFLOW_API_BASE environment variable, or litellm.api_base")
# Parse model name to extract endpoint type and ID
endpoint_type, entity_id, _ = self._parse_ragflow_model(model)
# Remove trailing slash from api_base if present
api_base = api_base.rstrip("/")
# Strip /v1 or /api/v1 from api_base if present, since we'll add the full path
# Check /api/v1 first because /api/v1 ends with /v1
if api_base.endswith("/api/v1"):
api_base = api_base[:-7] # Remove /api/v1
elif api_base.endswith("/v1"):
api_base = api_base[:-3] # Remove /v1
# Construct the RAGFlow-specific path
if endpoint_type == "chat":
path = f"/api/v1/chats_openai/{entity_id}/chat/completions"
else: # agent
path = f"/api/v1/agents_openai/{entity_id}/chat/completions"
# Ensure path starts with /
if not path.startswith("/"):
path = "/" + path
return f"{api_base}{path}"
def _get_openai_compatible_provider_info(
self,
model: str,
api_base: Optional[str],
api_key: Optional[str],
custom_llm_provider: str,
) -> Tuple[Optional[str], Optional[str], str]:
"""
Get OpenAI-compatible provider information for RAGFlow.
Args:
model: Model name (will be parsed to extract actual model name)
api_base: Base API URL (from input params)
api_key: API key (from input params)
custom_llm_provider: Custom LLM provider name
Returns:
Tuple of (api_base, api_key, custom_llm_provider)
"""
# Parse model to extract the actual model name
# The model name will be stored in litellm_params for use in requests
_, _, actual_model = self._parse_ragflow_model(model)
# Get api_base from multiple sources: input param, environment, or global litellm setting
dynamic_api_base = (
api_base
or litellm.api_base
or get_secret("RAGFLOW_API_BASE")
or get_secret_str("RAGFLOW_API_BASE")
)
# Get api_key from multiple sources: input param, environment, or global litellm setting
dynamic_api_key = (
api_key
or litellm.api_key
or get_secret_str("RAGFLOW_API_KEY")
)
return dynamic_api_base, dynamic_api_key, custom_llm_provider
def validate_environment(
self,
headers: dict,
model: str,
messages: List[AllMessageValues],
optional_params: dict,
litellm_params: dict,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
) -> dict:
"""
Validate environment and set up headers for RAGFlow API.
Args:
headers: Request headers
model: Model name
messages: Chat messages
optional_params: Optional parameters
litellm_params: LiteLLM parameters (may contain api_key)
api_key: API key (from input params)
api_base: Base API URL
Returns:
Updated headers dictionary
"""
# Use api_key from litellm_params if available, otherwise fall back to other sources
if litellm_params and hasattr(litellm_params, 'api_key') and litellm_params.api_key:
api_key = api_key or litellm_params.api_key
# Get api_key from multiple sources: input param, litellm_params, environment, or global litellm setting
api_key = (
api_key
or litellm.api_key
or get_secret_str("RAGFLOW_API_KEY")
)
if api_key is not None:
headers["Authorization"] = f"Bearer {api_key}"
# Ensure Content-Type is set to application/json
if "content-type" not in headers and "Content-Type" not in headers:
headers["Content-Type"] = "application/json"
# Parse model to extract actual model name and store it
# The actual model name should be used in the request body
try:
_, _, actual_model = self._parse_ragflow_model(model)
# Store the actual model name in litellm_params for use in transform_request
litellm_params["_ragflow_actual_model"] = actual_model
except ValueError:
# If parsing fails, use the original model name
pass
return headers
def transform_request(
self,
model: str,
messages: List[AllMessageValues],
optional_params: dict,
litellm_params: dict,
headers: dict,
) -> dict:
"""
Transform request for RAGFlow API.
Uses the actual model name extracted from the RAGFlow model format.
Args:
model: Model name in RAGFlow format
messages: Chat messages
optional_params: Optional parameters
litellm_params: LiteLLM parameters (may contain _ragflow_actual_model)
headers: Request headers
Returns:
Transformed request dictionary
"""
# Get the actual model name from litellm_params if available
actual_model = litellm_params.get("_ragflow_actual_model")
if actual_model is None:
# Fallback: try to parse the model name
try:
_, _, actual_model = self._parse_ragflow_model(model)
except ValueError:
# If parsing fails, use the original model name
actual_model = model
# Use parent's transform_request with the actual model name
return super().transform_request(
actual_model, messages, optional_params, litellm_params, headers
)

2
litellm/llms/ragflow/vector_stores/__init__.py Normal file
View file

@ -0,0 +1,2 @@
# RAGFlow vector stores module

249
litellm/llms/ragflow/vector_stores/transformation.py Normal file
View file

@ -0,0 +1,249 @@
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union
import httpx
from litellm.llms.base_llm.vector_store.transformation import BaseVectorStoreConfig
from litellm.secret_managers.main import get_secret_str
from litellm.types.router import GenericLiteLLMParams
from litellm.types.vector_stores import (
BaseVectorStoreAuthCredentials,
VectorStoreCreateOptionalRequestParams,
VectorStoreCreateResponse,
VectorStoreFileCounts,
VectorStoreIndexEndpoints,
VectorStoreSearchOptionalRequestParams,
VectorStoreSearchResponse,
)
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
else:
LiteLLMLoggingObj = Any
class RAGFlowVectorStoreConfig(BaseVectorStoreConfig):
"""Vector store configuration for RAGFlow datasets."""
def get_auth_credentials(
self, litellm_params: dict
) -> BaseVectorStoreAuthCredentials:
api_key = litellm_params.get("api_key")
if api_key is None:
# Try to get from environment variable
api_key = get_secret_str("RAGFLOW_API_KEY")
if api_key is None:
raise ValueError("api_key is required (set RAGFLOW_API_KEY env var or pass in litellm_params)")
return {
"headers": {
"Authorization": f"Bearer {api_key}",
},
}
def get_vector_store_endpoints_by_type(self) -> VectorStoreIndexEndpoints:
"""RAGFlow vector stores are management-only, no search support."""
return {
"read": [],
"write": [],
}
def validate_environment(
self, headers: dict, litellm_params: Optional[GenericLiteLLMParams]
) -> dict:
"""Validate environment and set headers for RAGFlow API."""
litellm_params = litellm_params or GenericLiteLLMParams()
api_key = (
litellm_params.api_key
or get_secret_str("RAGFLOW_API_KEY")
)
if api_key is None:
raise ValueError("RAGFLOW_API_KEY is required (set env var or pass in litellm_params)")
headers.update(
{
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
}
)
return headers
def get_complete_url(
self,
api_base: Optional[str],
litellm_params: dict,
) -> str:
"""
Get the complete URL for RAGFlow datasets API.
Supports:
- RAGFLOW_API_BASE env var
- api_base in litellm_params
- Default: http://localhost:9380
"""
api_base = (
api_base
or litellm_params.get("api_base")
or get_secret_str("RAGFLOW_API_BASE")
or "http://localhost:9380"
)
# Remove trailing slashes
api_base = api_base.rstrip("/")
# RAGFlow datasets API endpoint
return f"{api_base}/api/v1/datasets"
def transform_search_vector_store_request(
self,
vector_store_id: str,
query: Union[str, List[str]],
vector_store_search_optional_params: VectorStoreSearchOptionalRequestParams,
api_base: str,
litellm_logging_obj: LiteLLMLoggingObj,
litellm_params: dict,
) -> Tuple[str, Dict]:
"""RAGFlow vector stores are management-only, search is not supported."""
raise NotImplementedError(
"RAGFlow vector stores support dataset management only, not search/retrieval"
)
def transform_search_vector_store_response(
self, response: httpx.Response, litellm_logging_obj: LiteLLMLoggingObj
) -> VectorStoreSearchResponse:
"""RAGFlow vector stores are management-only, search is not supported."""
raise NotImplementedError(
"RAGFlow vector stores support dataset management only, not search/retrieval"
)
def transform_create_vector_store_request(
self,
vector_store_create_optional_params: VectorStoreCreateOptionalRequestParams,
api_base: str,
) -> Tuple[str, Dict]:
"""
Transform create request to RAGFlow POST /api/v1/datasets format.
Maps LiteLLM params to RAGFlow dataset creation parameters.
RAGFlow-specific fields can be passed via metadata.
"""
url = api_base # Already includes /api/v1/datasets from get_complete_url
# Extract name (required by RAGFlow)
name = vector_store_create_optional_params.get("name")
if not name:
raise ValueError("name is required for RAGFlow dataset creation")
# Build request body
request_body: Dict[str, Any] = {
"name": name,
}
# Extract RAGFlow-specific fields from metadata
metadata = vector_store_create_optional_params.get("metadata")
if metadata:
# RAGFlow-specific fields that can be in metadata
ragflow_fields = [
"avatar",
"description",
"embedding_model",
"permission",
"chunk_method",
"parser_config",
"parse_type",
"pipeline_id",
]
for field in ragflow_fields:
if field in metadata:
request_body[field] = metadata[field]
# Validate: chunk_method and pipeline_id are mutually exclusive
if "chunk_method" in request_body and "pipeline_id" in request_body:
raise ValueError(
"chunk_method and pipeline_id are mutually exclusive. "
"Specify either chunk_method or pipeline_id, not both."
)
# If neither chunk_method nor pipeline_id is specified, default to naive
if "chunk_method" not in request_body and "pipeline_id" not in request_body:
request_body["chunk_method"] = "naive"
return url, request_body
def transform_create_vector_store_response(
self, response: httpx.Response
) -> VectorStoreCreateResponse:
"""
Transform RAGFlow response to VectorStoreCreateResponse format.
RAGFlow response format:
{
"code": 0,
"data": {
"id": "...",
"name": "...",
"create_time": 1745836841611, # milliseconds
...
}
}
"""
try:
response_json = response.json()
# Check for RAGFlow error response
if response_json.get("code") != 0:
error_message = response_json.get("message", "Unknown error")
raise self.get_error_class(
error_message=error_message,
status_code=response.status_code,
headers=response.headers,
)
data = response_json.get("data", {})
# Extract dataset ID
dataset_id = data.get("id")
if not dataset_id:
raise ValueError("RAGFlow response missing dataset id")
# Extract name
name = data.get("name")
# Convert create_time from milliseconds to seconds (Unix timestamp)
create_time_ms = data.get("create_time", 0)
created_at = int(create_time_ms / 1000) if create_time_ms else None
# Build VectorStoreCreateResponse
return VectorStoreCreateResponse(
id=dataset_id,
object="vector_store",
created_at=created_at or 0,
name=name,
bytes=0, # RAGFlow doesn't provide bytes in response
file_counts=VectorStoreFileCounts(
in_progress=0,
completed=0,
failed=0,
cancelled=0,
total=0,
),
status="completed",
expires_after=None,
expires_at=None,
last_active_at=None,
metadata=None,
)
except Exception as e:
# If it's already a ValueError we raised, re-raise it
if isinstance(e, ValueError) and "RAGFlow response" in str(e):
raise
# If it's already our error class (has status_code), re-raise
if hasattr(e, "status_code"):
raise
# Otherwise, wrap in our error class
raise self.get_error_class(
error_message=str(e),
status_code=response.status_code,
headers=response.headers,
)

17
litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
View file

@ -1091,6 +1091,11 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
if "thoughtSignature" in part:
part_copy = part.copy()
part_copy.pop("thoughtSignature")
text_content = part_copy.get("text")
if isinstance(text_content, str) and text_content.strip() == "":
continue
thinking_blocks.append(
ChatCompletionThinkingBlock(
type="thinking",
@ -1205,14 +1210,16 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
}
# Embed thought signature in ID for OpenAI client compatibility
if thought_signature:
_tool_response_chunk[
"id"
] = _encode_tool_call_id_with_signature(
_tool_response_chunk["id"] or "", thought_signature
)
_tool_response_chunk["provider_specific_fields"] = { # type: ignore
"thought_signature": thought_signature
}
# Only embed in ID if preview features are enabled
if litellm.enable_preview_features:
_tool_response_chunk[
"id"
] = _encode_tool_call_id_with_signature(
_tool_response_chunk["id"] or "", thought_signature
)
_tools.append(_tool_response_chunk)
cumulative_tool_call_idx += 1
if len(_tools) == 0:

472
litellm/llms/vertex_ai/text_to_speech/transformation.py Normal file
View file

@ -0,0 +1,472 @@
"""
Vertex AI Text-to-Speech transformation
Maps OpenAI TTS spec to Google Cloud Text-to-Speech API
Reference: https://cloud.google.com/text-to-speech/docs/reference/rest/v1/text/synthesize
"""
import base64
from typing import TYPE_CHECKING, Any, Coroutine, Dict, Optional, Tuple, Union
import httpx
from litellm.llms.base_llm.text_to_speech.transformation import (
BaseTextToSpeechConfig,
TextToSpeechRequestData,
)
from litellm.llms.vertex_ai.vertex_llm_base import VertexBase
from litellm.types.llms.vertex_ai import VERTEX_CREDENTIALS_TYPES
from litellm.types.llms.vertex_ai_text_to_speech import (
VertexTextToSpeechAudioConfig,
VertexTextToSpeechInput,
VertexTextToSpeechVoice,
)
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.types.llms.openai import HttpxBinaryResponseContent
else:
LiteLLMLoggingObj = Any
HttpxBinaryResponseContent = Any
class VertexAITextToSpeechConfig(BaseTextToSpeechConfig, VertexBase):
"""
Configuration for Google Cloud/Vertex AI Text-to-Speech
Reference: https://cloud.google.com/text-to-speech/docs/reference/rest/v1/text/synthesize
"""
# Default values
DEFAULT_LANGUAGE_CODE = "en-US"
DEFAULT_VOICE_NAME = "en-US-Studio-O"
DEFAULT_AUDIO_ENCODING = "LINEAR16"
DEFAULT_SPEAKING_RATE = "1"
# API endpoint
TTS_API_URL = "https://texttospeech.googleapis.com/v1/text:synthesize"
# Voice name mappings from OpenAI voices to Google Cloud voices
# Users can pass either:
# 1. OpenAI voice names (alloy, echo, fable, onyx, nova, shimmer) - will be mapped
# 2. Google Cloud/Vertex AI voice names (en-US-Studio-O, en-US-Wavenet-D, etc.) - used directly
VOICE_MAPPINGS = {
"alloy": "en-US-Studio-O",
"echo": "en-US-Studio-M",
"fable": "en-GB-Studio-B",
"onyx": "en-US-Wavenet-D",
"nova": "en-US-Studio-O",
"shimmer": "en-US-Wavenet-F",
}
# Response format mappings from OpenAI to Google Cloud audio encoding
FORMAT_MAPPINGS = {
"mp3": "MP3",
"opus": "OGG_OPUS",
"aac": "MP3", # Google doesn't have AAC, use MP3
"flac": "FLAC",
"wav": "LINEAR16",
"pcm": "LINEAR16",
}
def __init__(self) -> None:
BaseTextToSpeechConfig.__init__(self)
VertexBase.__init__(self)
def _map_voice_to_vertex_format(
self,
voice: Optional[Union[str, Dict]],
) -> Tuple[Optional[str], Optional[Dict]]:
"""
Map voice to Vertex AI format.
Supports both:
1. OpenAI voice names (alloy, echo, fable, onyx, nova, shimmer) - will be mapped
2. Vertex AI voice names (en-US-Studio-O, en-US-Wavenet-D, etc.) - used directly
3. Dict with languageCode and name - used as-is
Returns:
Tuple of (voice_str, voice_dict) where:
- voice_str: Original string voice (for interface compatibility)
- voice_dict: Vertex AI format dict with languageCode and name
"""
if voice is None:
return None, None
if isinstance(voice, dict):
# Already in Vertex AI format
return None, voice
# voice is a string
voice_str = voice
# Map OpenAI voice if it's a known OpenAI voice, otherwise use directly
if voice in self.VOICE_MAPPINGS:
mapped_voice_name = self.VOICE_MAPPINGS[voice]
else:
# Assume it's already a Vertex AI voice name
mapped_voice_name = voice
# Extract language code from voice name (e.g., "en-US-Studio-O" -> "en-US")
parts = mapped_voice_name.split("-")
if len(parts) >= 2:
language_code = f"{parts[0]}-{parts[1]}"
else:
language_code = self.DEFAULT_LANGUAGE_CODE
voice_dict = {
"languageCode": language_code,
"name": mapped_voice_name,
}
return voice_str, voice_dict
def dispatch_text_to_speech(
self,
model: str,
input: str,
voice: Optional[Union[str, Dict]],
optional_params: Dict,
litellm_params_dict: Dict,
logging_obj: "LiteLLMLoggingObj",
timeout: Union[float, httpx.Timeout],
extra_headers: Optional[Dict[str, Any]],
base_llm_http_handler: Any,
aspeech: bool,
api_base: Optional[str],
api_key: Optional[str],
**kwargs: Any,
) -> Union[
"HttpxBinaryResponseContent",
Coroutine[Any, Any, "HttpxBinaryResponseContent"],
]:
"""
Dispatch method to handle Vertex AI TTS requests
This method encapsulates Vertex AI-specific credential resolution and parameter handling.
Voice mapping is handled in map_openai_params (similar to Azure AVA pattern).
Args:
base_llm_http_handler: The BaseLLMHTTPHandler instance from main.py
"""
# Resolve Vertex AI credentials using VertexBase helpers
vertex_credentials = self.safe_get_vertex_ai_credentials(litellm_params_dict)
vertex_project = self.safe_get_vertex_ai_project(litellm_params_dict)
vertex_location = self.safe_get_vertex_ai_location(litellm_params_dict)
# Convert voice to string if it's a dict (extract name)
# Actual voice mapping happens in map_openai_params
voice_str: Optional[str] = None
if isinstance(voice, str):
voice_str = voice
elif isinstance(voice, dict):
# Extract voice name from dict if needed
voice_str = voice.get("name") if voice else None
# Store credentials in litellm_params for use in transform methods
litellm_params_dict.update({
"vertex_credentials": vertex_credentials,
"vertex_project": vertex_project,
"vertex_location": vertex_location,
"api_base": api_base,
})
# Call the text_to_speech_handler
response = base_llm_http_handler.text_to_speech_handler(
model=model,
input=input,
voice=voice_str,
text_to_speech_provider_config=self,
text_to_speech_optional_params=optional_params,
custom_llm_provider="vertex_ai",
litellm_params=litellm_params_dict,
logging_obj=logging_obj,
timeout=timeout,
extra_headers=extra_headers,
client=None,
_is_async=aspeech,
)
return response
def get_supported_openai_params(self, model: str) -> list:
"""
Vertex AI TTS supports these OpenAI parameters
Note: Vertex AI also supports additional parameters like audioConfig
which can be passed but are not part of the OpenAI spec
"""
return ["voice", "response_format", "speed"]
def map_openai_params(
self,
model: str,
optional_params: Dict,
voice: Optional[Union[str, Dict]] = None,
drop_params: bool = False,
kwargs: Dict = {},
) -> Tuple[Optional[str], Dict]:
"""
Map OpenAI parameters to Vertex AI TTS parameters
Voice handling (similar to Azure AVA):
- If voice is an OpenAI voice name (alloy, echo, etc.), it maps to a Vertex AI voice
- If voice is already a Vertex AI voice name (en-US-Studio-O, etc.), it's used directly
- If voice is a dict with languageCode and name, it's used as-is
Note: For Vertex AI, voice dict is stored in mapped_params["vertex_voice_dict"]
because the base class interface expects voice to be a string.
Returns:
Tuple of (mapped_voice_str, mapped_params)
"""
mapped_params = {}
##########################################################
# Map voice using helper
##########################################################
mapped_voice_str, voice_dict = self._map_voice_to_vertex_format(voice)
if voice_dict is not None:
mapped_params["vertex_voice_dict"] = voice_dict
# Map response format
if "response_format" in optional_params:
format_name = optional_params["response_format"]
if format_name in self.FORMAT_MAPPINGS:
mapped_params["audioEncoding"] = self.FORMAT_MAPPINGS[format_name]
else:
# Try to use it directly as Google Cloud format
mapped_params["audioEncoding"] = format_name
else:
# Default to LINEAR16
mapped_params["audioEncoding"] = self.DEFAULT_AUDIO_ENCODING
# Map speed (OpenAI: 0.25-4.0, Vertex AI: speakingRate 0.25-4.0)
if "speed" in optional_params:
speed = optional_params["speed"]
if speed is not None:
mapped_params["speakingRate"] = str(speed)
# Pass through Vertex AI-specific parameters from kwargs
if "audioConfig" in kwargs:
mapped_params["audioConfig"] = kwargs["audioConfig"]
if "use_ssml" in kwargs:
mapped_params["use_ssml"] = kwargs["use_ssml"]
return mapped_voice_str, mapped_params
def validate_environment(
self,
headers: dict,
model: str,
api_key: Optional[str] = None,
api_base: Optional[str] = None,
) -> dict:
"""
Validate Vertex AI environment and set up authentication headers
Note: Actual authentication is handled in transform_text_to_speech_request
because Vertex AI requires OAuth2 token refresh
"""
validated_headers = headers.copy()
# Content-Type for JSON
validated_headers["Content-Type"] = "application/json"
validated_headers["charset"] = "UTF-8"
return validated_headers
def get_complete_url(
self,
model: str,
api_base: Optional[str],
litellm_params: dict,
) -> str:
"""
Get the complete URL for Vertex AI TTS request
Google Cloud TTS endpoint: https://texttospeech.googleapis.com/v1/text:synthesize
"""
if api_base:
return api_base
return self.TTS_API_URL
def _validate_vertex_input(
self,
input_data: VertexTextToSpeechInput,
optional_params: Dict,
) -> VertexTextToSpeechInput:
"""
Validate and transform input for Vertex AI TTS
Handles text vs SSML input detection and validation
"""
# Remove None values
if input_data.get("text") is None:
input_data.pop("text", None)
if input_data.get("ssml") is None:
input_data.pop("ssml", None)
# Check if use_ssml is set
use_ssml = optional_params.get("use_ssml", False)
if use_ssml:
if "text" in input_data:
input_data["ssml"] = input_data.pop("text")
elif "ssml" not in input_data:
raise ValueError("SSML input is required when use_ssml is True.")
else:
# LiteLLM will auto-detect if text is in ssml format
# check if "text" is an ssml - in this case we should pass it as ssml instead of text
if input_data:
_text = input_data.get("text", None) or ""
if "<speak>" in _text:
input_data["ssml"] = input_data.pop("text")
if not input_data:
raise ValueError("Either 'text' or 'ssml' must be provided.")
if "text" in input_data and "ssml" in input_data:
raise ValueError("Only one of 'text' or 'ssml' should be provided, not both.")
return input_data
def transform_text_to_speech_request(
self,
model: str,
input: str,
voice: Optional[str],
optional_params: Dict,
litellm_params: Dict,
headers: dict,
) -> TextToSpeechRequestData:
"""
Transform OpenAI TTS request to Vertex AI TTS format
This method handles:
1. Authentication with Vertex AI
2. Building the request body
3. Setting up headers
Returns:
TextToSpeechRequestData: Contains dict_body and headers
"""
# Get Vertex AI credentials from litellm_params
vertex_credentials: Optional[VERTEX_CREDENTIALS_TYPES] = litellm_params.get(
"vertex_credentials"
)
vertex_project: Optional[str] = litellm_params.get("vertex_project")
####### Authenticate with Vertex AI ########
_auth_header, vertex_project = self._ensure_access_token(
credentials=vertex_credentials,
project_id=vertex_project,
custom_llm_provider="vertex_ai_beta",
)
auth_header, _ = self._get_token_and_url(
model="",
auth_header=_auth_header,
gemini_api_key=None,
vertex_credentials=vertex_credentials,
vertex_project=vertex_project,
vertex_location=litellm_params.get("vertex_location"),
stream=False,
custom_llm_provider="vertex_ai_beta",
api_base=litellm_params.get("api_base"),
)
# Set authentication headers
headers["Authorization"] = f"Bearer {auth_header}"
headers["x-goog-user-project"] = vertex_project
####### Build the request ################
vertex_input = VertexTextToSpeechInput(text=input)
vertex_input = self._validate_vertex_input(vertex_input, optional_params)
# Build voice configuration
# Check for voice dict stored in:
# 1. litellm_params by dispatch method
# 2. optional_params by map_openai_params
voice_dict = (
litellm_params.get("vertex_voice_dict")
or optional_params.get("vertex_voice_dict")
)
if voice_dict is not None and isinstance(voice_dict, dict):
vertex_voice = VertexTextToSpeechVoice(**voice_dict)
elif voice is not None and isinstance(voice, str):
# Handle string voice (shouldn't normally happen if dispatch was called)
parts = voice.split("-")
if len(parts) >= 2:
language_code = f"{parts[0]}-{parts[1]}"
else:
language_code = self.DEFAULT_LANGUAGE_CODE
vertex_voice = VertexTextToSpeechVoice(
languageCode=language_code,
name=voice,
)
else:
# Use defaults
vertex_voice = VertexTextToSpeechVoice(
languageCode=self.DEFAULT_LANGUAGE_CODE,
name=self.DEFAULT_VOICE_NAME,
)
# Build audio configuration
audio_encoding = optional_params.get("audioEncoding", self.DEFAULT_AUDIO_ENCODING)
speaking_rate = optional_params.get("speakingRate", self.DEFAULT_SPEAKING_RATE)
# Check for full audioConfig in optional_params
if "audioConfig" in optional_params:
vertex_audio_config = VertexTextToSpeechAudioConfig(**optional_params["audioConfig"])
else:
vertex_audio_config = VertexTextToSpeechAudioConfig(
audioEncoding=audio_encoding,
speakingRate=speaking_rate,
)
request_body: Dict[str, Any] = {
"input": dict(vertex_input),
"voice": dict(vertex_voice),
"audioConfig": dict(vertex_audio_config),
}
return TextToSpeechRequestData(
dict_body=request_body,
headers=headers,
)
def transform_text_to_speech_response(
self,
model: str,
raw_response: httpx.Response,
logging_obj: "LiteLLMLoggingObj",
) -> "HttpxBinaryResponseContent":
"""
Transform Vertex AI TTS response to standard format
Vertex AI returns JSON with base64-encoded audio content.
We decode it and return as HttpxBinaryResponseContent.
"""
from litellm.types.llms.openai import HttpxBinaryResponseContent
# Parse JSON response
_json_response = raw_response.json()
# Get base64-encoded audio content
response_content = _json_response.get("audioContent")
if not response_content:
raise ValueError("No audioContent in Vertex AI TTS response")
# Decode base64 to get binary content
binary_data = base64.b64decode(response_content)
# Create an httpx.Response object with the binary data
response = httpx.Response(
status_code=200,
content=binary_data,
)
# Initialize the HttpxBinaryResponseContent instance
return HttpxBinaryResponseContent(response)

94
litellm/main.py
View file

@ -206,7 +206,6 @@ from .llms.vertex_ai.image_generation.image_generation_handler import (
from .llms.vertex_ai.multimodal_embeddings.embedding_handler import (
VertexMultimodalEmbedding,
)
from .llms.vertex_ai.text_to_speech.text_to_speech_handler import VertexTextToSpeechAPI
from .llms.vertex_ai.vertex_ai_partner_models.main import VertexAIPartnerModels
from .llms.vertex_ai.vertex_embeddings.embedding_handler import VertexEmbedding
from .llms.vertex_ai.vertex_gemma_models.main import VertexAIGemmaModels
@ -277,7 +276,7 @@ google_batch_embeddings = GoogleBatchEmbeddings()
vertex_partner_models_chat_completion = VertexAIPartnerModels()
vertex_gemma_chat_completion = VertexAIGemmaModels()
vertex_model_garden_chat_completion = VertexAIModelGardenModels()
vertex_text_to_speech = VertexTextToSpeechAPI()
# vertex_text_to_speech is now replaced by VertexAITextToSpeechConfig
sagemaker_llm = SagemakerLLM()
watsonx_chat_completion = WatsonXChatHandler()
openai_like_embedding = OpenAILikeEmbeddingHandler()
@ -1989,6 +1988,36 @@ def completion( # type: ignore # noqa: PLR0915
)
raise e
elif custom_llm_provider == "ragflow":
## COMPLETION CALL - RAGFlow uses HTTP handler to support custom URL paths
try:
response = base_llm_http_handler.completion(
model=model,
messages=messages,
headers=headers,
model_response=model_response,
api_key=api_key,
api_base=api_base,
acompletion=acompletion,
logging_obj=logging,
optional_params=optional_params,
litellm_params=litellm_params,
shared_session=shared_session,
timeout=timeout,
client=client,
custom_llm_provider=custom_llm_provider,
encoding=encoding,
stream=stream,
provider_config=provider_config,
)
except Exception as e:
logging.post_call(
input=messages,
api_key=api_key,
original_response=str(e),
additional_args={"headers": headers},
)
raise e
elif custom_llm_provider == "xai":
## COMPLETION CALL
try:
@ -6145,30 +6174,13 @@ def speech( # noqa: PLR0915
_is_async=aspeech or False,
)
elif custom_llm_provider == "vertex_ai" or custom_llm_provider == "vertex_ai_beta":
from litellm.llms.vertex_ai.text_to_speech.transformation import (
VertexAITextToSpeechConfig,
)
generic_optional_params = GenericLiteLLMParams(**kwargs)
api_base = generic_optional_params.api_base or ""
vertex_ai_project = (
generic_optional_params.vertex_project
or litellm.vertex_project
or get_secret_str("VERTEXAI_PROJECT")
)
vertex_ai_location = (
generic_optional_params.vertex_location
or litellm.vertex_location
or get_secret_str("VERTEXAI_LOCATION")
)
vertex_credentials = (
generic_optional_params.vertex_credentials
or get_secret_str("VERTEXAI_CREDENTIALS")
)
if voice is not None and not isinstance(voice, dict):
raise litellm.BadRequestError(
message=f"'voice' is required to be passed as a dict for Vertex AI TTS, passed in voice={voice}",
model=model,
llm_provider=custom_llm_provider,
)
# Handle Gemini models separately (they use speech_to_completion_bridge)
if "gemini" in model:
from .endpoints.speech.speech_to_completion_bridge.handler import (
speech_to_completion_bridge_handler,
@ -6184,19 +6196,37 @@ def speech( # noqa: PLR0915
logging_obj=logging_obj,
custom_llm_provider=custom_llm_provider,
)
response = vertex_text_to_speech.audio_speech(
_is_async=aspeech,
vertex_credentials=vertex_credentials,
vertex_project=vertex_ai_project,
vertex_location=vertex_ai_location,
timeout=timeout,
api_base=api_base,
# Vertex AI Text-to-Speech (Google Cloud TTS)
if text_to_speech_provider_config is None:
text_to_speech_provider_config = VertexAITextToSpeechConfig()
# Cast to specific Vertex AI config type to access dispatch method
vertex_config = cast(
VertexAITextToSpeechConfig, text_to_speech_provider_config
)
# Store Vertex AI specific params in litellm_params_dict
litellm_params_dict.update({
"vertex_project": generic_optional_params.vertex_project,
"vertex_location": generic_optional_params.vertex_location,
"vertex_credentials": generic_optional_params.vertex_credentials,
})
response = vertex_config.dispatch_text_to_speech(
model=model,
input=input,
voice=voice,
optional_params=optional_params,
kwargs=kwargs,
litellm_params_dict=litellm_params_dict,
logging_obj=logging_obj,
timeout=timeout,
extra_headers=headers,
base_llm_http_handler=base_llm_http_handler,
aspeech=aspeech or False,
api_base=generic_optional_params.api_base,
api_key=None, # Vertex AI uses OAuth, not API key
**kwargs,
)
elif custom_llm_provider == "gemini":
from .endpoints.speech.speech_to_completion_bridge.handler import (

231
litellm/model_prices_and_context_window_backup.json
View file

@ -269,6 +269,71 @@
"supports_response_schema": true,
"supports_vision": true
},
"amazon.nova-2-lite-v1:0": {
"input_cost_per_token": 3e-07,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 2.5e-06,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_video_input": true,
"supports_vision": true
},
"apac.amazon.nova-2-lite-v1:0": {
"input_cost_per_token": 6e-08,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 2.75e-06,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_video_input": true,
"supports_vision": true
},
"eu.amazon.nova-2-lite-v1:0": {
"input_cost_per_token": 6e-08,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 2.75e-06,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_video_input": true,
"supports_vision": true
},
"us.amazon.nova-2-lite-v1:0": {
"input_cost_per_token": 6e-08,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 2.75e-06,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_video_input": true,
"supports_vision": true
},
"amazon.nova-micro-v1:0": {
"input_cost_per_token": 3.5e-08,
"litellm_provider": "bedrock_converse",
@ -9564,6 +9629,21 @@
"supports_prompt_caching": true,
"supports_tool_choice": true
},
"deepseek/deepseek-v3.2": {
"input_cost_per_token": 2.8e-07,
"input_cost_per_token_cache_hit": 2.8e-08,
"litellm_provider": "deepseek",
"max_input_tokens": 163840,
"max_output_tokens": 163840,
"max_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 4e-07,
"supports_assistant_prefill": true,
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_tool_choice": true
},
"deepseek.v3-v1:0": {
"input_cost_per_token": 5.8e-07,
"litellm_provider": "bedrock_converse",
@ -10421,6 +10501,19 @@
"supports_response_schema": true,
"supports_tool_choice": true
},
"fireworks_ai/accounts/fireworks/models/deepseek-v3p2": {
"input_cost_per_token": 1.2e-06,
"litellm_provider": "fireworks_ai",
"max_input_tokens": 163840,
"max_output_tokens": 163840,
"max_tokens": 163840,
"mode": "chat",
"output_cost_per_token": 1.2e-06,
"source": "https://fireworks.ai/models/fireworks/deepseek-v3p2",
"supports_function_calling": true,
"supports_response_schema": true,
"supports_tool_choice": true
},
"fireworks_ai/accounts/fireworks/models/firefunction-v2": {
"input_cost_per_token": 9e-07,
"litellm_provider": "fireworks_ai",
@ -10759,25 +10852,25 @@
"supports_tool_choice": true
},
"ft:babbage-002": {
"input_cost_per_token": 4e-07,
"input_cost_per_token": 1.6e-06,
"input_cost_per_token_batches": 2e-07,
"litellm_provider": "text-completion-openai",
"max_input_tokens": 16384,
"max_output_tokens": 4096,
"max_tokens": 16384,
"mode": "completion",
"output_cost_per_token": 4e-07,
"output_cost_per_token": 1.6e-06,
"output_cost_per_token_batches": 2e-07
},
"ft:davinci-002": {
"input_cost_per_token": 2e-06,
"input_cost_per_token": 1.2e-05,
"input_cost_per_token_batches": 1e-06,
"litellm_provider": "text-completion-openai",
"max_input_tokens": 16384,
"max_output_tokens": 4096,
"max_tokens": 16384,
"mode": "completion",
"output_cost_per_token": 2e-06,
"output_cost_per_token": 1.2e-05,
"output_cost_per_token_batches": 1e-06
},
"ft:gpt-3.5-turbo": {
@ -10840,6 +10933,7 @@
"supports_tool_choice": true
},
"ft:gpt-4o-2024-08-06": {
"cache_read_input_token_cost": 1.875e-06,
"input_cost_per_token": 3.75e-06,
"input_cost_per_token_batches": 1.875e-06,
"litellm_provider": "openai",
@ -10852,6 +10946,7 @@
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
@ -10872,8 +10967,7 @@
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true
"supports_tool_choice": true
},
"ft:gpt-4o-mini-2024-07-18": {
"cache_read_input_token_cost": 1.5e-07,
@ -10892,8 +10986,79 @@
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true
"supports_tool_choice": true
},
"ft:gpt-4.1-2025-04-14": {
"cache_read_input_token_cost": 7.5e-07,
"input_cost_per_token": 3e-06,
"input_cost_per_token_batches": 1.5e-06,
"litellm_provider": "openai",
"max_input_tokens": 1047576,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "chat",
"output_cost_per_token": 1.2e-05,
"output_cost_per_token_batches": 6e-06,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"ft:gpt-4.1-mini-2025-04-14": {
"cache_read_input_token_cost": 2e-07,
"input_cost_per_token": 8e-07,
"input_cost_per_token_batches": 4e-07,
"litellm_provider": "openai",
"max_input_tokens": 1047576,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "chat",
"output_cost_per_token": 3.2e-06,
"output_cost_per_token_batches": 1.6e-06,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"ft:gpt-4.1-nano-2025-04-14": {
"cache_read_input_token_cost": 5e-08,
"input_cost_per_token": 2e-07,
"input_cost_per_token_batches": 1e-07,
"litellm_provider": "openai",
"max_input_tokens": 1047576,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "chat",
"output_cost_per_token": 8e-07,
"output_cost_per_token_batches": 4e-07,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"ft:o4-mini-2025-04-16": {
"cache_read_input_token_cost": 1e-06,
"input_cost_per_token": 4e-06,
"input_cost_per_token_batches": 2e-06,
"litellm_provider": "openai",
"max_input_tokens": 200000,
"max_output_tokens": 100000,
"max_tokens": 100000,
"mode": "chat",
"output_cost_per_token": 1.6e-05,
"output_cost_per_token_batches": 8e-06,
"supports_function_calling": true,
"supports_parallel_function_calling": false,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true
},
"gemini-1.0-pro": {
"input_cost_per_character": 1.25e-07,
@ -20487,6 +20652,21 @@
"supports_reasoning": true,
"supports_tool_choice": true
},
"openrouter/deepseek/deepseek-v3.2": {
"input_cost_per_token": 2.8e-07,
"input_cost_per_token_cache_hit": 2.8e-08,
"litellm_provider": "openrouter",
"max_input_tokens": 163840,
"max_output_tokens": 163840,
"max_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 4e-07,
"supports_assistant_prefill": true,
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_tool_choice": true
},
"openrouter/deepseek/deepseek-v3.2-exp": {
"input_cost_per_token": 2e-07,
"input_cost_per_token_cache_hit": 2e-08,
@ -23742,6 +23922,32 @@
"supports_vision": true,
"tool_use_system_prompt_tokens": 159
},
"global.anthropic.claude-opus-4-5-20251101-v1:0": {
"cache_creation_input_token_cost": 6.25e-06,
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 2.5e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_assistant_prefill": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true,
"tool_use_system_prompt_tokens": 159
},
"us.anthropic.claude-sonnet-4-20250514-v1:0": {
"cache_creation_input_token_cost": 3.75e-06,
"cache_read_input_token_cost": 3e-07,
@ -24825,6 +25031,15 @@
"supports_parallel_function_calling": true,
"supports_tool_choice": true
},
"vertex_ai/chirp": {
"input_cost_per_character": 30e-06,
"litellm_provider": "vertex_ai",
"mode": "audio_speech",
"source": "https://cloud.google.com/text-to-speech/pricing",
"supported_endpoints": [
"/v1/audio/speech"
]
},
"vertex_ai/claude-3-5-haiku": {
"input_cost_per_token": 1e-06,
"litellm_provider": "vertex_ai-anthropic_models",

1
litellm/proxy/_types.py
View file

@ -2444,6 +2444,7 @@ class CallInfo(LiteLLMPydanticObjectBase):
user_id: Optional[str] = None
team_id: Optional[str] = None
team_alias: Optional[str] = None
organization_id: Optional[str] = None
user_email: Optional[str] = None
key_alias: Optional[str] = None
projected_exceeded_date: Optional[str] = None

83
litellm/proxy/auth/auth_checks.py
View file

@ -143,6 +143,14 @@ async def common_checks(
valid_token=valid_token,
)
# 3.1. If organization is in budget
await _organization_max_budget_check(
valid_token=valid_token,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
proxy_logging_obj=proxy_logging_obj,
)
await _tag_max_budget_check(
request_body=request_body,
prisma_client=prisma_client,
@ -182,7 +190,18 @@ async def common_checks(
general_settings.get("enforce_user_param", None) is not None
and general_settings["enforce_user_param"] is True
):
if RouteChecks.is_llm_api_route(route=route) and "user" not in request_body:
# Get HTTP method from request
http_method = request.method if hasattr(request, 'method') else None
# Check if it's a POST request and if it's an OpenAI route but not MCP
is_post_method = http_method and http_method.upper() == "POST"
is_openai_route = RouteChecks.is_llm_api_route(route=route)
is_mcp_route = route in LiteLLMRoutes.mcp_routes.value or RouteChecks.check_route_access(
route=route, allowed_routes=LiteLLMRoutes.mcp_routes.value
)
# Enforce user param only for POST requests on OpenAI routes (excluding MCP routes)
if is_post_method and is_openai_route and not is_mcp_route and "user" not in request_body:
raise Exception(
f"'user' param not passed in. 'enforce_user_param'={general_settings['enforce_user_param']}"
)
@ -1893,6 +1912,7 @@ async def _virtual_key_max_budget_check(
max_budget=valid_token.max_budget,
user_id=valid_token.user_id,
team_id=valid_token.team_id,
organization_id=valid_token.org_id,
user_email=user_email,
key_alias=valid_token.key_alias,
event_group=Litellm_EntityType.KEY,
@ -1939,6 +1959,7 @@ async def _virtual_key_soft_budget_check(
user_id=valid_token.user_id,
team_id=valid_token.team_id,
team_alias=valid_token.team_alias,
organization_id=valid_token.org_id,
user_email=None,
key_alias=valid_token.key_alias,
event_group=Litellm_EntityType.KEY,
@ -1977,6 +1998,7 @@ async def _team_max_budget_check(
user_id=valid_token.user_id,
team_id=valid_token.team_id,
team_alias=valid_token.team_alias,
organization_id=valid_token.org_id,
event_group=Litellm_EntityType.TEAM,
)
asyncio.create_task(
@ -1993,6 +2015,65 @@ async def _team_max_budget_check(
)
async def _organization_max_budget_check(
valid_token: Optional[UserAPIKeyAuth],
prisma_client: Optional[PrismaClient],
user_api_key_cache: DualCache,
proxy_logging_obj: ProxyLogging,
):
"""
Check if the organization is over its max budget.
Raises:
BudgetExceededError if the organization is over its max budget.
Triggers a budget alert if the organization is over its max budget.
"""
# Only check if token has organization info and organization_max_budget is set
if (
valid_token is None
or valid_token.org_id is None
or valid_token.organization_max_budget is None
or valid_token.organization_max_budget <= 0
):
return
# Get organization object to check current spend
if prisma_client is not None:
org_table = await get_org_object(
org_id=valid_token.org_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
)
if (
org_table is not None
and org_table.spend >= valid_token.organization_max_budget
):
# Trigger budget alert
call_info = CallInfo(
token=valid_token.token,
spend=org_table.spend,
max_budget=valid_token.organization_max_budget,
user_id=valid_token.user_id,
team_id=valid_token.team_id,
team_alias=valid_token.team_alias,
organization_id=valid_token.org_id,
event_group=Litellm_EntityType.ORGANIZATION,
)
asyncio.create_task(
proxy_logging_obj.budget_alerts(
type="organization_budget",
user_info=call_info,
)
)
raise litellm.BudgetExceededError(
current_cost=org_table.spend,
max_budget=valid_token.organization_max_budget,
message=f"Budget has been exceeded! Organization={valid_token.org_id} Current cost: {org_table.spend}, Max budget: {valid_token.organization_max_budget}",
)
async def _tag_max_budget_check(
request_body: dict,
prisma_client: Optional[PrismaClient],

2
litellm/proxy/common_utils/custom_openapi_spec.py
View file

@ -72,7 +72,7 @@ class CustomOpenAPISpec:
openapi_schema["components"]["schemas"] = {}
# Add the schema
openapi_schema["components"]["schemas"][schema_name] = schema_def
CustomOpenAPISpec._move_defs_to_components(openapi_schema, {schema_name: schema_def})
@staticmethod
def add_request_body_to_paths(openapi_schema: Dict[str, Any], paths: List[str], schema_ref: str) -> None:

51
litellm/proxy/guardrails/guardrail_hooks/bedrock_guardrails.py
View file

@ -20,6 +20,7 @@ from typing import (
AsyncGenerator,
List,
Literal,
NamedTuple,
Optional,
Tuple,
Union,
@ -1291,38 +1292,38 @@ class BedrockGuardrail(CustomGuardrail, BaseAWSLLM):
request_data=request_data,
)
bedrock_response = await self.make_bedrock_api_request(
source="INPUT",
messages=mock_messages,
request_data=request_data,
bedrock_response = await self.make_bedrock_api_request(
source="INPUT",
messages=mock_messages,
request_data=request_data,
)
if bedrock_response.get("action") == "BLOCKED":
raise Exception(
f"Content blocked by Bedrock guardrail: {bedrock_response.get('reason', 'Unknown reason')}"
)
if bedrock_response.get("action") == "BLOCKED":
raise Exception(
f"Content blocked by Bedrock guardrail: {bedrock_response.get('reason', 'Unknown reason')}"
)
# Apply any masking that was applied by the guardrail
masked_text = text
output_list = bedrock_response.get("output")
if output_list:
# If the guardrail returned modified content, use that
for output_item in output_list:
# Apply any masking that was applied by the guardrail
masked_text = text
output_list = bedrock_response.get("output")
if output_list:
# If the guardrail returned modified content, use that
for output_item in output_list:
text_content = output_item.get("text")
if text_content:
masked_text = str(text_content)
break
else:
outputs_list = bedrock_response.get("outputs")
if outputs_list:
# Fallback to outputs field if output is not available
for output_item in outputs_list:
text_content = output_item.get("text")
if text_content:
masked_text = str(text_content)
break
else:
outputs_list = bedrock_response.get("outputs")
if outputs_list:
# Fallback to outputs field if output is not available
for output_item in outputs_list:
text_content = output_item.get("text")
if text_content:
masked_text = str(text_content)
break
masked_texts.append(masked_text)
masked_texts.append(masked_text)
verbose_proxy_logger.debug(
"Bedrock Guardrail: Successfully applied guardrail"

110
litellm/proxy/guardrails/guardrail_hooks/unified_guardrail/unified_guardrail.py
View file

@ -13,6 +13,7 @@ from litellm.caching.caching import DualCache
from litellm.cost_calculator import _infer_call_type
from litellm.integrations.custom_guardrail import CustomGuardrail
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.api_route_to_call_types import get_call_types_for_route
from litellm.llms import load_guardrail_translation_mappings
from litellm.proxy._types import UserAPIKeyAuth
from litellm.types.guardrails import GuardrailEventHooks
@ -176,6 +177,113 @@ class UnifiedLLMGuardrails(CustomLogger):
See Aim guardrail implementation for an example - https://github.com/BerriAI/litellm/blob/d0e022cfacb8e9ebc5409bb652059b6fd97b45c0/litellm/proxy/guardrails/guardrail_hooks/aim.py#L168
Triggered by mode: 'post_call'
Supports sampling_rate parameter to control how often chunks are processed.
sampling_rate=1 means every chunk, sampling_rate=5 means every 5th chunk, etc.
"""
global endpoint_guardrail_translation_mappings
from litellm.proxy.common_utils.callback_utils import (
add_guardrail_to_applied_guardrails_header,
)
guardrail_to_apply: CustomGuardrail = request_data.pop(
"guardrail_to_apply", None
)
# Get sampling rate from guardrail config or optional_params, default to 5
sampling_rate = 5
if guardrail_to_apply is not None:
# Check guardrail config first
guardrail_config = getattr(guardrail_to_apply, "guardrail_config", {})
sampling_rate = guardrail_config.get(
"streaming_sampling_rate", sampling_rate
)
# Also check optional_params as fallback
sampling_rate = self.optional_params.get(
"streaming_sampling_rate", sampling_rate
)
if guardrail_to_apply is None:
async for item in response:
yield item
return
event_type: GuardrailEventHooks = GuardrailEventHooks.post_call
if (
guardrail_to_apply.should_run_guardrail(
data=request_data, event_type=event_type
)
is not True
):
verbose_proxy_logger.debug(
"UnifiedLLMGuardrails: Post-call streaming scanning disabled for %s",
guardrail_to_apply.guardrail_name,
)
async for item in response:
yield item
return
# Initialize translation mappings if needed
if endpoint_guardrail_translation_mappings is None:
endpoint_guardrail_translation_mappings = (
load_guardrail_translation_mappings()
)
# Infer call type from first chunk
call_type = None
chunk_counter = 0
async for item in response:
yield item
chunk_counter += 1
# Infer call type from first chunk if not already done
if call_type is None and user_api_key_dict.request_route is not None:
call_types = get_call_types_for_route(user_api_key_dict.request_route)
if call_types is not None:
call_type = call_types[0]
# If call type not supported, just pass through all chunks
if (
call_type is None
or CallTypes(call_type)
not in endpoint_guardrail_translation_mappings
):
yield item
async for remaining_item in response:
yield remaining_item
return
# Process chunk based on sampling rate
if chunk_counter % sampling_rate == 0:
verbose_proxy_logger.debug(
"Processing streaming chunk %s (sampling_rate=%s) with guardrail %s",
chunk_counter,
sampling_rate,
guardrail_to_apply.guardrail_name,
)
endpoint_translation = endpoint_guardrail_translation_mappings[
CallTypes(call_type)
]()
processed_item = (
await endpoint_translation.process_output_streaming_response(
response=item,
guardrail_to_apply=guardrail_to_apply,
litellm_logging_obj=request_data.get("litellm_logging_obj"),
user_api_key_dict=user_api_key_dict,
)
)
# Add guardrail to applied guardrails header (only once, on first processed chunk)
if chunk_counter == sampling_rate:
add_guardrail_to_applied_guardrails_header(
request_data=request_data,
guardrail_name=guardrail_to_apply.guardrail_name,
)
yield processed_item
else:
yield item

3
litellm/proxy/proxy_server.py
View file

@ -32,6 +32,7 @@ from litellm._uuid import uuid
from litellm.constants import (
AIOHTTP_CONNECTOR_LIMIT,
AIOHTTP_KEEPALIVE_TIMEOUT,
AIOHTTP_NEEDS_CLEANUP_CLOSED,
AIOHTTP_TTL_DNS_CACHE,
AUDIO_SPEECH_CHUNK_SIZE,
BASE_MCP_ROUTE,
@ -635,7 +636,7 @@ async def _initialize_shared_aiohttp_session():
limit=AIOHTTP_CONNECTOR_LIMIT,
keepalive_timeout=AIOHTTP_KEEPALIVE_TIMEOUT,
ttl_dns_cache=AIOHTTP_TTL_DNS_CACHE,
enable_cleanup_closed=True,
enable_cleanup_closed=AIOHTTP_NEEDS_CLEANUP_CLOSED,
)
session = ClientSession(connector=connector)

10
litellm/proxy/public_endpoints/provider_create_fields.json
View file

@ -2607,6 +2607,16 @@
"options": null,
"default_value": null
},
{
"key": "api_base",
"label": "Google Private API Endpoint",
"placeholder": null,
"tooltip": null,
"required": false,
"field_type": "text",
"options": null,
"default_value": null
},
{
"key": "vertex_credentials",
"label": "Vertex Credentials",

22
litellm/proxy/utils.py
View file

@ -1072,6 +1072,7 @@ class ProxyLogging:
"user_budget",
"soft_budget",
"team_budget",
"organization_budget",
"proxy_budget",
"projected_limit_exceeded",
],
@ -1559,6 +1560,7 @@ class ProxyLogging:
Covers:
1. /chat/completions
"""
for callback in litellm.callbacks:
_callback: Optional[CustomLogger] = None
if isinstance(callback, str):
@ -1573,11 +1575,21 @@ class ProxyLogging:
) or _callback.should_run_guardrail(
data=request_data, event_type=GuardrailEventHooks.post_call
):
response = _callback.async_post_call_streaming_iterator_hook(
user_api_key_dict=user_api_key_dict,
response=response,
request_data=request_data,
)
if "apply_guardrail" in type(callback).__dict__:
request_data["guardrail_to_apply"] = callback
response = (
unified_guardrail.async_post_call_streaming_iterator_hook(
user_api_key_dict=user_api_key_dict,
request_data=request_data,
response=response,
)
)
else:
response = _callback.async_post_call_streaming_iterator_hook(
user_api_key_dict=user_api_key_dict,
response=response,
request_data=request_data,
)
return response
def _init_response_taking_too_long_task(self, data: Optional[dict] = None):

54
litellm/types/llms/vertex_ai_text_to_speech.py Normal file
View file

@ -0,0 +1,54 @@
"""
Type definitions for Vertex AI Text-to-Speech API
Reference: https://cloud.google.com/text-to-speech/docs/reference/rest/v1/text/synthesize
"""
from typing import Optional
from typing_extensions import TypedDict
class VertexTextToSpeechInput(TypedDict, total=False):
"""
Input for Vertex AI Text-to-Speech synthesis.
Exactly one of text or ssml must be provided.
"""
text: Optional[str]
ssml: Optional[str]
class VertexTextToSpeechVoice(TypedDict, total=False):
"""
Voice configuration for Vertex AI Text-to-Speech.
Attributes:
languageCode: The language code (e.g., "en-US", "de-DE")
name: The voice name (e.g., "en-US-Studio-O", "en-US-Wavenet-D")
"""
languageCode: str
name: str
class VertexTextToSpeechAudioConfig(TypedDict, total=False):
"""
Audio configuration for Vertex AI Text-to-Speech.
Attributes:
audioEncoding: The audio encoding format (e.g., "LINEAR16", "MP3", "OGG_OPUS")
speakingRate: The speaking rate (0.25 to 4.0, default "1")
"""
audioEncoding: str
speakingRate: str
class VertexTextToSpeechRequest(TypedDict, total=False):
"""
Request body for Vertex AI Text-to-Speech API.
Reference: https://cloud.google.com/text-to-speech/docs/reference/rest/v1/text/synthesize
"""
input: VertexTextToSpeechInput
voice: VertexTextToSpeechVoice
audioConfig: Optional[VertexTextToSpeechAudioConfig]

328
litellm/types/utils.py
View file

@ -401,6 +401,328 @@ CallTypesLiteral = Literal[
"responses",
]
# Mapping of API routes to their corresponding call types
API_ROUTE_TO_CALL_TYPES = {
# Chat Completions
"/chat/completions": [CallTypes.acompletion, CallTypes.completion],
"/v1/chat/completions": [CallTypes.acompletion, CallTypes.completion],
"/engines/{model}/chat/completions": [CallTypes.acompletion, CallTypes.completion],
"/openai/deployments/{model}/chat/completions": [
CallTypes.acompletion,
CallTypes.completion,
],
# Text Completions
"/completions": [CallTypes.atext_completion, CallTypes.text_completion],
"/v1/completions": [CallTypes.atext_completion, CallTypes.text_completion],
"/engines/{model}/completions": [
CallTypes.atext_completion,
CallTypes.text_completion,
],
"/openai/deployments/{model}/completions": [
CallTypes.atext_completion,
CallTypes.text_completion,
],
# Embeddings
"/embeddings": [CallTypes.aembedding, CallTypes.embedding],
"/v1/embeddings": [CallTypes.aembedding, CallTypes.embedding],
"/engines/{model}/embeddings": [CallTypes.aembedding, CallTypes.embedding],
"/openai/deployments/{model}/embeddings": [
CallTypes.aembedding,
CallTypes.embedding,
],
# Image Generation
"/images/generations": [CallTypes.aimage_generation, CallTypes.image_generation],
"/v1/images/generations": [CallTypes.aimage_generation, CallTypes.image_generation],
"/engines/{model}/images/generations": [
CallTypes.aimage_generation,
CallTypes.image_generation,
],
"/openai/deployments/{model}/images/generations": [
CallTypes.aimage_generation,
CallTypes.image_generation,
],
# Image Edits
"/images/edits": [CallTypes.aimage_edit, CallTypes.image_edit],
"/v1/images/edits": [CallTypes.aimage_edit, CallTypes.image_edit],
# Audio Transcriptions
"/audio/transcriptions": [CallTypes.atranscription, CallTypes.transcription],
"/v1/audio/transcriptions": [CallTypes.atranscription, CallTypes.transcription],
# Audio Speech
"/audio/speech": [CallTypes.aspeech, CallTypes.speech],
"/v1/audio/speech": [CallTypes.aspeech, CallTypes.speech],
# Moderations
"/moderations": [CallTypes.amoderation, CallTypes.moderation],
"/v1/moderations": [CallTypes.amoderation, CallTypes.moderation],
# Rerank
"/rerank": [CallTypes.arerank, CallTypes.rerank],
"/v1/rerank": [CallTypes.arerank, CallTypes.rerank],
"/v2/rerank": [CallTypes.arerank, CallTypes.rerank],
# Search
"/search": [CallTypes.asearch, CallTypes.search],
"/v1/search": [CallTypes.asearch, CallTypes.search],
# Batches
"/batches": [CallTypes.acreate_batch, CallTypes.create_batch],
"/v1/batches": [CallTypes.acreate_batch, CallTypes.create_batch],
"/batches/{batch_id}": [CallTypes.aretrieve_batch, CallTypes.retrieve_batch],
"/v1/batches/{batch_id}": [CallTypes.aretrieve_batch, CallTypes.retrieve_batch],
# Files
"/files": [
CallTypes.acreate_file,
CallTypes.create_file,
CallTypes.afile_list,
CallTypes.file_list,
],
"/v1/files": [
CallTypes.acreate_file,
CallTypes.create_file,
CallTypes.afile_list,
CallTypes.file_list,
],
"/files/{file_id}": [
CallTypes.afile_retrieve,
CallTypes.file_retrieve,
CallTypes.afile_delete,
CallTypes.file_delete,
],
"/v1/files/{file_id}": [
CallTypes.afile_retrieve,
CallTypes.file_retrieve,
CallTypes.afile_delete,
CallTypes.file_delete,
],
"/files/{file_id}/content": [CallTypes.afile_content, CallTypes.file_content],
"/v1/files/{file_id}/content": [CallTypes.afile_content, CallTypes.file_content],
# Assistants
"/assistants": [
CallTypes.aget_assistants,
CallTypes.get_assistants,
CallTypes.acreate_assistants,
CallTypes.create_assistants,
],
"/v1/assistants": [
CallTypes.aget_assistants,
CallTypes.get_assistants,
CallTypes.acreate_assistants,
CallTypes.create_assistants,
],
"/assistants/{assistant_id}": [
CallTypes.adelete_assistant,
CallTypes.delete_assistant,
],
"/v1/assistants/{assistant_id}": [
CallTypes.adelete_assistant,
CallTypes.delete_assistant,
],
# Threads
"/threads": [CallTypes.acreate_thread, CallTypes.create_thread],
"/v1/threads": [CallTypes.acreate_thread, CallTypes.create_thread],
"/threads/{thread_id}": [CallTypes.aget_thread, CallTypes.get_thread],
"/v1/threads/{thread_id}": [CallTypes.aget_thread, CallTypes.get_thread],
# Thread Messages
"/threads/{thread_id}/messages": [
CallTypes.a_add_message,
CallTypes.add_message,
CallTypes.aget_messages,
CallTypes.get_messages,
],
"/v1/threads/{thread_id}/messages": [
CallTypes.a_add_message,
CallTypes.add_message,
CallTypes.aget_messages,
CallTypes.get_messages,
],
# Thread Runs
"/threads/{thread_id}/runs": [
CallTypes.arun_thread,
CallTypes.run_thread,
CallTypes.arun_thread_stream,
CallTypes.run_thread_stream,
],
"/v1/threads/{thread_id}/runs": [
CallTypes.arun_thread,
CallTypes.run_thread,
CallTypes.arun_thread_stream,
CallTypes.run_thread_stream,
],
# Fine-tuning Jobs
"/fine_tuning/jobs": [
CallTypes.acreate_fine_tuning_job,
CallTypes.create_fine_tuning_job,
CallTypes.alist_fine_tuning_jobs,
CallTypes.list_fine_tuning_jobs,
],
"/v1/fine_tuning/jobs": [
CallTypes.acreate_fine_tuning_job,
CallTypes.create_fine_tuning_job,
CallTypes.alist_fine_tuning_jobs,
CallTypes.list_fine_tuning_jobs,
],
"/fine_tuning/jobs/{fine_tuning_job_id}": [
CallTypes.aretrieve_fine_tuning_job,
CallTypes.retrieve_fine_tuning_job,
],
"/v1/fine_tuning/jobs/{fine_tuning_job_id}": [
CallTypes.aretrieve_fine_tuning_job,
CallTypes.retrieve_fine_tuning_job,
],
"/fine_tuning/jobs/{fine_tuning_job_id}/cancel": [
CallTypes.acancel_fine_tuning_job,
CallTypes.cancel_fine_tuning_job,
],
"/v1/fine_tuning/jobs/{fine_tuning_job_id}/cancel": [
CallTypes.acancel_fine_tuning_job,
CallTypes.cancel_fine_tuning_job,
],
# Video Generation
"/videos": [
CallTypes.acreate_video,
CallTypes.create_video,
CallTypes.avideo_list,
CallTypes.video_list,
],
"/v1/videos": [
CallTypes.acreate_video,
CallTypes.create_video,
CallTypes.avideo_list,
CallTypes.video_list,
],
"/videos/{video_id}": [
CallTypes.avideo_retrieve,
CallTypes.video_retrieve,
CallTypes.avideo_delete,
CallTypes.video_delete,
],
"/v1/videos/{video_id}": [
CallTypes.avideo_retrieve,
CallTypes.video_retrieve,
CallTypes.avideo_delete,
CallTypes.video_delete,
],
"/videos/{video_id}/content": [CallTypes.avideo_content, CallTypes.video_content],
"/v1/videos/{video_id}/content": [
CallTypes.avideo_content,
CallTypes.video_content,
],
"/videos/{video_id}/remix": [CallTypes.avideo_remix, CallTypes.video_remix],
"/v1/videos/{video_id}/remix": [CallTypes.avideo_remix, CallTypes.video_remix],
# Vector Stores
"/vector_stores": [CallTypes.avector_store_create, CallTypes.vector_store_create],
"/v1/vector_stores": [
CallTypes.avector_store_create,
CallTypes.vector_store_create,
],
"/vector_stores/{vector_store_id}/search": [
CallTypes.avector_store_search,
CallTypes.vector_store_search,
],
"/v1/vector_stores/{vector_store_id}/search": [
CallTypes.avector_store_search,
CallTypes.vector_store_search,
],
"/vector_stores/{vector_store_id}/files": [
CallTypes.avector_store_file_create,
CallTypes.vector_store_file_create,
CallTypes.avector_store_file_list,
CallTypes.vector_store_file_list,
],
"/v1/vector_stores/{vector_store_id}/files": [
CallTypes.avector_store_file_create,
CallTypes.vector_store_file_create,
CallTypes.avector_store_file_list,
CallTypes.vector_store_file_list,
],
"/vector_stores/{vector_store_id}/files/{file_id}": [
CallTypes.avector_store_file_retrieve,
CallTypes.vector_store_file_retrieve,
CallTypes.avector_store_file_delete,
CallTypes.vector_store_file_delete,
],
"/v1/vector_stores/{vector_store_id}/files/{file_id}": [
CallTypes.avector_store_file_retrieve,
CallTypes.vector_store_file_retrieve,
CallTypes.avector_store_file_delete,
CallTypes.vector_store_file_delete,
],
"/vector_stores/{vector_store_id}/files/{file_id}/content": [
CallTypes.avector_store_file_content,
CallTypes.vector_store_file_content,
],
"/v1/vector_stores/{vector_store_id}/files/{file_id}/content": [
CallTypes.avector_store_file_content,
CallTypes.vector_store_file_content,
],
"/vector_stores/{vector_store_id}/files/{file_id}/update": [
CallTypes.avector_store_file_update,
CallTypes.vector_store_file_update,
],
"/v1/vector_stores/{vector_store_id}/files/{file_id}/update": [
CallTypes.avector_store_file_update,
CallTypes.vector_store_file_update,
],
# Containers
"/containers": [
CallTypes.acreate_container,
CallTypes.create_container,
CallTypes.alist_containers,
CallTypes.list_containers,
],
"/v1/containers": [
CallTypes.acreate_container,
CallTypes.create_container,
CallTypes.alist_containers,
CallTypes.list_containers,
],
"/containers/{container_id}": [
CallTypes.aretrieve_container,
CallTypes.retrieve_container,
CallTypes.adelete_container,
CallTypes.delete_container,
],
"/v1/containers/{container_id}": [
CallTypes.aretrieve_container,
CallTypes.retrieve_container,
CallTypes.adelete_container,
CallTypes.delete_container,
],
# Responses API
"/responses": [CallTypes.aresponses, CallTypes.responses],
"/v1/responses": [CallTypes.aresponses, CallTypes.responses],
"/responses/{response_id}": [CallTypes.aresponses, CallTypes.responses],
"/v1/responses/{response_id}": [CallTypes.aresponses, CallTypes.responses],
"/responses/{response_id}/input_items": [CallTypes.alist_input_items],
"/v1/responses/{response_id}/input_items": [CallTypes.alist_input_items],
# Realtime API
"/realtime": [CallTypes.arealtime],
"/v1/realtime": [CallTypes.arealtime],
# Provider-specific routes
"/anthropic/v1/messages": [CallTypes.anthropic_messages],
# Google GenAI routes
"/generate_content": [CallTypes.agenerate_content, CallTypes.generate_content],
"/models/{model}:generateContent": [
CallTypes.agenerate_content,
CallTypes.generate_content,
],
"/generate_content_stream": [
CallTypes.agenerate_content_stream,
CallTypes.generate_content_stream,
],
"/models/{model}:streamGenerateContent": [
CallTypes.agenerate_content_stream,
CallTypes.generate_content_stream,
],
# MCP (Model Context Protocol)
"/mcp/call_tool": [CallTypes.call_mcp_tool],
# Passthrough endpoints
"/llm_passthrough": [
CallTypes.llm_passthrough_route,
CallTypes.allm_passthrough_route,
],
"/v1/llm_passthrough": [
CallTypes.llm_passthrough_route,
CallTypes.allm_passthrough_route,
],
}
class PassthroughCallTypes(Enum):
passthrough_image_generation = "passthrough-image-generation"
@ -1060,7 +1382,10 @@ class Usage(CompletionUsage):
# Auto-calculate text_tokens only if provider didn't set it explicitly
# Formula: text_tokens = completion_tokens - reasoning_tokens - image_tokens - audio_tokens
if _completion_tokens_details.text_tokens is None and completion_tokens is not None:
if (
_completion_tokens_details.text_tokens is None
and completion_tokens is not None
):
calculated_text_tokens = completion_tokens - reasoning_tokens
# Subtract other modality tokens if present
@ -2618,6 +2943,7 @@ class LlmProviders(str, Enum):
DATABRICKS = "databricks"
EMPOWER = "empower"
GITHUB = "github"
RAGFLOW = "ragflow"
COMPACTIFAI = "compactifai"
DOCKER_MODEL_RUNNER = "docker_model_runner"
CUSTOM = "custom"

4
litellm/types/vector_stores.py
View file

@ -3,17 +3,15 @@ from datetime import datetime
from enum import Enum
from typing import Any, Dict, List, Literal, Optional, Tuple, Union
from annotated_types import Ge
from pydantic import BaseModel
from typing_extensions import TypedDict
from litellm.types.router import CredentialLiteLLMParams, GenericLiteLLMParams
class SupportedVectorStoreIntegrations(str, Enum):
"""Supported vector store integrations."""
BEDROCK = "bedrock"
RAGFLOW = "ragflow"
class LiteLLM_VectorStoreConfig(TypedDict, total=False):

14
litellm/utils.py
View file

@ -7110,6 +7110,8 @@ class ProviderConfigManager:
return litellm.CompactifAIChatConfig()
elif litellm.LlmProviders.GITHUB_COPILOT == provider:
return litellm.GithubCopilotConfig()
elif litellm.LlmProviders.RAGFLOW == provider:
return litellm.RAGFlowConfig()
elif (
litellm.LlmProviders.CUSTOM == provider
or litellm.LlmProviders.CUSTOM_OPENAI == provider
@ -7631,6 +7633,12 @@ class ProviderConfigManager:
)
return GeminiVectorStoreConfig()
elif litellm.LlmProviders.RAGFLOW == provider:
from litellm.llms.ragflow.vector_stores.transformation import (
RAGFlowVectorStoreConfig,
)
return RAGFlowVectorStoreConfig()
return None
@staticmethod
@ -7910,6 +7918,12 @@ class ProviderConfigManager:
)
return RunwayMLTextToSpeechConfig()
elif litellm.LlmProviders.VERTEX_AI == provider:
from litellm.llms.vertex_ai.text_to_speech.transformation import (
VertexAITextToSpeechConfig,
)
return VertexAITextToSpeechConfig()
return None
@staticmethod

231
model_prices_and_context_window.json
View file

@ -269,6 +269,71 @@
"supports_response_schema": true,
"supports_vision": true
},
"amazon.nova-2-lite-v1:0": {
"input_cost_per_token": 3e-07,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 2.5e-06,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_video_input": true,
"supports_vision": true
},
"apac.amazon.nova-2-lite-v1:0": {
"input_cost_per_token": 6e-08,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 2.75e-06,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_video_input": true,
"supports_vision": true
},
"eu.amazon.nova-2-lite-v1:0": {
"input_cost_per_token": 6e-08,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 2.75e-06,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_video_input": true,
"supports_vision": true
},
"us.amazon.nova-2-lite-v1:0": {
"input_cost_per_token": 6e-08,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 1000000,
"max_output_tokens": 64000,
"max_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 2.75e-06,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_video_input": true,
"supports_vision": true
},
"amazon.nova-micro-v1:0": {
"input_cost_per_token": 3.5e-08,
"litellm_provider": "bedrock_converse",
@ -9564,6 +9629,21 @@
"supports_prompt_caching": true,
"supports_tool_choice": true
},
"deepseek/deepseek-v3.2": {
"input_cost_per_token": 2.8e-07,
"input_cost_per_token_cache_hit": 2.8e-08,
"litellm_provider": "deepseek",
"max_input_tokens": 163840,
"max_output_tokens": 163840,
"max_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 4e-07,
"supports_assistant_prefill": true,
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_tool_choice": true
},
"deepseek.v3-v1:0": {
"input_cost_per_token": 5.8e-07,
"litellm_provider": "bedrock_converse",
@ -10421,6 +10501,19 @@
"supports_response_schema": true,
"supports_tool_choice": true
},
"fireworks_ai/accounts/fireworks/models/deepseek-v3p2": {
"input_cost_per_token": 1.2e-06,
"litellm_provider": "fireworks_ai",
"max_input_tokens": 163840,
"max_output_tokens": 163840,
"max_tokens": 163840,
"mode": "chat",
"output_cost_per_token": 1.2e-06,
"source": "https://fireworks.ai/models/fireworks/deepseek-v3p2",
"supports_function_calling": true,
"supports_response_schema": true,
"supports_tool_choice": true
},
"fireworks_ai/accounts/fireworks/models/firefunction-v2": {
"input_cost_per_token": 9e-07,
"litellm_provider": "fireworks_ai",
@ -10759,25 +10852,25 @@
"supports_tool_choice": true
},
"ft:babbage-002": {
"input_cost_per_token": 4e-07,
"input_cost_per_token": 1.6e-06,
"input_cost_per_token_batches": 2e-07,
"litellm_provider": "text-completion-openai",
"max_input_tokens": 16384,
"max_output_tokens": 4096,
"max_tokens": 16384,
"mode": "completion",
"output_cost_per_token": 4e-07,
"output_cost_per_token": 1.6e-06,
"output_cost_per_token_batches": 2e-07
},
"ft:davinci-002": {
"input_cost_per_token": 2e-06,
"input_cost_per_token": 1.2e-05,
"input_cost_per_token_batches": 1e-06,
"litellm_provider": "text-completion-openai",
"max_input_tokens": 16384,
"max_output_tokens": 4096,
"max_tokens": 16384,
"mode": "completion",
"output_cost_per_token": 2e-06,
"output_cost_per_token": 1.2e-05,
"output_cost_per_token_batches": 1e-06
},
"ft:gpt-3.5-turbo": {
@ -10840,6 +10933,7 @@
"supports_tool_choice": true
},
"ft:gpt-4o-2024-08-06": {
"cache_read_input_token_cost": 1.875e-06,
"input_cost_per_token": 3.75e-06,
"input_cost_per_token_batches": 1.875e-06,
"litellm_provider": "openai",
@ -10852,6 +10946,7 @@
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
@ -10872,8 +10967,7 @@
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true
"supports_tool_choice": true
},
"ft:gpt-4o-mini-2024-07-18": {
"cache_read_input_token_cost": 1.5e-07,
@ -10892,8 +10986,79 @@
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true,
"supports_vision": true
"supports_tool_choice": true
},
"ft:gpt-4.1-2025-04-14": {
"cache_read_input_token_cost": 7.5e-07,
"input_cost_per_token": 3e-06,
"input_cost_per_token_batches": 1.5e-06,
"litellm_provider": "openai",
"max_input_tokens": 1047576,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "chat",
"output_cost_per_token": 1.2e-05,
"output_cost_per_token_batches": 6e-06,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"ft:gpt-4.1-mini-2025-04-14": {
"cache_read_input_token_cost": 2e-07,
"input_cost_per_token": 8e-07,
"input_cost_per_token_batches": 4e-07,
"litellm_provider": "openai",
"max_input_tokens": 1047576,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "chat",
"output_cost_per_token": 3.2e-06,
"output_cost_per_token_batches": 1.6e-06,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"ft:gpt-4.1-nano-2025-04-14": {
"cache_read_input_token_cost": 5e-08,
"input_cost_per_token": 2e-07,
"input_cost_per_token_batches": 1e-07,
"litellm_provider": "openai",
"max_input_tokens": 1047576,
"max_output_tokens": 32768,
"max_tokens": 32768,
"mode": "chat",
"output_cost_per_token": 8e-07,
"output_cost_per_token_batches": 4e-07,
"supports_function_calling": true,
"supports_parallel_function_calling": true,
"supports_prompt_caching": true,
"supports_response_schema": true,
"supports_system_messages": true,
"supports_tool_choice": true
},
"ft:o4-mini-2025-04-16": {
"cache_read_input_token_cost": 1e-06,
"input_cost_per_token": 4e-06,
"input_cost_per_token_batches": 2e-06,
"litellm_provider": "openai",
"max_input_tokens": 200000,
"max_output_tokens": 100000,
"max_tokens": 100000,
"mode": "chat",
"output_cost_per_token": 1.6e-05,
"output_cost_per_token_batches": 8e-06,
"supports_function_calling": true,
"supports_parallel_function_calling": false,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true
},
"gemini-1.0-pro": {
"input_cost_per_character": 1.25e-07,
@ -20487,6 +20652,21 @@
"supports_reasoning": true,
"supports_tool_choice": true
},
"openrouter/deepseek/deepseek-v3.2": {
"input_cost_per_token": 2.8e-07,
"input_cost_per_token_cache_hit": 2.8e-08,
"litellm_provider": "openrouter",
"max_input_tokens": 163840,
"max_output_tokens": 163840,
"max_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 4e-07,
"supports_assistant_prefill": true,
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_tool_choice": true
},
"openrouter/deepseek/deepseek-v3.2-exp": {
"input_cost_per_token": 2e-07,
"input_cost_per_token_cache_hit": 2e-08,
@ -23742,6 +23922,32 @@
"supports_vision": true,
"tool_use_system_prompt_tokens": 159
},
"global.anthropic.claude-opus-4-5-20251101-v1:0": {
"cache_creation_input_token_cost": 6.25e-06,
"cache_read_input_token_cost": 5e-07,
"input_cost_per_token": 5e-06,
"litellm_provider": "bedrock_converse",
"max_input_tokens": 200000,
"max_output_tokens": 64000,
"max_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 2.5e-05,
"search_context_cost_per_query": {
"search_context_size_high": 0.01,
"search_context_size_low": 0.01,
"search_context_size_medium": 0.01
},
"supports_assistant_prefill": true,
"supports_computer_use": true,
"supports_function_calling": true,
"supports_pdf_input": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_tool_choice": true,
"supports_vision": true,
"tool_use_system_prompt_tokens": 159
},
"us.anthropic.claude-sonnet-4-20250514-v1:0": {
"cache_creation_input_token_cost": 3.75e-06,
"cache_read_input_token_cost": 3e-07,
@ -24825,6 +25031,15 @@
"supports_parallel_function_calling": true,
"supports_tool_choice": true
},
"vertex_ai/chirp": {
"input_cost_per_character": 30e-06,
"litellm_provider": "vertex_ai",
"mode": "audio_speech",
"source": "https://cloud.google.com/text-to-speech/pricing",
"supported_endpoints": [
"/v1/audio/speech"
]
},
"vertex_ai/claude-3-5-haiku": {
"input_cost_per_token": 1e-06,
"litellm_provider": "vertex_ai-anthropic_models",

16
provider_endpoints_support.json
View file

@ -734,6 +734,22 @@
"ocr": true
}
},
"vertex_ai/chirp": {
"display_name": "Google - Vertex AI Chirp3 HD (`vertex_ai/chirp`)",
"url": "https://docs.litellm.ai/docs/providers/vertex_speech",
"endpoints": {
"chat_completions": false,
"messages": false,
"responses": false,
"embeddings": false,
"image_generations": false,
"audio_transcriptions": false,
"audio_speech": true,
"moderations": false,
"batches": false,
"rerank": false
}
},
"gemini": {
"display_name": "Google AI Studio - Gemini (`gemini`)",
"url": "https://docs.litellm.ai/docs/providers/gemini",

BIN
tests/audio_tests/speech_vertex.mp3 Normal file
View file

Binary file not shown.

116
tests/enterprise/litellm_enterprise/proxy/guardrails/test_apply_guardrail_endpoint.py
View file

@ -1,6 +1,7 @@
"""
Test the /guardrails/apply_guardrail endpoint
"""
import os
import sys
from unittest.mock import AsyncMock, Mock, patch
@ -22,37 +23,45 @@ async def test_apply_guardrail_endpoint_returns_correct_response():
from litellm.proxy.guardrails.guardrail_endpoints import apply_guardrail
# Mock the guardrail registry
with patch("litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY") as mock_registry:
with patch(
"litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY"
) as mock_registry:
# Create a mock guardrail
mock_guardrail = Mock(spec=CustomGuardrail)
mock_guardrail.apply_guardrail = AsyncMock(return_value="Redacted text: [REDACTED] and [REDACTED]")
# Apply guardrail now returns a tuple (List[str], Optional[List[str]])
mock_guardrail.apply_guardrail = AsyncMock(
return_value=(["Redacted text: [REDACTED] and [REDACTED]"], None)
)
# Configure the registry to return our mock guardrail
mock_registry.get_initialized_guardrail_callback.return_value = mock_guardrail
# Create the request
request = ApplyGuardrailRequest(
guardrail_name="test-guardrail",
text="Test text with PII",
language="en",
entities=["EMAIL_ADDRESS", "PERSON"]
entities=["EMAIL_ADDRESS", "PERSON"],
)
# Create a mock user API key
user_api_key_dict = UserAPIKeyAuth(api_key="test-key")
# Call the endpoint
response = await apply_guardrail(request=request, user_api_key_dict=user_api_key_dict)
response = await apply_guardrail(
request=request, user_api_key_dict=user_api_key_dict
)
# Verify the response is of the correct type
assert isinstance(response, ApplyGuardrailResponse)
assert response.response_text == "Redacted text: [REDACTED] and [REDACTED]"
# Verify the guardrail was called with correct parameters
# Verify the guardrail was called with correct parameters (new signature)
mock_guardrail.apply_guardrail.assert_called_once_with(
text="Test text with PII",
language="en",
entities=["EMAIL_ADDRESS", "PERSON"]
texts=["Test text with PII"],
request_data={},
input_type="request",
images=None,
)
@ -63,23 +72,23 @@ async def test_apply_guardrail_endpoint_guardrail_not_found():
from litellm.proxy.guardrails.guardrail_endpoints import apply_guardrail
# Mock the guardrail registry to return None
with patch("litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY") as mock_registry:
with patch(
"litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY"
) as mock_registry:
mock_registry.get_initialized_guardrail_callback.return_value = None
# Create the request
request = ApplyGuardrailRequest(
guardrail_name="non-existent-guardrail",
text="Test text",
language="en"
guardrail_name="non-existent-guardrail", text="Test text", language="en"
)
# Create a mock user API key
user_api_key_dict = UserAPIKeyAuth(api_key="test-key")
# Verify exception is raised
with pytest.raises(ProxyException) as exc_info:
await apply_guardrail(request=request, user_api_key_dict=user_api_key_dict)
assert "non-existent-guardrail" in exc_info.value.message
assert "not found" in exc_info.value.message
@ -90,34 +99,41 @@ async def test_apply_guardrail_endpoint_with_presidio_guardrail():
from litellm.proxy.guardrails.guardrail_endpoints import apply_guardrail
# Mock the guardrail registry
with patch("litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY") as mock_registry:
with patch(
"litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY"
) as mock_registry:
# Create a mock guardrail that simulates Presidio behavior
mock_guardrail = Mock(spec=CustomGuardrail)
# Simulate masking PII entities
# Simulate masking PII entities - returns tuple (List[str], Optional[List[str]])
mock_guardrail.apply_guardrail = AsyncMock(
return_value="My name is [PERSON] and my email is [EMAIL_ADDRESS]"
return_value=(["My name is [PERSON] and my email is [EMAIL_ADDRESS]"], None)
)
# Configure the registry to return our mock guardrail
mock_registry.get_initialized_guardrail_callback.return_value = mock_guardrail
# Create the request
request = ApplyGuardrailRequest(
guardrail_name="pii-detection-guard",
text="My name is John Doe and my email is john@example.com",
language="en",
entities=["EMAIL_ADDRESS", "PERSON"]
entities=["EMAIL_ADDRESS", "PERSON"],
)
# Create a mock user API key
user_api_key_dict = UserAPIKeyAuth(api_key="test-key")
# Call the endpoint
response = await apply_guardrail(request=request, user_api_key_dict=user_api_key_dict)
response = await apply_guardrail(
request=request, user_api_key_dict=user_api_key_dict
)
# Verify the response is of the correct type
assert isinstance(response, ApplyGuardrailResponse)
assert response.response_text == "My name is [PERSON] and my email is [EMAIL_ADDRESS]"
assert (
response.response_text
== "My name is [PERSON] and my email is [EMAIL_ADDRESS]"
)
assert "john@example.com" not in response.response_text
assert "John Doe" not in response.response_text
@ -128,33 +144,37 @@ async def test_apply_guardrail_endpoint_without_optional_params():
from litellm.proxy.guardrails.guardrail_endpoints import apply_guardrail
# Mock the guardrail registry
with patch("litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY") as mock_registry:
with patch(
"litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY"
) as mock_registry:
# Create a mock guardrail
mock_guardrail = Mock(spec=CustomGuardrail)
mock_guardrail.apply_guardrail = AsyncMock(return_value="Processed text")
# Returns tuple (List[str], Optional[List[str]])
mock_guardrail.apply_guardrail = AsyncMock(
return_value=(["Processed text"], None)
)
# Configure the registry to return our mock guardrail
mock_registry.get_initialized_guardrail_callback.return_value = mock_guardrail
# Create the request without optional parameters
request = ApplyGuardrailRequest(
guardrail_name="test-guardrail",
text="Test text"
guardrail_name="test-guardrail", text="Test text"
)
# Create a mock user API key
user_api_key_dict = UserAPIKeyAuth(api_key="test-key")
# Call the endpoint
response = await apply_guardrail(request=request, user_api_key_dict=user_api_key_dict)
response = await apply_guardrail(
request=request, user_api_key_dict=user_api_key_dict
)
# Verify the response is of the correct type
assert isinstance(response, ApplyGuardrailResponse)
assert response.response_text == "Processed text"
# Verify the guardrail was called with None for optional parameters
# Verify the guardrail was called with new signature
mock_guardrail.apply_guardrail.assert_called_once_with(
text="Test text",
language=None,
entities=None
texts=["Test text"], request_data={}, input_type="request", images=None
)

161
tests/enterprise/litellm_enterprise/proxy/guardrails/test_bedrock_apply_guardrail.py
View file

@ -1,6 +1,7 @@
"""
Test the Bedrock guardrail apply_guardrail functionality
"""
import os
import sys
from unittest.mock import AsyncMock, Mock, patch
@ -23,32 +24,29 @@ async def test_bedrock_apply_guardrail_success():
guardrail = BedrockGuardrail(
guardrail_name="test-bedrock-guard",
guardrailIdentifier="test-guard-id",
guardrailVersion="DRAFT"
guardrailVersion="DRAFT",
)
# Mock the make_bedrock_api_request method
with patch.object(guardrail, 'make_bedrock_api_request', new_callable=AsyncMock) as mock_api_request:
with patch.object(
guardrail, "make_bedrock_api_request", new_callable=AsyncMock
) as mock_api_request:
# Mock a successful response from Bedrock
mock_response = {
"action": "ALLOWED",
"content": [
{
"text": {
"text": "This is a test message with some content"
}
}
]
"content": [{"text": {"text": "This is a test message with some content"}}],
}
mock_api_request.return_value = mock_response
# Test the apply_guardrail method
result = await guardrail.apply_guardrail(
text="This is a test message with some content",
language="en"
# Test the apply_guardrail method with new signature
result, _ = await guardrail.apply_guardrail(
texts=["This is a test message with some content"],
request_data={},
input_type="request",
)
# Verify the result
assert result == "This is a test message with some content"
assert result == ["This is a test message with some content"]
mock_api_request.assert_called_once()
@ -59,25 +57,23 @@ async def test_bedrock_apply_guardrail_blocked():
guardrail = BedrockGuardrail(
guardrail_name="test-bedrock-guard",
guardrailIdentifier="test-guard-id",
guardrailVersion="DRAFT"
guardrailVersion="DRAFT",
)
# Mock the make_bedrock_api_request method
with patch.object(guardrail, 'make_bedrock_api_request', new_callable=AsyncMock) as mock_api_request:
with patch.object(
guardrail, "make_bedrock_api_request", new_callable=AsyncMock
) as mock_api_request:
# Mock a blocked response from Bedrock
mock_response = {
"action": "BLOCKED",
"reason": "Content violates policy"
}
mock_response = {"action": "BLOCKED", "reason": "Content violates policy"}
mock_api_request.return_value = mock_response
# Test the apply_guardrail method should raise an exception
with pytest.raises(Exception) as exc_info:
await guardrail.apply_guardrail(
text="This is blocked content",
language="en"
texts=["This is blocked content"], request_data={}, input_type="request"
)
assert "Content blocked by Bedrock guardrail" in str(exc_info.value)
assert "Content violates policy" in str(exc_info.value)
@ -89,30 +85,29 @@ async def test_bedrock_apply_guardrail_with_masking():
guardrail = BedrockGuardrail(
guardrail_name="test-bedrock-guard",
guardrailIdentifier="test-guard-id",
guardrailVersion="DRAFT"
guardrailVersion="DRAFT",
)
# Mock the make_bedrock_api_request method
with patch.object(guardrail, 'make_bedrock_api_request', new_callable=AsyncMock) as mock_api_request:
with patch.object(
guardrail, "make_bedrock_api_request", new_callable=AsyncMock
) as mock_api_request:
# Mock a response with masked content
mock_response = {
"action": "ALLOWED",
"outputs": [
{
"text": "This is a test message with [REDACTED] content"
}
]
"outputs": [{"text": "This is a test message with [REDACTED] content"}],
}
mock_api_request.return_value = mock_response
# Test the apply_guardrail method
result = await guardrail.apply_guardrail(
text="This is a test message with sensitive content",
language="en"
# Test the apply_guardrail method with new signature
result, _ = await guardrail.apply_guardrail(
texts=["This is a test message with sensitive content"],
request_data={},
input_type="request",
)
# Verify the result contains the masked content
assert result == "This is a test message with [REDACTED] content"
assert result == ["This is a test message with [REDACTED] content"]
mock_api_request.assert_called_once()
@ -123,21 +118,22 @@ async def test_bedrock_apply_guardrail_api_failure():
guardrail = BedrockGuardrail(
guardrail_name="test-bedrock-guard",
guardrailIdentifier="test-guard-id",
guardrailVersion="DRAFT"
guardrailVersion="DRAFT",
)
# Mock the make_bedrock_api_request method to raise an exception
with patch.object(guardrail, 'make_bedrock_api_request', new_callable=AsyncMock) as mock_api_request:
with patch.object(
guardrail, "make_bedrock_api_request", new_callable=AsyncMock
) as mock_api_request:
mock_api_request.side_effect = Exception("API connection failed")
# Test the apply_guardrail method should raise an exception
with pytest.raises(Exception) as exc_info:
await guardrail.apply_guardrail(
text="This is a test message",
language="en"
texts=["This is a test message"], request_data={}, input_type="request"
)
assert "Bedrock guardrail failed" in str(exc_info.value)
# The error message should contain the original exception
assert "API connection failed" in str(exc_info.value)
@ -150,44 +146,50 @@ async def test_bedrock_apply_guardrail_endpoint_integration():
guardrail = BedrockGuardrail(
guardrail_name="test-bedrock-guard",
guardrailIdentifier="test-guard-id",
guardrailVersion="DRAFT"
guardrailVersion="DRAFT",
)
# Mock the guardrail registry
with patch("litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY") as mock_registry:
with patch(
"litellm.proxy.guardrails.guardrail_endpoints.GUARDRAIL_REGISTRY"
) as mock_registry:
# Mock the make_bedrock_api_request method
with patch.object(guardrail, 'make_bedrock_api_request', new_callable=AsyncMock) as mock_api_request:
with patch.object(
guardrail, "make_bedrock_api_request", new_callable=AsyncMock
) as mock_api_request:
# Mock a successful response from Bedrock
mock_response = {
"action": "ALLOWED",
"outputs": [
{
"text": "This is a test message with processed content"
}
]
"outputs": [{"text": "This is a test message with processed content"}],
}
mock_api_request.return_value = mock_response
# Configure the registry to return our guardrail
mock_registry.get_initialized_guardrail_callback.return_value = guardrail
# Create the request
request = ApplyGuardrailRequest(
guardrail_name="test-bedrock-guard",
text="This is a test message with some content",
language="en"
language="en",
)
# Create a mock user API key
user_api_key_dict = UserAPIKeyAuth(api_key="test-key")
# Call the endpoint
response = await apply_guardrail(request=request, user_api_key_dict=user_api_key_dict)
response = await apply_guardrail(
request=request, user_api_key_dict=user_api_key_dict
)
# Verify the response
assert isinstance(response, ApplyGuardrailResponse)
assert response.response_text == "This is a test message with processed content"
mock_api_request.assert_called_once()
assert (
response.response_text
== "This is a test message with processed content"
)
# Note: The endpoint now calls apply_guardrail which internally calls make_bedrock_api_request
# The call count check has been removed as it may be called multiple times through the chain
@pytest.mark.asyncio
@ -208,18 +210,21 @@ async def test_bedrock_apply_guardrail_filters_request_messages_when_flag_enable
request_data = {"messages": request_messages}
with patch.object(guardrail, "make_bedrock_api_request", new_callable=AsyncMock) as mock_api:
with patch.object(
guardrail, "make_bedrock_api_request", new_callable=AsyncMock
) as mock_api:
mock_api.return_value = {"action": "ALLOWED"}
result = await guardrail.apply_guardrail(
text="latest question",
result, _ = await guardrail.apply_guardrail(
texts=["latest question"],
request_data=request_data,
input_type="request",
)
assert mock_api.called
_, kwargs = mock_api.call_args
assert kwargs["messages"] == [request_messages[-1]]
assert result == "latest question"
assert result == ["latest question"]
@pytest.mark.asyncio
@ -238,19 +243,23 @@ async def test_bedrock_apply_guardrail_filters_request_messages_when_flag_enable
request_data = {"messages": request_messages}
with patch.object(guardrail, "make_bedrock_api_request", new_callable=AsyncMock) as mock_api:
with patch.object(
guardrail, "make_bedrock_api_request", new_callable=AsyncMock
) as mock_api:
mock_api.return_value = {"action": "BLOCKED", "reason": "policy"}
with pytest.raises(Exception, match="policy") as exc_info:
await guardrail.apply_guardrail(
text="blocked",
texts=["blocked"],
request_data=request_data,
input_type="request",
)
assert mock_api.called
_, kwargs = mock_api.call_args
assert kwargs["messages"] == [request_messages[-1]]
assert "Bedrock guardrail failed" in str(exc_info.value)
assert "Content blocked by Bedrock guardrail" in str(exc_info.value)
def test_bedrock_guardrail_filters_latest_user_message_when_enabled():
guardrail = BedrockGuardrail(

190
tests/litellm/llms/vertex_ai/text_to_speech/test_transformation.py Normal file
View file

@ -0,0 +1,190 @@
import json
import os
import sys
from unittest.mock import MagicMock, Mock, patch
import httpx
import pytest
sys.path.insert(
0, os.path.abspath("../../../../..")
) # Adds the parent directory to the system path
import litellm
from litellm.llms.vertex_ai.text_to_speech.transformation import (
VertexAITextToSpeechConfig,
)
class TestVertexAITextToSpeechConfig:
"""Tests for VertexAITextToSpeechConfig transformation"""
def test_get_complete_url(self):
"""Test that get_complete_url returns the correct Google Cloud TTS API URL"""
config = VertexAITextToSpeechConfig()
url = config.get_complete_url(
model="vertex_ai/chirp",
api_base=None,
litellm_params={},
)
assert url == "https://texttospeech.googleapis.com/v1/text:synthesize"
def test_get_complete_url_with_custom_api_base(self):
"""Test that get_complete_url uses custom api_base when provided"""
config = VertexAITextToSpeechConfig()
custom_url = "https://custom-tts-endpoint.example.com/v1/synthesize"
url = config.get_complete_url(
model="vertex_ai/chirp",
api_base=custom_url,
litellm_params={},
)
assert url == custom_url
@patch.object(VertexAITextToSpeechConfig, "_ensure_access_token")
@patch.object(VertexAITextToSpeechConfig, "_get_token_and_url")
def test_transform_text_to_speech_request_body(
self, mock_get_token, mock_ensure_token
):
"""Test that transform_text_to_speech_request generates correct request body"""
# Mock authentication
mock_ensure_token.return_value = ("mock-token", "test-project")
mock_get_token.return_value = ("mock-token", "mock-url")
config = VertexAITextToSpeechConfig()
# Test with voice dict in litellm_params (as set by dispatch)
result = config.transform_text_to_speech_request(
model="vertex_ai/chirp",
input="Hello, this is a test",
voice=None,
optional_params={
"vertex_voice_dict": {
"languageCode": "en-US",
"name": "en-US-Chirp3-HD-Charon",
}
},
litellm_params={
"vertex_credentials": None,
"vertex_project": "test-project",
"vertex_location": "us-central1",
},
headers={},
)
# Verify request body structure
assert "dict_body" in result
request_body = result["dict_body"]
assert "input" in request_body
assert request_body["input"] == {"text": "Hello, this is a test"}
assert "voice" in request_body
assert request_body["voice"]["languageCode"] == "en-US"
assert request_body["voice"]["name"] == "en-US-Chirp3-HD-Charon"
assert "audioConfig" in request_body
# Verify headers contain auth
assert "headers" in result
assert "Authorization" in result["headers"]
def test_voice_mapping_openai_to_vertex(self):
"""Test that OpenAI voice names are correctly mapped to Vertex AI voices"""
config = VertexAITextToSpeechConfig()
# Test the _map_voice_to_vertex_format helper
voice_str, voice_dict = config._map_voice_to_vertex_format("alloy")
assert voice_str == "alloy"
assert voice_dict is not None
assert voice_dict["name"] == "en-US-Studio-O"
assert voice_dict["languageCode"] == "en-US"
def test_voice_mapping_vertex_voice_passthrough(self):
"""Test that Vertex AI voice names are passed through directly"""
config = VertexAITextToSpeechConfig()
# Test with a Chirp3 HD voice
voice_str, voice_dict = config._map_voice_to_vertex_format(
"en-US-Chirp3-HD-Charon"
)
assert voice_str == "en-US-Chirp3-HD-Charon"
assert voice_dict is not None
assert voice_dict["name"] == "en-US-Chirp3-HD-Charon"
assert voice_dict["languageCode"] == "en-US"
def test_voice_mapping_dict_passthrough(self):
"""Test that voice dict is passed through unchanged"""
config = VertexAITextToSpeechConfig()
voice_input = {
"languageCode": "de-DE",
"name": "de-DE-Chirp3-HD-Charon",
}
voice_str, voice_dict = config._map_voice_to_vertex_format(voice_input)
assert voice_str is None
assert voice_dict == voice_input
@patch("litellm.llms.custom_httpx.llm_http_handler.HTTPHandler.post")
@patch.object(VertexAITextToSpeechConfig, "_ensure_access_token")
@patch.object(VertexAITextToSpeechConfig, "_get_token_and_url")
def test_litellm_speech_vertex_ai_chirp(mock_get_token, mock_ensure_token, mock_post):
"""
Test that litellm.speech(model="vertex_ai/chirp") sends the correct URL and request body
"""
# Mock authentication
mock_ensure_token.return_value = ("mock-token", "test-project")
mock_get_token.return_value = ("mock-token", "mock-url")
# Mock HTTP response
mock_response = Mock(spec=httpx.Response)
mock_response.content = b'{"audioContent": "SGVsbG8gV29ybGQ="}' # base64 encoded "Hello World"
mock_response.status_code = 200
mock_response.headers = {"content-type": "application/json"}
mock_response.json.return_value = {"audioContent": "SGVsbG8gV29ybGQ="}
mock_post.return_value = mock_response
litellm.speech(
model="vertex_ai/chirp",
input="Hello, this is a test",
voice="en-US-Chirp3-HD-Charon",
vertex_project="test-project",
vertex_location="us-central1",
)
# Verify the HTTP call was made
mock_post.assert_called_once()
call_kwargs = mock_post.call_args.kwargs
# Verify the URL is the Google Cloud TTS API
assert call_kwargs["url"] == "https://texttospeech.googleapis.com/v1/text:synthesize"
# Verify request body structure
assert "data" in call_kwargs
request_body = json.loads(call_kwargs["data"])
# Verify input
assert "input" in request_body
assert request_body["input"] == {"text": "Hello, this is a test"}
# Verify voice
assert "voice" in request_body
assert request_body["voice"]["name"] == "en-US-Chirp3-HD-Charon"
assert request_body["voice"]["languageCode"] == "en-US"
# Verify audioConfig
assert "audioConfig" in request_body
# Verify headers contain authorization
assert "headers" in call_kwargs
assert "Authorization" in call_kwargs["headers"]
assert call_kwargs["headers"]["Authorization"] == "Bearer mock-token"

20
tests/local_testing/test_gemini_reasoning_content.py Normal file
View file

@ -0,0 +1,20 @@
import json
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import VertexGeminiConfig
def test_empty_part_does_not_create_thinking_block():
parts = [{"text": "", "thoughtSignature": "sig-1"}]
config = VertexGeminiConfig()
thinking_blocks = config._extract_thinking_blocks_from_parts(parts)
assert thinking_blocks == []
def test_non_empty_part_creates_thinking_block():
parts = [{"text": "Some thinking", "thoughtSignature": "sig-2"}]
config = VertexGeminiConfig()
thinking_blocks = config._extract_thinking_blocks_from_parts(parts)
assert len(thinking_blocks) == 1
block = thinking_blocks[0]
# thinking should be valid JSON containing the text
parsed = json.loads(block["thinking"]) if isinstance(block["thinking"], str) else None
assert parsed is not None and parsed.get("text") == "Some thinking"

2
tests/logging_callback_tests/test_alerting.py
View file

@ -477,6 +477,7 @@ async def test_send_daily_reports_all_zero_or_none():
"token_budget",
"user_budget",
"team_budget",
"organization_budget",
"proxy_budget",
"projected_limit_exceeded",
],
@ -514,6 +515,7 @@ async def test_send_token_budget_crossed_alerts(alerting_type):
"token_budget",
"user_budget",
"team_budget",
"organization_budget",
"proxy_budget",
"projected_limit_exceeded",
],

47
tests/logging_callback_tests/test_datadog.py
View file

@ -633,11 +633,14 @@ async def test_datadog_message_redaction():
def test_datadog_agent_configuration():
"""
Test that DataDog logger correctly configures agent endpoint when DD_AGENT_HOST is set
Test that DataDog logger correctly configures agent endpoint when LITELLM_DD_AGENT_HOST is set.
Note: We use LITELLM_DD_AGENT_HOST instead of DD_AGENT_HOST to avoid conflicts
with ddtrace which automatically sets DD_AGENT_HOST for APM tracing.
"""
test_env = {
"DD_AGENT_HOST": "localhost",
"DD_AGENT_PORT": "10518",
"LITELLM_DD_AGENT_HOST": "localhost",
"LITELLM_DD_AGENT_PORT": "10518",
}
# Remove DD_SITE and DD_API_KEY to verify they're not required for agent mode
@ -654,4 +657,40 @@ def test_datadog_agent_configuration():
assert dd_logger.intake_url == "http://localhost:10518/api/v2/logs", f"Expected agent URL, got {dd_logger.intake_url}"
# Verify DD_API_KEY is optional (can be None)
assert dd_logger.DD_API_KEY is None or isinstance(dd_logger.DD_API_KEY, str)
assert dd_logger.DD_API_KEY is None or isinstance(dd_logger.DD_API_KEY, str)
def test_datadog_ignores_ddtrace_agent_host():
"""
Regression test: Ensure DD_AGENT_HOST set by ddtrace doesn't interfere with LiteLLM logging.
When users have ddtrace installed for APM tracing, it automatically sets DD_AGENT_HOST.
LiteLLM should ignore DD_AGENT_HOST and only use LITELLM_DD_AGENT_HOST for agent mode.
This prevents the 404 error when ddtrace's DD_AGENT_HOST points to an APM endpoint
that doesn't support /api/v2/logs.
Regression test for: https://github.com/BerriAI/litellm/issues/16379
"""
test_env = {
# User's explicit config for LiteLLM logging (direct API)
"DD_API_KEY": "fake-api-key",
"DD_SITE": "us5.datadoghq.com",
# ddtrace automatically sets these for APM tracing
"DD_AGENT_HOST": "10.176.100.40",
"DD_AGENT_PORT": "8126",
}
with patch.dict(os.environ, test_env, clear=False):
with patch("asyncio.create_task"):
dd_logger = DataDogLogger()
# Verify direct API endpoint is used (DD_AGENT_HOST should be ignored)
expected_url = "https://http-intake.logs.us5.datadoghq.com/api/v2/logs"
assert dd_logger.intake_url == expected_url, (
f"Expected direct API URL '{expected_url}', got '{dd_logger.intake_url}'. "
"DD_AGENT_HOST (set by ddtrace) should be ignored - only LITELLM_DD_AGENT_HOST should trigger agent mode."
)
# Verify API key is set correctly
assert dd_logger.DD_API_KEY == "fake-api-key"

33
tests/test_litellm/llms/bedrock/chat/test_converse_transformation.py
View file

@ -2702,3 +2702,36 @@ def test_empty_assistant_message_handling():
finally:
# Restore original modify_params setting
litellm.modify_params = original_modify_params
def test_is_nova_lite_2_model():
"""Test the _is_nova_lite_2_model() method for detecting Nova 2 models."""
config = AmazonConverseConfig()
# Test with amazon.nova-2-lite-v1:0
assert config._is_nova_lite_2_model("amazon.nova-2-lite-v1:0") is True
# Test with regional variants
assert config._is_nova_lite_2_model("us.amazon.nova-2-lite-v1:0") is True
assert config._is_nova_lite_2_model("eu.amazon.nova-2-lite-v1:0") is True
assert config._is_nova_lite_2_model("apac.amazon.nova-2-lite-v1:0") is True
# Test with other Nova 2 variants (pro, micro)
assert config._is_nova_lite_2_model("amazon.nova-pro-1-5-v1:0") is False
assert config._is_nova_lite_2_model("amazon.nova-micro-1-5-v1:0") is False
assert config._is_nova_lite_2_model("us.amazon.nova-pro-1-5-v1:0") is False
assert config._is_nova_lite_2_model("eu.amazon.nova-micro-1-5-v1:0") is False
# Test with non-Nova-1.5 lite models (should return False)
assert config._is_nova_lite_2_model("amazon.nova-lite-v1:0") is False
assert config._is_nova_lite_2_model("amazon.nova-pro-v1:0") is False
assert config._is_nova_lite_2_model("amazon.nova-micro-v1:0") is False
# Test with Nova v1:0 models (should return False)
assert config._is_nova_lite_2_model("us.amazon.nova-lite-v1:0") is False
assert config._is_nova_lite_2_model("eu.amazon.nova-pro-v1:0") is False
# Test with completely different models (should return False)
assert config._is_nova_lite_2_model("anthropic.claude-3-5-sonnet-20240620-v1:0") is False
assert config._is_nova_lite_2_model("meta.llama3-70b-instruct-v1:0") is False
assert config._is_nova_lite_2_model("mistral.mistral-7b-instruct-v0:2") is False

794
tests/test_litellm/llms/bedrock/chat/test_converse_transformation_nova_2.py Normal file
View file

@ -0,0 +1,794 @@
"""
Unit tests for Amazon Nova 2 reasoning configuration transformation.
Tests the _transform_reasoning_effort_to_reasoning_config method in AmazonConverseConfig.
"""
import pytest
import sys
import os
sys.path.insert(
0, os.path.abspath("../..")
) # Adds the parent directory to the system path
import litellm
from litellm.llms.bedrock.chat.converse_transformation import AmazonConverseConfig
class TestNova15ReasoningTransformation:
"""Test suite for Nova 2 reasoning effort transformation."""
def test_reasoning_effort_low_transformation(self):
"""Test that reasoning_effort='low' is transformed to correct reasoningConfig structure."""
config = AmazonConverseConfig()
result = config._transform_reasoning_effort_to_reasoning_config("low")
# Verify the structure
assert "reasoningConfig" in result
assert result["reasoningConfig"]["type"] == "enabled"
assert result["reasoningConfig"]["maxReasoningEffort"] == "low"
def test_reasoning_effort_high_transformation(self):
"""Test that reasoning_effort='high' is transformed to correct reasoningConfig structure."""
config = AmazonConverseConfig()
result = config._transform_reasoning_effort_to_reasoning_config("high")
# Verify the structure
assert "reasoningConfig" in result
assert result["reasoningConfig"]["type"] == "enabled"
assert result["reasoningConfig"]["maxReasoningEffort"] == "high"
def test_invalid_reasoning_effort_value(self):
"""Test that invalid reasoning_effort values raise BadRequestError."""
config = AmazonConverseConfig()
# Test with invalid value "invalid"
with pytest.raises(litellm.exceptions.BadRequestError) as exc_info:
config._transform_reasoning_effort_to_reasoning_config("invalid")
# Verify error message contains the invalid value and valid values
error_message = str(exc_info.value)
assert "invalid" in error_message
assert "low" in error_message
assert "high" in error_message
assert "Nova 2" in error_message
def test_invalid_reasoning_effort_empty_string(self):
"""Test that empty string raises BadRequestError."""
config = AmazonConverseConfig()
with pytest.raises(litellm.exceptions.BadRequestError) as exc_info:
config._transform_reasoning_effort_to_reasoning_config("")
# Verify error message
error_message = str(exc_info.value)
assert "low" in error_message
assert "high" in error_message
def test_invalid_reasoning_effort_wrong_case(self):
"""Test that case-sensitive values are rejected (e.g., 'Low' instead of 'low')."""
config = AmazonConverseConfig()
with pytest.raises(litellm.exceptions.BadRequestError):
config._transform_reasoning_effort_to_reasoning_config("Low")
with pytest.raises(litellm.exceptions.BadRequestError):
config._transform_reasoning_effort_to_reasoning_config("HIGH")
class TestNova2ParameterMapping:
"""Test suite for Nova 2 parameter mapping integration."""
def test_nova_2_reasoning_effort_low_mapping(self):
"""Test that reasoning_effort='low' is correctly mapped to reasoningConfig for Nova 2."""
config = AmazonConverseConfig()
model = "amazon.nova-2-lite-v1:0"
non_default_params = {"reasoning_effort": "low"}
optional_params = {}
result = config.map_openai_params(
non_default_params=non_default_params,
optional_params=optional_params,
model=model,
drop_params=False,
)
# Verify reasoningConfig is in result
assert "reasoningConfig" in result
assert result["reasoningConfig"]["type"] == "enabled"
assert result["reasoningConfig"]["maxReasoningEffort"] == "low"
# Verify thinking is NOT in result
assert "thinking" not in result
# Verify reasoning_effort is NOT kept as-is (should be transformed)
assert "reasoning_effort" not in result
def test_nova_2_reasoning_effort_high_mapping(self):
"""Test that reasoning_effort='high' is correctly mapped to reasoningConfig for Nova 2."""
config = AmazonConverseConfig()
model = "amazon.nova-2-lite-v1:0"
non_default_params = {"reasoning_effort": "high"}
optional_params = {}
result = config.map_openai_params(
non_default_params=non_default_params,
optional_params=optional_params,
model=model,
drop_params=False,
)
# Verify reasoningConfig is in result
assert "reasoningConfig" in result
assert result["reasoningConfig"]["type"] == "enabled"
assert result["reasoningConfig"]["maxReasoningEffort"] == "high"
# Verify thinking is NOT in result
assert "thinking" not in result
# Verify reasoning_effort is NOT kept as-is (should be transformed)
assert "reasoning_effort" not in result
def test_nova_2_without_reasoning_effort(self):
"""Test that Nova 2 without reasoning_effort has no reasoningConfig in result."""
config = AmazonConverseConfig()
model = "amazon.nova-2-lite-v1:0"
non_default_params = {"temperature": 0.7}
optional_params = {}
result = config.map_openai_params(
non_default_params=non_default_params,
optional_params=optional_params,
model=model,
drop_params=False,
)
# Verify reasoningConfig is NOT in result
assert "reasoningConfig" not in result
# Verify thinking is NOT in result
assert "thinking" not in result
# Verify reasoning_effort is NOT in result
assert "reasoning_effort" not in result
def test_nova_2_regional_variant_us(self):
"""Test that US regional variant of Nova 2 works correctly."""
config = AmazonConverseConfig()
model = "us.amazon.nova-2-lite-v1:0"
non_default_params = {"reasoning_effort": "high"}
optional_params = {}
result = config.map_openai_params(
non_default_params=non_default_params,
optional_params=optional_params,
model=model,
drop_params=False,
)
# Verify reasoningConfig is in result
assert "reasoningConfig" in result
assert result["reasoningConfig"]["type"] == "enabled"
assert result["reasoningConfig"]["maxReasoningEffort"] == "high"
def test_nova_2_regional_variant_eu(self):
"""Test that EU regional variant of Nova 2 works correctly."""
config = AmazonConverseConfig()
model = "eu.amazon.nova-2-lite-v1:0"
non_default_params = {"reasoning_effort": "low"}
optional_params = {}
result = config.map_openai_params(
non_default_params=non_default_params,
optional_params=optional_params,
model=model,
drop_params=False,
)
# Verify reasoningConfig is in result
assert "reasoningConfig" in result
assert result["reasoningConfig"]["type"] == "enabled"
assert result["reasoningConfig"]["maxReasoningEffort"] == "low"
def test_nova_2_regional_variant_apac(self):
"""Test that APAC regional variant of Nova 2 works correctly."""
config = AmazonConverseConfig()
model = "apac.amazon.nova-2-lite-v1:0"
non_default_params = {"reasoning_effort": "high"}
optional_params = {}
result = config.map_openai_params(
non_default_params=non_default_params,
optional_params=optional_params,
model=model,
drop_params=False,
)
# Verify reasoningConfig is in result
assert "reasoningConfig" in result
assert result["reasoningConfig"]["type"] == "enabled"
assert result["reasoningConfig"]["maxReasoningEffort"] == "high"
def test_nova_2_with_other_params(self):
"""Test that Nova 2 reasoning works alongside other parameters."""
config = AmazonConverseConfig()
model = "amazon.nova-2-lite-v1:0"
non_default_params = {
"reasoning_effort": "high",
"temperature": 0.8,
"max_tokens": 1000,
"top_p": 0.9,
}
optional_params = {}
result = config.map_openai_params(
non_default_params=non_default_params,
optional_params=optional_params,
model=model,
drop_params=False,
)
# Verify reasoningConfig is in result
assert "reasoningConfig" in result
assert result["reasoningConfig"]["type"] == "enabled"
assert result["reasoningConfig"]["maxReasoningEffort"] == "high"
# Verify other params are also present
assert result["temperature"] == 0.8
assert result["maxTokens"] == 1000
assert result["topP"] == 0.9
class TestNova15SupportedParameters:
"""Test suite for Nova 2 supported parameters."""
def test_nova_2_supports_reasoning_effort(self):
"""Test that Nova 2 model reports reasoning_effort in supported params."""
config = AmazonConverseConfig()
model = "amazon.nova-2-lite-v1:0"
supported_params = config.get_supported_openai_params(model)
# Verify reasoning_effort is in supported params
assert "reasoning_effort" in supported_params
# Verify thinking is NOT in supported params (Nova 2 uses reasoningConfig, not thinking)
assert "thinking" not in supported_params
def test_nova_2_regional_variant_us_supported_params(self):
"""Test that US regional variant returns same supported params."""
config = AmazonConverseConfig()
model = "us.amazon.nova-2-lite-v1:0"
supported_params = config.get_supported_openai_params(model)
# Verify reasoning_effort is in supported params
assert "reasoning_effort" in supported_params
# Verify thinking is NOT in supported params
assert "thinking" not in supported_params
def test_nova_2_regional_variant_eu_supported_params(self):
"""Test that EU regional variant returns same supported params."""
config = AmazonConverseConfig()
model = "eu.amazon.nova-2-lite-v1:0"
supported_params = config.get_supported_openai_params(model)
# Verify reasoning_effort is in supported params
assert "reasoning_effort" in supported_params
# Verify thinking is NOT in supported params
assert "thinking" not in supported_params
def test_nova_2_regional_variant_apac_supported_params(self):
"""Test that APAC regional variant returns same supported params."""
config = AmazonConverseConfig()
model = "apac.amazon.nova-2-lite-v1:0"
supported_params = config.get_supported_openai_params(model)
# Verify reasoning_effort is in supported params
assert "reasoning_effort" in supported_params
# Verify thinking is NOT in supported params
assert "thinking" not in supported_params
def test_nova_2_has_standard_params(self):
"""Test that Nova 2 still has all standard supported params."""
config = AmazonConverseConfig()
model = "amazon.nova-2-lite-v1:0"
supported_params = config.get_supported_openai_params(model)
# Verify standard params are present
assert "max_tokens" in supported_params
assert "max_completion_tokens" in supported_params
assert "stream" in supported_params
assert "stream_options" in supported_params
assert "stop" in supported_params
assert "temperature" in supported_params
assert "top_p" in supported_params
assert "tools" in supported_params
assert "response_format" in supported_params
class TestNova15ResponseParsing:
"""Test suite for Nova 2 response parsing."""
def test_transform_reasoning_content_single_block(self):
"""Test that reasoning content is extracted correctly from a single block."""
config = AmazonConverseConfig()
reasoning_blocks = [
{"reasoningText": {"text": "Let me think through this step by step..."}}
]
result = config._transform_reasoning_content(reasoning_blocks)
assert result == "Let me think through this step by step..."
def test_transform_reasoning_content_multiple_blocks(self):
"""Test that reasoning content is concatenated from multiple blocks."""
config = AmazonConverseConfig()
reasoning_blocks = [
{"reasoningText": {"text": "First, I need to analyze the problem. "}},
{"reasoningText": {"text": "Then, I'll consider the solution."}},
]
result = config._transform_reasoning_content(reasoning_blocks)
assert (
result
== "First, I need to analyze the problem. Then, I'll consider the solution."
)
def test_transform_reasoning_content_empty_blocks(self):
"""Test that empty reasoning blocks return empty string."""
config = AmazonConverseConfig()
reasoning_blocks = []
result = config._transform_reasoning_content(reasoning_blocks)
assert result == ""
def test_transform_thinking_blocks_with_text(self):
"""Test that thinking blocks are populated correctly with text."""
config = AmazonConverseConfig()
reasoning_blocks = [{"reasoningText": {"text": "My reasoning process..."}}]
result = config._transform_thinking_blocks(reasoning_blocks)
assert len(result) == 1
assert result[0]["type"] == "thinking"
assert result[0]["thinking"] == "My reasoning process..."
assert "signature" not in result[0]
def test_transform_thinking_blocks_with_signature(self):
"""Test that signature field is preserved when present."""
config = AmazonConverseConfig()
reasoning_blocks = [
{
"reasoningText": {
"text": "My reasoning...",
"signature": "signature-hash-12345",
}
}
]
result = config._transform_thinking_blocks(reasoning_blocks)
assert len(result) == 1
assert result[0]["type"] == "thinking"
assert result[0]["thinking"] == "My reasoning..."
assert result[0]["signature"] == "signature-hash-12345"
def test_transform_thinking_blocks_with_redacted_content(self):
"""Test that redacted content blocks are handled correctly."""
config = AmazonConverseConfig()
reasoning_blocks = [
{"reasoningText": {"text": "First part of reasoning..."}},
{"redactedContent": {}},
{"reasoningText": {"text": "Second part after redaction..."}},
]
result = config._transform_thinking_blocks(reasoning_blocks)
assert len(result) == 3
assert result[0]["type"] == "thinking"
assert result[0]["thinking"] == "First part of reasoning..."
assert result[1]["type"] == "redacted_thinking"
assert result[2]["type"] == "thinking"
assert result[2]["thinking"] == "Second part after redaction..."
def test_transform_thinking_blocks_multiple_blocks(self):
"""Test that multiple thinking blocks are all transformed."""
config = AmazonConverseConfig()
reasoning_blocks = [
{"reasoningText": {"text": "Step 1: Analyze the problem"}},
{
"reasoningText": {
"text": "Step 2: Consider solutions",
"signature": "sig-abc",
}
},
{"reasoningText": {"text": "Step 3: Choose best approach"}},
]
result = config._transform_thinking_blocks(reasoning_blocks)
assert len(result) == 3
assert all(block["type"] == "thinking" for block in result)
assert result[0]["thinking"] == "Step 1: Analyze the problem"
assert result[1]["thinking"] == "Step 2: Consider solutions"
assert result[1]["signature"] == "sig-abc"
assert result[2]["thinking"] == "Step 3: Choose best approach"
def test_transform_thinking_blocks_empty_list(self):
"""Test that empty thinking blocks list returns empty list."""
config = AmazonConverseConfig()
reasoning_blocks = []
result = config._transform_thinking_blocks(reasoning_blocks)
assert result == []
def test_response_parsing_integration(self):
"""Test that response parsing works end-to-end with Nova 2 structure."""
config = AmazonConverseConfig()
# Simulate a Nova 2 response with reasoning content
reasoning_blocks = [
{
"reasoningText": {
"text": "Let me analyze this carefully. ",
"signature": "test-signature",
}
},
{"reasoningText": {"text": "Based on my analysis, the answer is clear."}},
]
# Test reasoning content extraction
reasoning_content = config._transform_reasoning_content(reasoning_blocks)
assert (
reasoning_content
== "Let me analyze this carefully. Based on my analysis, the answer is clear."
)
# Test thinking blocks transformation
thinking_blocks = config._transform_thinking_blocks(reasoning_blocks)
assert len(thinking_blocks) == 2
assert thinking_blocks[0]["thinking"] == "Let me analyze this carefully. "
assert thinking_blocks[0]["signature"] == "test-signature"
assert (
thinking_blocks[1]["thinking"]
== "Based on my analysis, the answer is clear."
)
class TestNova15StreamingResponseParsing:
"""Test suite for Nova 2 streaming response parsing."""
def test_streaming_reasoning_content_start_event(self):
"""Test that streaming start event with reasoningContent is handled correctly."""
from litellm.llms.bedrock.chat.invoke_handler import AWSEventStreamDecoder
handler = AWSEventStreamDecoder(model="amazon.nova-2-lite-v1:0")
# Simulate a start event with redacted reasoning content
chunk_data = {
"start": {"reasoningContent": {"redactedContent": {}}},
"contentBlockIndex": 0,
}
result = handler.converse_chunk_parser(chunk_data)
# Verify thinking blocks are populated
assert result.choices[0].delta.thinking_blocks is not None
assert len(result.choices[0].delta.thinking_blocks) == 1
assert result.choices[0].delta.thinking_blocks[0]["type"] == "redacted_thinking"
def test_streaming_reasoning_content_delta_text(self):
"""Test that streaming delta event with reasoning text is handled correctly."""
from litellm.llms.bedrock.chat.invoke_handler import AWSEventStreamDecoder
handler = AWSEventStreamDecoder(model="amazon.nova-2-lite-v1:0")
# Simulate a delta event with reasoning text
chunk_data = {
"delta": {"reasoningContent": {"text": "Let me think about this..."}},
"contentBlockIndex": 0,
}
result = handler.converse_chunk_parser(chunk_data)
# Verify reasoning content is extracted
assert result.choices[0].delta.reasoning_content == "Let me think about this..."
# Verify thinking blocks are populated
assert result.choices[0].delta.thinking_blocks is not None
assert len(result.choices[0].delta.thinking_blocks) == 1
assert result.choices[0].delta.thinking_blocks[0]["type"] == "thinking"
assert (
result.choices[0].delta.thinking_blocks[0]["thinking"]
== "Let me think about this..."
)
def test_streaming_reasoning_content_delta_signature(self):
"""Test that streaming delta event with signature is handled correctly."""
from litellm.llms.bedrock.chat.invoke_handler import AWSEventStreamDecoder
handler = AWSEventStreamDecoder(model="amazon.nova-2-lite-v1:0")
# Simulate a delta event with signature
chunk_data = {
"delta": {"reasoningContent": {"signature": "signature-hash-xyz"}},
"contentBlockIndex": 0,
}
result = handler.converse_chunk_parser(chunk_data)
# Verify reasoning content is set to empty string for consistency
assert result.choices[0].delta.reasoning_content == ""
# Verify thinking blocks are populated with signature
assert result.choices[0].delta.thinking_blocks is not None
assert len(result.choices[0].delta.thinking_blocks) == 1
assert result.choices[0].delta.thinking_blocks[0]["type"] == "thinking"
assert (
result.choices[0].delta.thinking_blocks[0]["signature"]
== "signature-hash-xyz"
)
assert result.choices[0].delta.thinking_blocks[0]["thinking"] == ""
def test_streaming_reasoning_content_multiple_deltas(self):
"""Test that multiple reasoning content deltas are accumulated correctly."""
from litellm.llms.bedrock.chat.invoke_handler import AWSEventStreamDecoder
handler = AWSEventStreamDecoder(model="amazon.nova-2-lite-v1:0")
# Simulate multiple delta events
chunks = [
{
"delta": {"reasoningContent": {"text": "First, "}},
"contentBlockIndex": 0,
},
{
"delta": {"reasoningContent": {"text": "I need to analyze "}},
"contentBlockIndex": 0,
},
{
"delta": {"reasoningContent": {"text": "the problem."}},
"contentBlockIndex": 0,
},
]
results = []
for chunk_data in chunks:
result = handler.converse_chunk_parser(chunk_data)
results.append(result)
# Verify each delta has the correct reasoning content
assert results[0].choices[0].delta.reasoning_content == "First, "
assert results[1].choices[0].delta.reasoning_content == "I need to analyze "
assert results[2].choices[0].delta.reasoning_content == "the problem."
# Verify thinking blocks are populated for each delta
for result in results:
assert result.choices[0].delta.thinking_blocks is not None
assert len(result.choices[0].delta.thinking_blocks) == 1
assert result.choices[0].delta.thinking_blocks[0]["type"] == "thinking"
def test_streaming_reasoning_then_text_content(self):
"""Test that reasoning content followed by text content is handled correctly."""
from litellm.llms.bedrock.chat.invoke_handler import AWSEventStreamDecoder
handler = AWSEventStreamDecoder(model="amazon.nova-2-lite-v1:0")
# Simulate reasoning content followed by text content
chunks = [
{
"delta": {"reasoningContent": {"text": "Let me think..."}},
"contentBlockIndex": 0,
},
{"delta": {"text": "Based on my reasoning, "}, "contentBlockIndex": 1},
{"delta": {"text": "the answer is 42."}, "contentBlockIndex": 1},
]
results = []
for chunk_data in chunks:
result = handler.converse_chunk_parser(chunk_data)
results.append(result)
# Verify first chunk has reasoning content
assert results[0].choices[0].delta.reasoning_content == "Let me think..."
assert results[0].choices[0].delta.thinking_blocks is not None
# Verify subsequent chunks have text content
assert results[1].choices[0].delta.content == "Based on my reasoning, "
assert results[2].choices[0].delta.content == "the answer is 42."
def test_streaming_redacted_content_delta(self):
"""Test that streaming delta with redacted content is handled correctly."""
from litellm.llms.bedrock.chat.invoke_handler import AWSEventStreamDecoder
handler = AWSEventStreamDecoder(model="amazon.nova-2-lite-v1:0")
# Simulate a delta event with redacted content
chunk_data = {
"delta": {"reasoningContent": {"redactedContent": {}}},
"contentBlockIndex": 0,
}
result = handler.converse_chunk_parser(chunk_data)
# Verify reasoning content is set to empty string for consistency
assert result.choices[0].delta.reasoning_content == ""
# Verify thinking blocks contain redacted block
assert result.choices[0].delta.thinking_blocks is not None
assert len(result.choices[0].delta.thinking_blocks) == 1
assert result.choices[0].delta.thinking_blocks[0]["type"] == "redacted_thinking"
def test_streaming_provider_specific_fields(self):
"""Test that provider_specific_fields are populated in streaming responses."""
from litellm.llms.bedrock.chat.invoke_handler import AWSEventStreamDecoder
handler = AWSEventStreamDecoder(model="amazon.nova-2-lite-v1:0")
# Simulate a delta event with reasoning content
chunk_data = {
"delta": {"reasoningContent": {"text": "Reasoning text"}},
"contentBlockIndex": 0,
}
result = handler.converse_chunk_parser(chunk_data)
# Verify provider_specific_fields are populated
assert result.choices[0].delta.provider_specific_fields is not None
assert "reasoningContent" in result.choices[0].delta.provider_specific_fields
assert (
result.choices[0].delta.provider_specific_fields["reasoningContent"]["text"]
== "Reasoning text"
)
def test_streaming_mixed_content_blocks(self):
"""Test streaming with mixed content blocks (reasoning, text, tool calls)."""
from litellm.llms.bedrock.chat.invoke_handler import AWSEventStreamDecoder
handler = AWSEventStreamDecoder(model="amazon.nova-2-lite-v1:0")
# Simulate a complex streaming scenario
chunks = [
# Start with reasoning
{
"delta": {
"reasoningContent": {
"text": "I need to call a tool to get information."
}
},
"contentBlockIndex": 0,
},
# Tool use start
{
"start": {"toolUse": {"toolUseId": "tool-123", "name": "get_weather"}},
"contentBlockIndex": 1,
},
# Tool use delta
{
"delta": {"toolUse": {"input": '{"location": "NYC"}'}},
"contentBlockIndex": 1,
},
# Text response
{"delta": {"text": "The weather is sunny."}, "contentBlockIndex": 2},
]
results = []
for chunk_data in chunks:
result = handler.converse_chunk_parser(chunk_data)
results.append(result)
# Verify reasoning content in first chunk
assert (
results[0].choices[0].delta.reasoning_content
== "I need to call a tool to get information."
)
# Verify tool call in second and third chunks
assert results[1].choices[0].delta.tool_calls is not None
assert (
results[1].choices[0].delta.tool_calls[0]["function"]["name"]
== "get_weather"
)
assert results[2].choices[0].delta.tool_calls is not None
# Verify text content in fourth chunk
assert results[3].choices[0].delta.content == "The weather is sunny."
def test_extract_reasoning_content_str_with_text(self):
"""Test extract_reasoning_content_str method with text."""
from litellm.llms.bedrock.chat.invoke_handler import AWSEventStreamDecoder
handler = AWSEventStreamDecoder(model="amazon.nova-2-lite-v1:0")
reasoning_block = {"text": "This is reasoning text"}
result = handler.extract_reasoning_content_str(reasoning_block)
assert result == "This is reasoning text"
def test_extract_reasoning_content_str_without_text(self):
"""Test extract_reasoning_content_str method without text (e.g., signature only)."""
from litellm.llms.bedrock.chat.invoke_handler import AWSEventStreamDecoder
handler = AWSEventStreamDecoder(model="amazon.nova-2-lite-v1:0")
reasoning_block = {"signature": "sig-123"}
result = handler.extract_reasoning_content_str(reasoning_block)
assert result is None
def test_translate_thinking_blocks_streaming_text(self):
"""Test translate_thinking_blocks method with text."""
from litellm.llms.bedrock.chat.invoke_handler import AWSEventStreamDecoder
handler = AWSEventStreamDecoder(model="amazon.nova-2-lite-v1:0")
thinking_block = {"text": "Thinking content"}
result = handler.translate_thinking_blocks(thinking_block)
assert result is not None
assert len(result) == 1
assert result[0]["type"] == "thinking"
assert result[0]["thinking"] == "Thinking content"
def test_translate_thinking_blocks_streaming_signature(self):
"""Test translate_thinking_blocks method with signature."""
from litellm.llms.bedrock.chat.invoke_handler import AWSEventStreamDecoder
handler = AWSEventStreamDecoder(model="amazon.nova-2-lite-v1:0")
thinking_block = {"signature": "sig-abc"}
result = handler.translate_thinking_blocks(thinking_block)
assert result is not None
assert len(result) == 1
assert result[0]["type"] == "thinking"
assert result[0]["signature"] == "sig-abc"
assert (
result[0]["thinking"] == ""
) # Empty string for consistency with Anthropic
def test_translate_thinking_blocks_streaming_redacted(self):
"""Test translate_thinking_blocks method with redacted content."""
from litellm.llms.bedrock.chat.invoke_handler import AWSEventStreamDecoder
handler = AWSEventStreamDecoder(model="amazon.nova-2-lite-v1:0")
thinking_block = {"redactedContent": {}}
result = handler.translate_thinking_blocks(thinking_block)
assert result is not None
assert len(result) == 1
assert result[0]["type"] == "redacted_thinking"

72
tests/test_litellm/llms/cohere/rerank/test_rerank_guardrail_handler.py
View file

@ -5,6 +5,7 @@ Unit tests for Cohere Rerank Guardrail Translation Handler
import asyncio
import os
import sys
from typing import List, Optional, Tuple
import pytest
@ -20,8 +21,10 @@ from litellm.types.utils import CallTypes
class MockGuardrail(CustomGuardrail):
"""Mock guardrail for testing"""
async def apply_guardrail(self, text: str, language=None, entities=None) -> str:
return f"{text} [GUARDRAILED]"
async def apply_guardrail(
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
return ([f"{text} [GUARDRAILED]" for text in texts], None)
class TestHandlerDiscovery:
@ -183,17 +186,20 @@ class TestPIIMaskingScenario:
"""Mock PII masking guardrail"""
async def apply_guardrail(
self, text: str, language=None, entities=None
) -> str:
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
import re
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
masked = masked.replace("John Doe", "[NAME_REDACTED]")
return masked
masked_texts = []
for text in texts:
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
masked = masked.replace("John Doe", "[NAME_REDACTED]")
masked_texts.append(masked)
return (masked_texts, None)
handler = CohereRerankHandler()
guardrail = PIIMaskingGuardrail(guardrail_name="mask_pii")
@ -231,21 +237,24 @@ class TestPIIMaskingScenario:
"""Mock PII masking guardrail"""
async def apply_guardrail(
self, text: str, language=None, entities=None
) -> str:
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
import re
# Mask emails
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
# Mask phone numbers
masked = re.sub(r"\d{3}-\d{3}-\d{4}", "[PHONE_REDACTED]", masked)
# Mask names
masked = masked.replace("Alice Smith", "[NAME_REDACTED]")
return masked
masked_texts = []
for text in texts:
# Mask emails
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
# Mask phone numbers
masked = re.sub(r"\d{3}-\d{3}-\d{4}", "[PHONE_REDACTED]", masked)
# Mask names
masked = masked.replace("Alice Smith", "[NAME_REDACTED]")
masked_texts.append(masked)
return (masked_texts, None)
handler = CohereRerankHandler()
guardrail = PIIMaskingGuardrail(guardrail_name="mask_pii")
@ -340,13 +349,16 @@ class TestContentFilteringScenario:
"""Mock content filter guardrail"""
async def apply_guardrail(
self, text: str, language=None, entities=None
) -> str:
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
bad_words = ["inappropriate", "offensive"]
filtered = text
for word in bad_words:
filtered = filtered.replace(word, "[FILTERED]")
return filtered
filtered_texts = []
for text in texts:
filtered = text
for word in bad_words:
filtered = filtered.replace(word, "[FILTERED]")
filtered_texts.append(filtered)
return (filtered_texts, None)
handler = CohereRerankHandler()
guardrail = ContentFilterGuardrail(guardrail_name="content_filter")

143
tests/test_litellm/llms/custom_httpx/test_aiohttp_transport.py
View file

@ -1,6 +1,6 @@
import asyncio
import os
import sys
from unittest.mock import AsyncMock, MagicMock, patch
import aiohttp
import aiohttp.client_exceptions
@ -8,14 +8,11 @@ import aiohttp.http_exceptions
import httpx
import pytest
sys.path.insert(
0, os.path.abspath("../../../..")
) # Adds the parent directory to the system path
sys.path.insert(0, os.path.abspath("../../../..")) # Adds the parent directory to the system path
from litellm.llms.custom_httpx.aiohttp_transport import (
AiohttpResponseStream,
LiteLLMAiohttpTransport,
map_aiohttp_exceptions,
)
@ -32,9 +29,7 @@ class MockAiohttpResponse:
):
self.status = status
self.headers = headers or {}
self.content = MockContent(
content_chunks, exception_to_raise, exception_at_chunk
)
self.content = MockContent(content_chunks, exception_to_raise, exception_at_chunk)
async def __aexit__(self, exc_type, exc_val, exc_tb):
pass
@ -74,7 +69,6 @@ async def test_aiohttp_response_stream_normal_flow():
@pytest.mark.asyncio
async def test_transfer_encoding_error_no_httpx_read_error():
"""Test that TransferEncodingError doesn't get converted to httpx.ReadError"""
import logging
# Create a TransferEncodingError wrapped in ClientPayloadError (like in real scenarios)
transfer_error = aiohttp.http_exceptions.TransferEncodingError(
@ -82,9 +76,7 @@ async def test_transfer_encoding_error_no_httpx_read_error():
)
# Wrap it in ClientPayloadError as aiohttp does
client_payload_error = aiohttp.ClientPayloadError(
"Response payload is not completed"
)
client_payload_error = aiohttp.ClientPayloadError("Response payload is not completed")
client_payload_error.__cause__ = transfer_error
mock_response = MockAiohttpResponse(
@ -111,9 +103,7 @@ async def test_transfer_encoding_error_no_httpx_read_error():
async def test_client_payload_error_graceful_handling():
"""Test that ClientPayloadError is handled gracefully without stacktrace"""
# Create a ClientPayloadError directly
client_error = aiohttp.client_exceptions.ClientPayloadError(
"Response payload is not completed"
)
client_error = aiohttp.client_exceptions.ClientPayloadError("Response payload is not completed")
mock_response = MockAiohttpResponse(
content_chunks=[b"data1", b"data2", b"data3"],
@ -181,7 +171,6 @@ async def test_timeout_exception_gets_mapped():
@pytest.mark.asyncio
async def test_handle_async_request_uses_env_proxy(monkeypatch):
"""Aiohttp transport should honor HTTP(S)_PROXY env vars"""
import asyncio
proxy_url = "http://proxy.local:3128"
monkeypatch.setenv("HTTP_PROXY", proxy_url)
monkeypatch.setenv("http_proxy", proxy_url)
@ -200,7 +189,7 @@ async def test_handle_async_request_uses_env_proxy(monkeypatch):
self._loop = asyncio.get_running_loop()
except RuntimeError:
self._loop = None
def request(self, *args, **kwargs):
captured["proxy"] = kwargs.get("proxy")
@ -231,30 +220,118 @@ async def test_handle_async_request_uses_env_proxy(monkeypatch):
assert captured["proxy"] == proxy_url
@pytest.mark.asyncio
async def test_handle_async_request_uses_env_proxy_per_url(monkeypatch):
"""Aiohttp transport should honor HTTP(S)_PROXY env vars unless NO_PROXY matches"""
proxy_url = "http://proxy.local:3128"
monkeypatch.setenv("NO_PROXY", "example.com")
monkeypatch.setenv("HTTP_PROXY", proxy_url)
monkeypatch.setenv("http_proxy", proxy_url)
monkeypatch.setenv("HTTPS_PROXY", proxy_url)
monkeypatch.setenv("https_proxy", proxy_url)
monkeypatch.delenv("DISABLE_AIOHTTP_TRUST_ENV", raising=False)
request_count = 0
proxied_count = 0
class FakeSession:
def __init__(self):
self.closed = False
try:
self._loop = asyncio.get_running_loop()
except RuntimeError:
self._loop = None
def request(self, *args, **kwargs):
nonlocal request_count
nonlocal proxied_count
request_count += 1
if kwargs.get("proxy") is not None:
proxied_count += 1
class Resp:
status = 200
headers = {}
async def __aenter__(self):
return self
async def __aexit__(self, exc_type, exc, tb):
pass
@property
def content(self):
class C:
async def iter_chunked(self, size):
yield b""
return C()
return Resp()
transport = LiteLLMAiohttpTransport(client=lambda: FakeSession()) # type: ignore
request = httpx.Request("GET", "http://example.com")
await transport.handle_async_request(request)
request = httpx.Request("GET", "http://foo.com")
await transport.handle_async_request(request)
assert request_count == 2
assert proxied_count == 1
@pytest.mark.asyncio
async def test_handle_async_request_proxy_cache_per_host(monkeypatch):
"""Aiohttp transport should only cache a proxy per host rather than full URL"""
proxy_url = "http://proxy.local:3128"
monkeypatch.setenv("NO_PROXY", "example.com")
monkeypatch.setenv("HTTP_PROXY", proxy_url)
monkeypatch.setenv("http_proxy", proxy_url)
monkeypatch.setenv("HTTPS_PROXY", proxy_url)
monkeypatch.setenv("https_proxy", proxy_url)
monkeypatch.delenv("DISABLE_AIOHTTP_TRUST_ENV", raising=False)
def factory():
return _make_mock_session()
transport = LiteLLMAiohttpTransport(client=factory) # type: ignore
request = httpx.Request("GET", "http://foo.com/path1")
await transport.handle_async_request(request)
request = httpx.Request("GET", "http://foo.com/path2")
await transport.handle_async_request(request)
assert len(transport.proxy_cache) == 1
def _make_mock_response(should_fail=False, fail_count={"count": 0}):
"""Helper to create a mock aiohttp response"""
class MockResp:
status = 200
headers = {}
async def __aenter__(self):
if should_fail and fail_count["count"] < 1:
fail_count["count"] += 1
raise RuntimeError("Session is closed")
return self
async def __aexit__(self, *args):
pass
@property
def content(self):
class C:
async def iter_chunked(self, size):
yield b"test"
return C()
return MockResp()
@pytest.mark.asyncio
async def test_handle_async_request_total_timeout_triggers():
"""
@ -298,10 +375,10 @@ async def test_handle_async_request_total_timeout_triggers():
await transport.aclose()
await runner.cleanup()
def _make_mock_session(closed=False):
"""Helper to create a mock aiohttp session"""
import asyncio
class MockSession:
def __init__(self):
self.closed = closed
@ -309,10 +386,10 @@ def _make_mock_session(closed=False):
self._loop = asyncio.get_running_loop()
except RuntimeError:
self._loop = None
def request(self, *args, **kwargs):
return _make_mock_response()
return MockSession()
@ -320,14 +397,14 @@ def _make_mock_session(closed=False):
async def test_handle_closed_session_before_request():
"""Test that closed sessions are detected and recreated"""
counts = {"sessions": 0}
def factory():
counts["sessions"] += 1
return _make_mock_session(closed=counts["sessions"] == 1)
transport = LiteLLMAiohttpTransport(client=factory) # type: ignore
response = await transport.handle_async_request(httpx.Request("GET", "http://example.com"))
assert counts["sessions"] == 2 # Created 2 sessions: closed one, then open one
assert response.status_code == 200
@ -337,7 +414,7 @@ async def test_handle_session_closed_during_request():
"""Test that sessions closed during request are handled with retry"""
counts = {"sessions": 0, "requests": 0}
fail_count = {"count": 0}
class MockSession:
def __init__(self):
self.closed = False
@ -345,18 +422,18 @@ async def test_handle_session_closed_during_request():
self._loop = __import__("asyncio").get_running_loop()
except RuntimeError:
self._loop = None
def request(self, *args, **kwargs):
counts["requests"] += 1
return _make_mock_response(should_fail=True, fail_count=fail_count)
def factory():
counts["sessions"] += 1
return MockSession()
transport = LiteLLMAiohttpTransport(client=factory) # type: ignore
response = await transport.handle_async_request(httpx.Request("GET", "http://example.com"))
assert counts["requests"] == 2 # First request failed, second succeeded
assert counts["sessions"] == 2 # Created 2 sessions for retry
assert response.status_code == 200

48
tests/test_litellm/llms/openai/completion/test_text_completion_guardrail_handler.py
View file

@ -4,6 +4,7 @@ Unit tests for OpenAI Text Completion Guardrail Translation Handler
import os
import sys
from typing import List, Optional, Tuple
from unittest.mock import MagicMock
import pytest
@ -21,8 +22,10 @@ from litellm.types.utils import CallTypes, TextChoices, TextCompletionResponse
class MockGuardrail(CustomGuardrail):
"""Mock guardrail for testing"""
async def apply_guardrail(self, text: str, language=None, entities=None) -> str:
return f"{text} [GUARDRAILED]"
async def apply_guardrail(
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
return ([f"{text} [GUARDRAILED]" for text in texts], None)
class TestHandlerDiscovery:
@ -243,19 +246,22 @@ class TestPIIMaskingScenario:
"""Mock PII masking guardrail"""
async def apply_guardrail(
self, text: str, language=None, entities=None
) -> str:
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
# Simple mock: replace email-like patterns
import re
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
# Replace names (simple mock)
masked = masked.replace("John Doe", "[NAME_REDACTED]")
return masked
masked_texts = []
for text in texts:
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
# Replace names (simple mock)
masked = masked.replace("John Doe", "[NAME_REDACTED]")
masked_texts.append(masked)
return (masked_texts, None)
handler = OpenAITextCompletionHandler()
guardrail = PIIMaskingGuardrail(guardrail_name="mask_pii")
@ -303,15 +309,19 @@ class TestPIIMaskingScenario:
"""Mock PII masking guardrail"""
async def apply_guardrail(
self, text: str, language=None, entities=None
) -> str:
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
import re
return re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
masked_texts = []
for text in texts:
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
masked_texts.append(masked)
return (masked_texts, None)
handler = OpenAITextCompletionHandler()
guardrail = PIIMaskingGuardrail(guardrail_name="mask_pii")

30
tests/test_litellm/llms/openai/image_generation/test_image_generation_guardrail_handler.py
View file

@ -4,6 +4,7 @@ Unit tests for OpenAI Image Generation Guardrail Translation Handler
import os
import sys
from typing import List, Optional, Tuple
import pytest
@ -20,8 +21,10 @@ from litellm.types.utils import CallTypes, ImageObject, ImageResponse
class MockGuardrail(CustomGuardrail):
"""Mock guardrail for testing"""
async def apply_guardrail(self, text: str, language=None, entities=None) -> str:
return f"{text} [GUARDRAILED]"
async def apply_guardrail(
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
return ([f"{text} [GUARDRAILED]" for text in texts], None)
class TestHandlerDiscovery:
@ -141,19 +144,22 @@ class TestPIIMaskingScenario:
"""Mock PII masking guardrail"""
async def apply_guardrail(
self, text: str, language=None, entities=None
) -> str:
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
# Simple mock: replace email-like patterns
import re
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
# Replace names (simple mock)
masked = masked.replace("John Doe", "[NAME_REDACTED]")
return masked
masked_texts = []
for text in texts:
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
# Replace names (simple mock)
masked = masked.replace("John Doe", "[NAME_REDACTED]")
masked_texts.append(masked)
return (masked_texts, None)
handler = OpenAIImageGenerationHandler()
guardrail = PIIMaskingGuardrail(guardrail_name="mask_pii")

14
tests/test_litellm/llms/openai/responses/test_openai_responses_guardrail_handler.py
View file

@ -7,7 +7,7 @@ with guardrail transformations.
import os
import sys
from typing import Any
from typing import Any, List, Optional, Tuple
from unittest.mock import AsyncMock, MagicMock
import pytest
@ -29,9 +29,11 @@ from litellm.types.utils import CallTypes
class MockGuardrail(CustomGuardrail):
"""Mock guardrail for testing that transforms text"""
async def apply_guardrail(self, text: str) -> str:
async def apply_guardrail(
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
"""Append [GUARDRAILED] to text"""
return f"{text} [GUARDRAILED]"
return ([f"{text} [GUARDRAILED]" for text in texts], None)
class TestOpenAIResponsesHandlerDiscovery:
@ -450,7 +452,10 @@ class TestOpenAIResponsesHandlerEdgeCases:
"role": "user",
"content": [
{"type": "text", "text": "List content"},
{"type": "image_url", "image_url": {"url": "http://example.com"}},
{
"type": "image_url",
"image_url": {"url": "http://example.com"},
},
],
"type": "message",
},
@ -492,4 +497,3 @@ class TestOpenAIResponsesHandlerEdgeCases:
# Should skip processing and return unchanged
assert result == response

73
tests/test_litellm/llms/openai/speech/test_text_to_speech_guardrail_handler.py
View file

@ -4,6 +4,7 @@ Unit tests for OpenAI Text-to-Speech Guardrail Translation Handler
import os
import sys
from typing import List, Optional, Tuple
import pytest
@ -20,8 +21,10 @@ from litellm.types.utils import CallTypes
class MockGuardrail(CustomGuardrail):
"""Mock guardrail for testing"""
async def apply_guardrail(self, text: str, language=None, entities=None) -> str:
return f"{text} [GUARDRAILED]"
async def apply_guardrail(
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
return ([f"{text} [GUARDRAILED]" for text in texts], None)
class MockBinaryResponse:
@ -169,20 +172,23 @@ class TestPIIMaskingScenario:
"""Mock PII masking guardrail"""
async def apply_guardrail(
self, text: str, language=None, entities=None
) -> str:
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
# Simple mock: replace email-like patterns
import re
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
# Replace names (simple mock)
masked = masked.replace("John Doe", "[NAME_REDACTED]")
masked = masked.replace("555-1234", "[PHONE_REDACTED]")
return masked
masked_texts = []
for text in texts:
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
# Replace names (simple mock)
masked = masked.replace("John Doe", "[NAME_REDACTED]")
masked = masked.replace("555-1234", "[PHONE_REDACTED]")
masked_texts.append(masked)
return (masked_texts, None)
handler = OpenAITextToSpeechHandler()
guardrail = PIIMaskingGuardrail(guardrail_name="mask_pii")
@ -211,17 +217,24 @@ class TestPIIMaskingScenario:
"""Mock PII masking guardrail"""
async def apply_guardrail(
self, text: str, language=None, entities=None
) -> str:
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
import re
# Mask account numbers
masked = re.sub(r"account number \d{8,12}", "account number [REDACTED]", text)
# Mask SSNs
masked = re.sub(r"\d{3}-\d{2}-\d{4}", "[SSN_REDACTED]", masked)
# Mask credit cards
masked = re.sub(r"\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}", "[CC_REDACTED]", masked)
return masked
masked_texts = []
for text in texts:
# Mask account numbers
masked = re.sub(
r"account number \d{8,12}", "account number [REDACTED]", text
)
# Mask SSNs
masked = re.sub(r"\d{3}-\d{2}-\d{4}", "[SSN_REDACTED]", masked)
# Mask credit cards
masked = re.sub(
r"\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}", "[CC_REDACTED]", masked
)
masked_texts.append(masked)
return (masked_texts, None)
handler = OpenAITextToSpeechHandler()
guardrail = PIIMaskingGuardrail(guardrail_name="mask_pii")
@ -256,14 +269,17 @@ class TestContentModerationScenario:
"""Mock content filter guardrail"""
async def apply_guardrail(
self, text: str, language=None, entities=None
) -> str:
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
# Simple mock: filter inappropriate words
bad_words = ["badword", "inappropriate", "offensive"]
filtered = text
for word in bad_words:
filtered = filtered.replace(word, "[FILTERED]")
return filtered
filtered_texts = []
for text in texts:
filtered = text
for word in bad_words:
filtered = filtered.replace(word, "[FILTERED]")
filtered_texts.append(filtered)
return (filtered_texts, None)
handler = OpenAITextToSpeechHandler()
guardrail = ContentFilterGuardrail(guardrail_name="content_filter")
@ -322,4 +338,3 @@ class TestMultilingualTTS:
assert f"Testing with {voice} voice [GUARDRAILED]" == result["input"]
assert result["voice"] == voice

80
tests/test_litellm/llms/openai/transcriptions/test_audio_transcription_guardrail_handler.py
View file

@ -4,6 +4,7 @@ Unit tests for OpenAI Audio Transcription Guardrail Translation Handler
import os
import sys
from typing import List, Optional, Tuple
import pytest
@ -21,8 +22,10 @@ from litellm.utils import TranscriptionResponse
class MockGuardrail(CustomGuardrail):
"""Mock guardrail for testing"""
async def apply_guardrail(self, text: str, language=None, entities=None) -> str:
return f"{text} [GUARDRAILED]"
async def apply_guardrail(
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
return ([f"{text} [GUARDRAILED]" for text in texts], None)
class TestHandlerDiscovery:
@ -140,20 +143,23 @@ class TestPIIMaskingScenario:
"""Mock PII masking guardrail"""
async def apply_guardrail(
self, text: str, language=None, entities=None
) -> str:
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
# Simple mock: replace email-like patterns
import re
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
# Replace names (simple mock)
masked = masked.replace("John Doe", "[NAME_REDACTED]")
masked = masked.replace("555-1234", "[PHONE_REDACTED]")
return masked
masked_texts = []
for text in texts:
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
text,
)
# Replace names (simple mock)
masked = masked.replace("John Doe", "[NAME_REDACTED]")
masked = masked.replace("555-1234", "[PHONE_REDACTED]")
masked_texts.append(masked)
return (masked_texts, None)
handler = OpenAIAudioTranscriptionHandler()
guardrail = PIIMaskingGuardrail(guardrail_name="mask_pii")
@ -181,23 +187,26 @@ class TestPIIMaskingScenario:
"""Mock PII masking guardrail"""
async def apply_guardrail(
self, text: str, language=None, entities=None
) -> str:
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
import re
# Mask credit card numbers
masked = re.sub(
r"\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}", "[CC_REDACTED]", text
)
# Mask SSNs
masked = re.sub(r"\d{3}-\d{2}-\d{4}", "[SSN_REDACTED]", masked)
# Mask emails
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
masked,
)
return masked
masked_texts = []
for text in texts:
# Mask credit card numbers
masked = re.sub(
r"\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}", "[CC_REDACTED]", text
)
# Mask SSNs
masked = re.sub(r"\d{3}-\d{2}-\d{4}", "[SSN_REDACTED]", masked)
# Mask emails
masked = re.sub(
r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"[EMAIL_REDACTED]",
masked,
)
masked_texts.append(masked)
return (masked_texts, None)
handler = OpenAIAudioTranscriptionHandler()
guardrail = PIIMaskingGuardrail(guardrail_name="mask_pii")
@ -231,14 +240,17 @@ class TestContentModerationScenario:
"""Mock profanity filter guardrail"""
async def apply_guardrail(
self, text: str, language=None, entities=None
) -> str:
self, texts: List[str], request_data: dict, input_type: str, **kwargs
) -> Tuple[List[str], Optional[List[str]]]:
# Simple mock: replace common profanity
bad_words = ["badword1", "badword2", "inappropriate"]
filtered = text
for word in bad_words:
filtered = filtered.replace(word, "[FILTERED]")
return filtered
filtered_texts = []
for text in texts:
filtered = text
for word in bad_words:
filtered = filtered.replace(word, "[FILTERED]")
filtered_texts.append(filtered)
return (filtered_texts, None)
handler = OpenAIAudioTranscriptionHandler()
guardrail = ProfanityFilterGuardrail(guardrail_name="content_filter")

27
tests/test_litellm/llms/openrouter/chat/test_openrouter_chat_transformation.py
View file

@ -489,3 +489,30 @@ def test_openrouter_cost_tracking_streaming():
# Verify cost field is preserved in the Usage object - this is the key data for cost tracking
# The chunk_parser converts the dict to a Usage Pydantic model which includes the cost field
assert result2.usage.cost == 0.0001
def test_openrouter_reasoning_models_allow_reasoning_effort_param():
"""
OpenRouter reasoning-capable models should accept the reasoning_effort param.
"""
config = OpenrouterConfig()
supported_params = config.get_supported_openai_params(
model="openrouter/deepseek/deepseek-v3.2"
)
assert "reasoning_effort" in supported_params
assert supported_params.count("reasoning_effort") == 1
def test_openrouter_non_reasoning_models_do_not_add_reasoning_effort():
"""
Models without reasoning support should not gain reasoning-specific params.
"""
config = OpenrouterConfig()
supported_params = config.get_supported_openai_params(
model="openrouter/anthropic/claude-3-5-haiku"
)
assert "reasoning_effort" not in supported_params

4
tests/test_litellm/llms/ragflow/chat/__init__.py Normal file
View file

@ -0,0 +1,4 @@
"""
RAGFlow chat transformation tests.
"""

376
tests/test_litellm/llms/ragflow/chat/test_ragflow_chat_transformation.py Normal file
View file

@ -0,0 +1,376 @@
"""
Test file for RAGFlow chat transformation functionality.
Tests the model name parsing, URL construction, and request transformation
for RAGFlow's OpenAI-compatible API with custom path structures.
"""
import os
import sys
from unittest.mock import Mock, patch
import pytest
# Add the project root to Python path
sys.path.insert(0, os.path.abspath("../../../../.."))
import litellm
from litellm.llms.ragflow.chat.transformation import RAGFlowConfig
from litellm.types.llms.openai import AllMessageValues
class TestRAGFlowChatTransformation:
"""Test suite for RAGFlow chat transformation functionality."""
def test_parse_ragflow_model_chat(self):
"""Test parsing of chat model format."""
config = RAGFlowConfig()
model = "ragflow/chat/my-chat-id/gpt-4o-mini"
endpoint_type, entity_id, model_name = config._parse_ragflow_model(model)
assert endpoint_type == "chat"
assert entity_id == "my-chat-id"
assert model_name == "gpt-4o-mini"
def test_parse_ragflow_model_agent(self):
"""Test parsing of agent model format."""
config = RAGFlowConfig()
model = "ragflow/agent/my-agent-id/gpt-4o-mini"
endpoint_type, entity_id, model_name = config._parse_ragflow_model(model)
assert endpoint_type == "agent"
assert entity_id == "my-agent-id"
assert model_name == "gpt-4o-mini"
def test_parse_ragflow_model_with_slashes_in_model_name(self):
"""Test parsing when model name contains slashes."""
config = RAGFlowConfig()
model = "ragflow/chat/my-chat-id/openai/gpt-4o-mini"
endpoint_type, entity_id, model_name = config._parse_ragflow_model(model)
assert endpoint_type == "chat"
assert entity_id == "my-chat-id"
assert model_name == "openai/gpt-4o-mini"
def test_parse_ragflow_model_invalid_format(self):
"""Test parsing with invalid model format."""
config = RAGFlowConfig()
with pytest.raises(ValueError, match="Invalid RAGFlow model format"):
config._parse_ragflow_model("ragflow/chat/model-name")
with pytest.raises(ValueError, match="Invalid RAGFlow model format"):
config._parse_ragflow_model("invalid/chat/id/model")
with pytest.raises(ValueError, match="Must start with 'ragflow/'"):
config._parse_ragflow_model("not-ragflow/chat/id/model")
def test_parse_ragflow_model_invalid_endpoint_type(self):
"""Test parsing with invalid endpoint type."""
config = RAGFlowConfig()
with pytest.raises(ValueError, match="Invalid RAGFlow endpoint type"):
config._parse_ragflow_model("ragflow/invalid/my-id/model")
def test_get_complete_url_chat(self):
"""Test URL construction for chat endpoint."""
config = RAGFlowConfig()
model = "ragflow/chat/my-chat-id/gpt-4o-mini"
api_base = "http://localhost:9380"
url = config.get_complete_url(
api_base=api_base,
api_key=None,
model=model,
optional_params={},
litellm_params={},
stream=False,
)
assert url == "http://localhost:9380/api/v1/chats_openai/my-chat-id/chat/completions"
def test_get_complete_url_agent(self):
"""Test URL construction for agent endpoint."""
config = RAGFlowConfig()
model = "ragflow/agent/my-agent-id/gpt-4o-mini"
api_base = "http://localhost:9380"
url = config.get_complete_url(
api_base=api_base,
api_key=None,
model=model,
optional_params={},
litellm_params={},
stream=False,
)
assert url == "http://localhost:9380/api/v1/agents_openai/my-agent-id/chat/completions"
def test_get_complete_url_strips_v1(self):
"""Test URL construction when api_base ends with /v1."""
config = RAGFlowConfig()
model = "ragflow/chat/my-chat-id/gpt-4o-mini"
api_base = "http://localhost:9380/v1"
url = config.get_complete_url(
api_base=api_base,
api_key=None,
model=model,
optional_params={},
litellm_params={},
stream=False,
)
assert url == "http://localhost:9380/api/v1/chats_openai/my-chat-id/chat/completions"
def test_get_complete_url_strips_api_v1(self):
"""Test URL construction when api_base ends with /api/v1."""
config = RAGFlowConfig()
model = "ragflow/agent/my-agent-id/gpt-4o-mini"
api_base = "http://localhost:9380/api/v1"
url = config.get_complete_url(
api_base=api_base,
api_key=None,
model=model,
optional_params={},
litellm_params={},
stream=False,
)
assert url == "http://localhost:9380/api/v1/agents_openai/my-agent-id/chat/completions"
def test_get_complete_url_from_litellm_params(self):
"""Test URL construction with api_base from litellm_params."""
config = RAGFlowConfig()
model = "ragflow/chat/my-chat-id/gpt-4o-mini"
# Create a simple dict-like object for litellm_params
class LiteLLMParams:
def __init__(self):
self.api_base = "http://ragflow-server:9380"
litellm_params = LiteLLMParams()
url = config.get_complete_url(
api_base=None,
api_key=None,
model=model,
optional_params={},
litellm_params=litellm_params,
stream=False,
)
assert url == "http://ragflow-server:9380/api/v1/chats_openai/my-chat-id/chat/completions"
def test_get_complete_url_missing_api_base(self):
"""Test URL construction when api_base is missing."""
config = RAGFlowConfig()
model = "ragflow/chat/my-chat-id/gpt-4o-mini"
with pytest.raises(ValueError, match="api_base is required"):
config.get_complete_url(
api_base=None,
api_key=None,
model=model,
optional_params={},
litellm_params={},
stream=False,
)
@patch.dict(os.environ, {"RAGFLOW_API_BASE": "http://env-ragflow:9380"})
def test_get_complete_url_from_environment(self):
"""Test URL construction with api_base from environment variable."""
config = RAGFlowConfig()
model = "ragflow/agent/my-agent-id/gpt-4o-mini"
url = config.get_complete_url(
api_base=None,
api_key=None,
model=model,
optional_params={},
litellm_params={},
stream=False,
)
assert url == "http://env-ragflow:9380/api/v1/agents_openai/my-agent-id/chat/completions"
def test_validate_environment_sets_headers(self):
"""Test that validate_environment sets proper headers."""
config = RAGFlowConfig()
headers = {}
model = "ragflow/chat/my-chat-id/gpt-4o-mini"
messages = [{"role": "user", "content": "Hello"}]
api_key = "test-api-key"
result_headers = config.validate_environment(
headers=headers,
model=model,
messages=messages,
optional_params={},
litellm_params={},
api_key=api_key,
api_base="http://localhost:9380",
)
assert result_headers["Authorization"] == "Bearer test-api-key"
assert result_headers["Content-Type"] == "application/json"
def test_validate_environment_stores_actual_model(self):
"""Test that validate_environment stores actual model name."""
config = RAGFlowConfig()
headers = {}
model = "ragflow/chat/my-chat-id/gpt-4o-mini"
messages = [{"role": "user", "content": "Hello"}]
litellm_params = {}
config.validate_environment(
headers=headers,
model=model,
messages=messages,
optional_params={},
litellm_params=litellm_params,
api_key="test-key",
api_base="http://localhost:9380",
)
assert litellm_params["_ragflow_actual_model"] == "gpt-4o-mini"
@patch.dict(os.environ, {"RAGFLOW_API_KEY": "env-api-key"})
def test_validate_environment_from_environment(self):
"""Test that validate_environment gets api_key from environment."""
config = RAGFlowConfig()
headers = {}
model = "ragflow/agent/my-agent-id/gpt-4o-mini"
messages = [{"role": "user", "content": "Hello"}]
result_headers = config.validate_environment(
headers=headers,
model=model,
messages=messages,
optional_params={},
litellm_params={},
api_key=None,
api_base="http://localhost:9380",
)
assert result_headers["Authorization"] == "Bearer env-api-key"
def test_validate_environment_from_litellm_params(self):
"""Test that validate_environment gets api_key from litellm_params."""
config = RAGFlowConfig()
headers = {}
model = "ragflow/chat/my-chat-id/gpt-4o-mini"
messages = [{"role": "user", "content": "Hello"}]
# Create a simple object for litellm_params with api_key attribute
class LiteLLMParams:
def __init__(self):
self.api_key = "litellm-params-key"
def __setitem__(self, key, value):
setattr(self, key, value)
litellm_params = LiteLLMParams()
result_headers = config.validate_environment(
headers=headers,
model=model,
messages=messages,
optional_params={},
litellm_params=litellm_params,
api_key=None,
api_base="http://localhost:9380",
)
assert result_headers["Authorization"] == "Bearer litellm-params-key"
def test_transform_request_uses_actual_model(self):
"""Test that transform_request uses the actual model name."""
config = RAGFlowConfig()
model = "ragflow/chat/my-chat-id/gpt-4o-mini"
messages = [{"role": "user", "content": "Hello"}]
litellm_params = {"_ragflow_actual_model": "gpt-4o-mini"}
# Test the actual behavior by checking the model in the result
result = config.transform_request(
model=model,
messages=messages,
optional_params={},
litellm_params=litellm_params,
headers={},
)
# The result should contain the actual model name, not the full ragflow path
assert result["model"] == "gpt-4o-mini"
assert result["messages"] == messages
def test_transform_request_fallback_parsing(self):
"""Test that transform_request falls back to parsing if _ragflow_actual_model is missing."""
config = RAGFlowConfig()
model = "ragflow/agent/my-agent-id/gpt-4o-mini"
messages = [{"role": "user", "content": "Hello"}]
litellm_params = {} # Missing _ragflow_actual_model
result = config.transform_request(
model=model,
messages=messages,
optional_params={},
litellm_params=litellm_params,
headers={},
)
# Should parse and use the actual model name
assert result["model"] == "gpt-4o-mini"
assert result["messages"] == messages
def test_get_openai_compatible_provider_info(self):
"""Test _get_openai_compatible_provider_info returns correct values."""
config = RAGFlowConfig()
model = "ragflow/chat/my-chat-id/gpt-4o-mini"
api_base = "http://localhost:9380"
api_key = "test-key"
result_api_base, result_api_key, result_provider = config._get_openai_compatible_provider_info(
model=model,
api_base=api_base,
api_key=api_key,
custom_llm_provider="ragflow",
)
assert result_api_base == api_base
assert result_api_key == api_key
assert result_provider == "ragflow"
@patch.dict(os.environ, {"RAGFLOW_API_BASE": "http://env-base:9380", "RAGFLOW_API_KEY": "env-key"})
def test_get_openai_compatible_provider_info_from_env(self):
"""Test _get_openai_compatible_provider_info gets values from environment."""
config = RAGFlowConfig()
model = "ragflow/agent/my-agent-id/gpt-4o-mini"
result_api_base, result_api_key, result_provider = config._get_openai_compatible_provider_info(
model=model,
api_base=None,
api_key=None,
custom_llm_provider="ragflow",
)
assert result_api_base == "http://env-base:9380"
assert result_api_key == "env-key"
assert result_provider == "ragflow"

277
tests/test_litellm/llms/vertex_ai/gemini/test_thought_signature_in_tool_call_id.py
View file

@ -4,9 +4,13 @@ Tests for embedding thought signatures in tool call IDs for OpenAI client compat
When using OpenAI clients (instead of LiteLLM SDK), provider_specific_fields are not preserved.
This test suite validates that thought signatures can be embedded in tool call IDs and extracted
when converting back to Gemini format.
Note: Embedding signatures in tool call IDs is a beta feature that requires
enable_preview_features=True to be enabled.
"""
import pytest
import litellm
from litellm.llms.vertex_ai.gemini.vertex_and_google_ai_studio_gemini import (
VertexGeminiConfig,
)
@ -62,36 +66,57 @@ def test_encode_tool_call_id_without_signature():
assert decoded_signature is None
def test_tool_call_id_includes_signature_in_response():
"""Test that tool call IDs in responses include embedded thought signatures"""
@pytest.mark.parametrize("enable_preview_features", [True, False])
def test_tool_call_id_includes_signature_in_response(enable_preview_features):
"""Test that tool call IDs in responses include embedded thought signatures only when preview features are enabled"""
test_signature = "Co4CAdHtim/rWgXbz2Ghp4tShzLeMASrPw6JJyYIC3cbVyZnKzU3uv8/wVzyS2sKRPL2m8QQHHXbNQhEEz500G7n/4ZMmksdTtfQcJMoT76S1DGwhnAiLwTgWCNXs3lEb4M19EVYoWFxhrH5Lr9YMIquoU9U4paydGwvZyIyigamIg4B6WnxrRsf0KZV12gJed0DZuKczvOFtHz3zUnmZRlOiTzd5gBVyQM+5jv1VI8m4WUKd6cN/5a5ZvaA0ggiO6kdVhlpIVs7GczSEVJD8KH4u02X7VSnb7CvykqDntZzV0y8rZFBEFGKrChmeHlWXP4D1IB3F9KQyhuLgWImMzg4BajKVxxMU737JGnNISy5"
parts_with_signature = [
HttpxPartType(
functionCall={
"name": "get_current_temperature",
"args": {"location": "Paris"},
},
thoughtSignature=test_signature,
# Save original state
original_flag = litellm.enable_preview_features
litellm.enable_preview_features = enable_preview_features
try:
parts_with_signature = [
HttpxPartType(
functionCall={
"name": "get_current_temperature",
"args": {"location": "Paris"},
},
thoughtSignature=test_signature,
)
]
function, tools, _ = VertexGeminiConfig._transform_parts(
parts=parts_with_signature,
cumulative_tool_call_idx=0,
is_function_call=False,
)
]
function, tools, _ = VertexGeminiConfig._transform_parts(
parts=parts_with_signature,
cumulative_tool_call_idx=0,
is_function_call=False,
)
# Verify tool call exists
assert tools is not None
assert len(tools) == 1
tool_call_id = tools[0]["id"]
# Verify signature is always in provider_specific_fields
assert tools[0].get("provider_specific_fields", {}).get("thought_signature") == test_signature
# Verify tool call ID includes thought signature
assert tools is not None
assert len(tools) == 1
tool_call_id = tools[0]["id"]
assert THOUGHT_SIGNATURE_SEPARATOR in tool_call_id
# Verify we can decode it using the factory function
tool_obj = {"id": tool_call_id, "type": "function"}
decoded_sig = _get_thought_signature_from_tool(tool_obj)
assert decoded_sig == test_signature
if enable_preview_features:
# When preview features enabled, signature should be embedded in ID
assert THOUGHT_SIGNATURE_SEPARATOR in tool_call_id
# Verify we can decode it using the factory function
tool_obj = {"id": tool_call_id, "type": "function"}
decoded_sig = _get_thought_signature_from_tool(tool_obj)
assert decoded_sig == test_signature
else:
# When preview features disabled, signature should NOT be embedded in ID
assert THOUGHT_SIGNATURE_SEPARATOR not in tool_call_id
# But we can still extract from provider_specific_fields
tool_obj = {"id": tool_call_id, "type": "function", "provider_specific_fields": {"thought_signature": test_signature}}
decoded_sig = _get_thought_signature_from_tool(tool_obj)
assert decoded_sig == test_signature
finally:
# Restore original state
litellm.enable_preview_features = original_flag
def test_get_thought_signature_backward_compatibility():
@ -168,97 +193,157 @@ def test_convert_to_gemini_with_embedded_signature():
assert gemini_parts[0]["thoughtSignature"] == test_signature
def test_openai_client_e2e_flow():
@pytest.mark.parametrize("enable_preview_features", [True, False])
def test_openai_client_e2e_flow(enable_preview_features):
"""
End-to-end test simulating OpenAI client usage:
1. LiteLLM receives response from Gemini with thought signature
2. LiteLLM embeds signature in tool call ID
2. LiteLLM embeds signature in tool call ID (if preview features enabled)
3. OpenAI client sends message back with same tool call ID
4. LiteLLM extracts signature from ID and sends to Gemini
4. LiteLLM extracts signature from ID/provider_specific_fields and sends to Gemini
"""
test_signature = "Co4CAdHtim/rWgXbz2Ghp4tShzLeMASrPw6JJyYIC3cbVyZnKzU3uv8/wVzyS2sKRPL2m8QQHHXbNQhEEz500G7n/4ZMmksdTtfQcJMoT76S1DGwhnAiLwTgWCNXs3lEb4M19EVYoWFxhrH5Lr9YMIquoU9U4paydGwvZyIyigamIg4B6WnxrRsf0KZV12gJed0DZuKczvOFtHz3zUnmZRlOiTzd5gBVyQM+5jv1VI8m4WUKd6cN/5a5ZvaA0ggiO6kdVhlpIVs7GczSEVJD8KH4u02X7VSnb7CvykqDntZzV0y8rZFBEFGKrChmeHlWXP4D1IB3F9KQyhuLgWImMzg4BajKVxxMU737JGnNISy5"
# Step 1: Gemini returns function call with thought signature
gemini_parts = [
HttpxPartType(
functionCall={
"name": "get_current_temperature",
"args": {"location": "Paris"},
},
thoughtSignature=test_signature,
)
]
# Save original state
original_flag = litellm.enable_preview_features
litellm.enable_preview_features = enable_preview_features
# Step 2: LiteLLM transforms to OpenAI format with embedded signature
function, tools, _ = VertexGeminiConfig._transform_parts(
parts=gemini_parts,
cumulative_tool_call_idx=0,
is_function_call=False,
)
assert tools is not None
assert len(tools) == 1
tool_call_id = tools[0]["id"]
assert THOUGHT_SIGNATURE_SEPARATOR in tool_call_id
# Step 3: OpenAI client sends back assistant message (preserves tool_call_id)
openai_assistant_message = {
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": tool_call_id, # Preserved from response
"type": "function",
"function": {
try:
# Step 1: Gemini returns function call with thought signature
gemini_parts = [
HttpxPartType(
functionCall={
"name": "get_current_temperature",
"arguments": '{"location": "Paris"}',
"args": {"location": "Paris"},
},
thoughtSignature=test_signature,
)
]
# Step 2: LiteLLM transforms to OpenAI format
function, tools, _ = VertexGeminiConfig._transform_parts(
parts=gemini_parts,
cumulative_tool_call_idx=0,
is_function_call=False,
)
assert tools is not None
assert len(tools) == 1
tool_call_id = tools[0]["id"]
if enable_preview_features:
# When preview features enabled, signature should be embedded in ID
assert THOUGHT_SIGNATURE_SEPARATOR in tool_call_id
else:
# When preview features disabled, signature should NOT be embedded in ID
assert THOUGHT_SIGNATURE_SEPARATOR not in tool_call_id
# Step 3: OpenAI client sends back assistant message
# For the disabled case, we simulate that the client might have provider_specific_fields
# or we use the embedded ID if preview features were enabled
if enable_preview_features:
openai_assistant_message = {
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": tool_call_id, # Preserved from response (with embedded signature)
"type": "function",
"function": {
"name": "get_current_temperature",
"arguments": '{"location": "Paris"}',
},
}
],
}
else:
# When preview features disabled, simulate that provider_specific_fields might be preserved
# (though in real OpenAI client usage, this might not happen)
# For this test, we'll use provider_specific_fields to show extraction still works
openai_assistant_message = {
"role": "assistant",
"content": "",
"tool_calls": [
{
"id": tool_call_id, # ID without embedded signature
"type": "function",
"function": {
"name": "get_current_temperature",
"arguments": '{"location": "Paris"}',
},
"provider_specific_fields": {"thought_signature": test_signature},
}
],
}
],
}
# Step 4: LiteLLM converts back to Gemini format, extracting signature
gemini_parts_converted = convert_to_gemini_tool_call_invoke(
openai_assistant_message
)
# Step 4: LiteLLM converts back to Gemini format, extracting signature
gemini_parts_converted = convert_to_gemini_tool_call_invoke(
openai_assistant_message
)
# Verify signature is preserved through the round trip
assert len(gemini_parts_converted) == 1
assert "thoughtSignature" in gemini_parts_converted[0]
assert gemini_parts_converted[0]["thoughtSignature"] == test_signature
# Verify signature is preserved through the round trip
assert len(gemini_parts_converted) == 1
assert "thoughtSignature" in gemini_parts_converted[0]
assert gemini_parts_converted[0]["thoughtSignature"] == test_signature
finally:
# Restore original state
litellm.enable_preview_features = original_flag
def test_parallel_tool_calls_with_signatures():
@pytest.mark.parametrize("enable_preview_features", [True, False])
def test_parallel_tool_calls_with_signatures(enable_preview_features):
"""Test that parallel tool calls preserve signatures correctly"""
signature1 = "signature_for_first_call"
# Only first call has signature (Gemini behavior for parallel calls)
gemini_parts = [
HttpxPartType(
functionCall={"name": "get_temperature", "args": {"location": "Paris"}},
thoughtSignature=signature1,
),
HttpxPartType(
functionCall={"name": "get_temperature", "args": {"location": "London"}},
# No signature for second parallel call
),
]
# Save original state
original_flag = litellm.enable_preview_features
litellm.enable_preview_features = enable_preview_features
function, tools, _ = VertexGeminiConfig._transform_parts(
parts=gemini_parts,
cumulative_tool_call_idx=0,
is_function_call=False,
)
try:
gemini_parts = [
HttpxPartType(
functionCall={"name": "get_temperature", "args": {"location": "Paris"}},
thoughtSignature=signature1,
),
HttpxPartType(
functionCall={"name": "get_temperature", "args": {"location": "London"}},
# No signature for second parallel call
),
]
assert tools is not None
assert len(tools) == 2
function, tools, _ = VertexGeminiConfig._transform_parts(
parts=gemini_parts,
cumulative_tool_call_idx=0,
is_function_call=False,
)
# First tool call has signature in ID
assert THOUGHT_SIGNATURE_SEPARATOR in tools[0]["id"]
sig1 = _get_thought_signature_from_tool({"id": tools[0]["id"], "type": "function"})
assert sig1 == signature1
assert tools is not None
assert len(tools) == 2
# Second tool call has no signature in ID
assert THOUGHT_SIGNATURE_SEPARATOR not in tools[1]["id"]
sig2 = _get_thought_signature_from_tool({"id": tools[1]["id"], "type": "function"})
assert sig2 is None
# First tool call should have signature in provider_specific_fields
assert tools[0].get("provider_specific_fields", {}).get("thought_signature") == signature1
if enable_preview_features:
# When preview features enabled, first tool call has signature in ID
assert THOUGHT_SIGNATURE_SEPARATOR in tools[0]["id"]
sig1 = _get_thought_signature_from_tool({"id": tools[0]["id"], "type": "function"})
assert sig1 == signature1
else:
# When preview features disabled, signature should NOT be in ID
assert THOUGHT_SIGNATURE_SEPARATOR not in tools[0]["id"]
# But we can extract from provider_specific_fields
sig1 = _get_thought_signature_from_tool({
"id": tools[0]["id"],
"type": "function",
"provider_specific_fields": {"thought_signature": signature1}
})
assert sig1 == signature1
# Second tool call has no signature in ID (regardless of flag)
assert THOUGHT_SIGNATURE_SEPARATOR not in tools[1]["id"]
sig2 = _get_thought_signature_from_tool({"id": tools[1]["id"], "type": "function"})
assert sig2 is None
finally:
# Restore original state
litellm.enable_preview_features = original_flag

344
tests/test_litellm/proxy/auth/test_organization_budget_enforcement.py Normal file
View file

@ -0,0 +1,344 @@
"""
Tests for organization budget enforcement.
These tests verify that organization-level budgets are properly enforced during
request authentication. When an organization's spend exceeds its max_budget,
requests should fail with BudgetExceededError.
This prevents teams within an organization from collectively exceeding the
organization's budget limit.
"""
import asyncio
import os
import sys
from typing import Optional
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
sys.path.insert(0, os.path.abspath("../../../"))
import litellm
from litellm.proxy._types import (
LiteLLM_BudgetTable,
LiteLLM_OrganizationTable,
LiteLLM_TeamTable,
UserAPIKeyAuth,
)
from litellm.proxy.auth.auth_checks import common_checks
from litellm.proxy.utils import ProxyLogging
@pytest.mark.asyncio
async def test_organization_budget_exceeded_blocks_request():
"""
Bug: Organization budget is retrieved but NEVER enforced.
When organization spend >= organization_max_budget, requests should fail
with BudgetExceededError. Currently this passes because no check exists.
"""
org_id = "test-org-budget-exceeded"
# Organization with max_budget of 100, but spend is 150
org_object = LiteLLM_OrganizationTable(
organization_id=org_id,
budget_id="org-budget-1",
spend=150.0, # Over budget!
models=["gpt-4"],
created_by="test",
updated_by="test",
litellm_budget_table=LiteLLM_BudgetTable(
max_budget=100.0, # Budget is 100
),
)
# Team within the organization (team itself is under budget)
team_object = LiteLLM_TeamTable(
team_id="test-team-1",
organization_id=org_id,
max_budget=50.0, # Team budget is 50
spend=10.0, # Team spend is only 10 - under budget
models=["gpt-4"],
)
# Valid token with organization info
valid_token = UserAPIKeyAuth(
token="sk-test-123",
team_id="test-team-1",
org_id=org_id,
organization_max_budget=100.0, # This is set but never checked!
)
mock_request = MagicMock()
mock_request.url.path = "/v1/chat/completions"
mock_proxy_logging = MagicMock(spec=ProxyLogging)
mock_proxy_logging.budget_alerts = AsyncMock()
with patch("litellm.proxy.proxy_server.prisma_client") as mock_prisma:
with patch("litellm.proxy.proxy_server.user_api_key_cache") as mock_cache:
with patch("litellm.proxy.auth.auth_checks.get_org_object", new_callable=AsyncMock) as mock_get_org:
mock_get_org.return_value = org_object
# BUG: This should raise BudgetExceededError but currently passes
with pytest.raises(litellm.BudgetExceededError) as exc_info:
await common_checks(
request_body={"model": "gpt-4"},
team_object=team_object,
user_object=None,
end_user_object=None,
global_proxy_spend=None,
general_settings={},
route="/v1/chat/completions",
llm_router=None,
proxy_logging_obj=mock_proxy_logging,
valid_token=valid_token,
request=mock_request,
)
assert "Organization" in str(exc_info.value.message)
assert exc_info.value.current_cost == 150.0
assert exc_info.value.max_budget == 100.0
@pytest.mark.asyncio
async def test_multiple_teams_exceed_organization_budget():
"""
Test that organization budget is enforced even when individual teams are under budget.
Scenario:
- Organization max_budget = $5000, spend = $5000 (at limit)
- Team A spend = $1500 (under team budget of $2000)
- Request via Team A should FAIL because org is at budget limit
Expected: Request fails with BudgetExceededError
"""
org_id = "multi-team-org"
# Organization at budget limit
org_object = LiteLLM_OrganizationTable(
organization_id=org_id,
budget_id="org-budget-2",
spend=5000.0, # At $5000 limit
models=["gpt-4"],
created_by="test",
updated_by="test",
litellm_budget_table=LiteLLM_BudgetTable(
max_budget=5000.0, # Org budget is $5000
),
)
# Team A - under its own budget, but org is almost at limit
team_a = LiteLLM_TeamTable(
team_id="team-a",
organization_id=org_id,
max_budget=2000.0,
spend=1500.0, # Team A has spent $1500 of its $2000 budget
models=["gpt-4"],
)
valid_token = UserAPIKeyAuth(
token="sk-team-a-key",
team_id="team-a",
org_id=org_id,
organization_max_budget=5000.0, # Set but never enforced
)
mock_request = MagicMock()
mock_request.url.path = "/v1/chat/completions"
mock_proxy_logging = MagicMock(spec=ProxyLogging)
mock_proxy_logging.budget_alerts = AsyncMock()
with patch("litellm.proxy.proxy_server.prisma_client") as mock_prisma:
with patch("litellm.proxy.proxy_server.user_api_key_cache") as mock_cache:
with patch("litellm.proxy.auth.auth_checks.get_org_object", new_callable=AsyncMock) as mock_get_org:
mock_get_org.return_value = org_object
# Org is at budget limit, should raise BudgetExceededError
with pytest.raises(litellm.BudgetExceededError) as exc_info:
await common_checks(
request_body={"model": "gpt-4"},
team_object=team_a,
user_object=None,
end_user_object=None,
global_proxy_spend=None,
general_settings={},
route="/v1/chat/completions",
llm_router=None,
proxy_logging_obj=mock_proxy_logging,
valid_token=valid_token,
request=mock_request,
)
# Verify the error message mentions organization
assert "Organization" in str(exc_info.value.message)
assert exc_info.value.current_cost == 5000.0
assert exc_info.value.max_budget == 5000.0
@pytest.mark.asyncio
async def test_organization_budget_fields_are_checked():
"""
Verify that organization_max_budget is populated in UserAPIKeyAuth
and BudgetExceededError is raised when organization is over budget.
"""
# Token has org budget info
valid_token = UserAPIKeyAuth(
token="sk-test",
team_id="test-team",
org_id="test-org",
organization_max_budget=100.0, # Budget is $100
)
# Verify the field exists and is set
assert valid_token.organization_max_budget == 100.0
assert valid_token.org_id == "test-org"
team_object = LiteLLM_TeamTable(
team_id="test-team",
organization_id="test-org",
max_budget=None,
spend=0.0,
models=["gpt-4"],
)
mock_request = MagicMock()
mock_request.url.path = "/v1/chat/completions"
mock_proxy_logging = MagicMock(spec=ProxyLogging)
mock_proxy_logging.budget_alerts = AsyncMock()
# Organization is over budget
org_over_budget = LiteLLM_OrganizationTable(
organization_id="test-org",
budget_id="budget-1",
spend=150.0, # Over $100 budget
models=["gpt-4"],
created_by="test",
updated_by="test",
litellm_budget_table=LiteLLM_BudgetTable(max_budget=100.0),
)
with patch("litellm.proxy.proxy_server.prisma_client") as mock_prisma:
with patch("litellm.proxy.proxy_server.user_api_key_cache") as mock_cache:
with patch("litellm.proxy.auth.auth_checks.get_org_object", new_callable=AsyncMock) as mock_get_org:
mock_get_org.return_value = org_over_budget
# Organization is over budget, should raise BudgetExceededError
with pytest.raises(litellm.BudgetExceededError) as exc_info:
await common_checks(
request_body={"model": "gpt-4"},
team_object=team_object,
user_object=None,
end_user_object=None,
global_proxy_spend=None,
general_settings={},
route="/v1/chat/completions",
llm_router=None,
proxy_logging_obj=mock_proxy_logging,
valid_token=valid_token,
request=mock_request,
)
assert exc_info.value.current_cost == 150.0
assert exc_info.value.max_budget == 100.0
@pytest.mark.asyncio
async def test_both_team_and_org_budget_enforced():
"""
Verify that both team budget and organization budget are enforced consistently.
This test verifies:
1. Team over budget raises BudgetExceededError
2. Organization over budget also raises BudgetExceededError
"""
mock_request = MagicMock()
mock_request.url.path = "/v1/chat/completions"
mock_proxy_logging = MagicMock(spec=ProxyLogging)
mock_proxy_logging.budget_alerts = AsyncMock()
# Scenario A: Team over budget - should raise BudgetExceededError
team_over_budget = LiteLLM_TeamTable(
team_id="team-over",
max_budget=100.0,
spend=150.0, # Over budget
models=["gpt-4"],
)
valid_token_team = UserAPIKeyAuth(
token="sk-team-test",
team_id="team-over",
)
with patch("litellm.proxy.proxy_server.prisma_client") as mock_prisma:
with patch("litellm.proxy.proxy_server.user_api_key_cache") as mock_cache:
with pytest.raises(litellm.BudgetExceededError) as exc_info:
await common_checks(
request_body={"model": "gpt-4"},
team_object=team_over_budget,
user_object=None,
end_user_object=None,
global_proxy_spend=None,
general_settings={},
route="/v1/chat/completions",
llm_router=None,
proxy_logging_obj=mock_proxy_logging,
valid_token=valid_token_team,
request=mock_request,
)
assert "Team" in str(exc_info.value.message)
# Scenario B: Org over budget - should also raise BudgetExceededError
org_over_budget = LiteLLM_OrganizationTable(
organization_id="org-over",
budget_id="budget-1",
spend=150.0, # Over $100 budget
models=["gpt-4"],
created_by="test",
updated_by="test",
litellm_budget_table=LiteLLM_BudgetTable(max_budget=100.0),
)
team_under_budget = LiteLLM_TeamTable(
team_id="team-under",
organization_id="org-over",
max_budget=50.0,
spend=10.0, # Team is fine
models=["gpt-4"],
)
valid_token_org = UserAPIKeyAuth(
token="sk-org-test",
team_id="team-under",
org_id="org-over",
organization_max_budget=100.0,
)
with patch("litellm.proxy.proxy_server.prisma_client") as mock_prisma:
with patch("litellm.proxy.proxy_server.user_api_key_cache") as mock_cache:
with patch("litellm.proxy.auth.auth_checks.get_org_object", new_callable=AsyncMock) as mock_get_org:
mock_get_org.return_value = org_over_budget
# Organization is over budget, should raise BudgetExceededError
with pytest.raises(litellm.BudgetExceededError) as exc_info:
await common_checks(
request_body={"model": "gpt-4"},
team_object=team_under_budget,
user_object=None,
end_user_object=None,
global_proxy_spend=None,
general_settings={},
route="/v1/chat/completions",
llm_router=None,
proxy_logging_obj=mock_proxy_logging,
valid_token=valid_token_org,
request=mock_request,
)
assert "Organization" in str(exc_info.value.message)
assert exc_info.value.current_cost == 150.0
assert exc_info.value.max_budget == 100.0

29
tests/test_litellm/proxy/common_utils/test_custom_openapi_spec.py
View file

@ -90,6 +90,35 @@ class TestCustomOpenAPISpec:
)
assert result == base_openapi_schema
def test_defs_rewritten_in_add_schema_to_components():
"""
Test that defs are rewritten to components/schemas in add_schema_to_components.
"""
openapi_schema = {}
schema_name = "SchemaName"
schema_def = {
"type": "object",
"properties": {
"messages": {
"type": "array",
"items": {
"anyOf": [
{"$ref": "#/$defs/UserMessage"},
{"$ref": "#/$defs/AssistantMessage"}
]
}
}
},
"$defs": {
"UserMessage": {"type": "object"},
"AssistantMessage": {"type": "object"}
}
}
CustomOpenAPISpec.add_schema_to_components(openapi_schema=openapi_schema, schema_name=schema_name, schema_def=schema_def)
assert "$defs" not in openapi_schema
assert openapi_schema["components"]["schemas"]["SchemaName"]["properties"]["messages"]["items"]["anyOf"][0]["$ref"] == "#/components/schemas/UserMessage"
assert openapi_schema["components"]["schemas"]["SchemaName"]["properties"]["messages"]["items"]["anyOf"][1]["$ref"] == "#/components/schemas/AssistantMessage"
def test_move_defs_to_components():
"""

Some files were not shown because too many files have changed in this diff Show more