scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-11 22:51:28 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 154

Merge branch 'redis_semantic_cache_v2' of https://github.com/fcenedes/litellm into redis_semantic_cache_v2

Browse source
This commit is contained in:
fcenedes 2025-12-04 20:25:55 +01:00
commit 71501d5b7a
298 changed files with 19073 additions and 4294 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
.circleci/config.yml
View file

@ -3496,8 +3496,13 @@ jobs:
command: |
npx playwright test e2e_ui_tests/ --reporter=html --output=test-results
no_output_timeout: 120m
- store_test_results:
- store_artifacts:
path: test-results
destination: playwright-results
- store_artifacts:
path: playwright-report
destination: playwright-report
test_nonroot_image:
machine:

15
Dockerfile
View file

@ -1,8 +1,8 @@
# Base image for building
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/python:latest-dev
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base
# Runtime image
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/python:latest-dev
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base
# Builder stage
FROM $LITELLM_BUILD_IMAGE AS builder
@ -12,11 +12,9 @@ WORKDIR /app
USER root
# Install build dependencies
RUN apk add --no-cache gcc python3-dev openssl openssl-dev
RUN apk add --no-cache bash gcc py3-pip python3 python3-dev openssl openssl-dev
RUN pip install --upgrade pip>=24.3.1 && \
pip install build
RUN python -m pip install build
# Copy the current directory contents into the container at /app
COPY . .
@ -48,10 +46,7 @@ FROM $LITELLM_RUNTIME_IMAGE AS runtime
USER root
# Install runtime dependencies
RUN apk add --no-cache openssl tzdata nodejs npm
# Upgrade pip to fix CVE-2025-8869
RUN pip install --upgrade pip>=24.3.1
RUN apk add --no-cache bash openssl tzdata nodejs npm python3 py3-pip
WORKDIR /app
# Copy the current directory contents into the container at /app

2
README.md
View file

@ -348,7 +348,7 @@ curl 'http://0.0.0.0:4000/key/generate' \
| [Fireworks AI (`fireworks_ai`)](https://docs.litellm.ai/docs/providers/fireworks_ai) | ✅ | ✅ | ✅ | | | | | | | |
| [FriendliAI (`friendliai`)](https://docs.litellm.ai/docs/providers/friendliai) | ✅ | ✅ | ✅ | | | | | | | |
| [Galadriel (`galadriel`)](https://docs.litellm.ai/docs/providers/galadriel) | ✅ | ✅ | ✅ | | | | | | | |
| [GitHub Copilot (`github_copilot`)](https://docs.litellm.ai/docs/providers/github_copilot) | ✅ | ✅ | ✅ | | | | | | | |
| [GitHub Copilot (`github_copilot`)](https://docs.litellm.ai/docs/providers/github_copilot) | ✅ | ✅ | ✅ | ✅ | | | | | | |
| [GitHub Models (`github`)](https://docs.litellm.ai/docs/providers/github) | ✅ | ✅ | ✅ | | | | | | | |
| [Google - PaLM](https://docs.litellm.ai/docs/providers/palm) | ✅ | ✅ | ✅ | | | | | | | |
| [Google - Vertex AI (`vertex_ai`)](https://docs.litellm.ai/docs/providers/vertex) | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | |

3
cookbook/mock_guardrail_server/mock_bedrock_guardrail_server.py
View file

@ -426,9 +426,12 @@ This is a beta API. Please help us improve it.
class LitellmBasicGuardrailRequest(BaseModel):
texts: List[str]
images: Optional[List[str]] = None
tools: Optional[List[dict]] = None
request_data: Dict[str, Any] = Field(default_factory=dict)
additional_provider_specific_params: Dict[str, Any] = Field(default_factory=dict)
input_type: Literal["request", "response"]
litellm_call_id: Optional[str] = None
litellm_trace_id: Optional[str] = None
class LitellmBasicGuardrailResponse(BaseModel):

4
deploy/charts/litellm-helm/Chart.yaml
View file

@ -18,7 +18,7 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.4.8
version: 0.4.9
# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
@ -33,5 +33,5 @@ dependencies:
condition: db.deployStandalone
- name: redis
version: ">=18.0.0"
repository: oci://registry-1.docker.io/bitnamicharts
repository: oci://registry-1.docker.io/bitnamicharts
condition: redis.enabled

125
deploy/charts/litellm-helm/README.md
View file

@ -10,46 +10,48 @@
- Helm 3.8.0+
If `db.deployStandalone` is used:
- PV provisioner support in the underlying infrastructure
If `db.useStackgresOperator` is used (not yet implemented):
- The Stackgres Operator must already be installed in the Kubernetes Cluster. This chart will **not** install the operator if it is missing.
- The Stackgres Operator must already be installed in the Kubernetes Cluster. This chart will **not** install the operator if it is missing.
## Parameters
### LiteLLM Proxy Deployment Settings
| Name | Description | Value |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
| `replicaCount` | The number of LiteLLM Proxy pods to be deployed | `1` |
| `masterkeySecretName` | The name of the Kubernetes Secret that contains the Master API Key for LiteLLM. If not specified, use the generated secret name. | N/A |
| `masterkeySecretKey` | The key within the Kubernetes Secret that contains the Master API Key for LiteLLM. If not specified, use `masterkey` as the key. | N/A |
| `masterkey` | The Master API Key for LiteLLM. If not specified, a random key in the `sk-...` format is generated. | N/A |
| `environmentSecrets` | An optional array of Secret object names. The keys and values in these secrets will be presented to the LiteLLM proxy pod as environment variables. See below for an example Secret object. | `[]` |
| `environmentConfigMaps` | An optional array of ConfigMap object names. The keys and values in these configmaps will be presented to the LiteLLM proxy pod as environment variables. See below for an example Secret object. | `[]` |
| `image.repository` | LiteLLM Proxy image repository | `ghcr.io/berriai/litellm` |
| `image.pullPolicy` | LiteLLM Proxy image pull policy | `IfNotPresent` |
| `image.tag` | Overrides the image tag whose default the latest version of LiteLLM at the time this chart was published. | `""` |
| `imagePullSecrets` | Registry credentials for the LiteLLM and initContainer images. | `[]` |
| `serviceAccount.create` | Whether or not to create a Kubernetes Service Account for this deployment. The default is `false` because LiteLLM has no need to access the Kubernetes API. | `false` |
| `service.type` | Kubernetes Service type (e.g. `LoadBalancer`, `ClusterIP`, etc.) | `ClusterIP` |
| `service.port` | TCP port that the Kubernetes Service will listen on. Also the TCP port within the Pod that the proxy will listen on. | `4000` |
| `service.loadBalancerClass` | Optional LoadBalancer implementation class (only used when `service.type` is `LoadBalancer`) | `""` |
| `ingress.*` | See [values.yaml](./values.yaml) for example settings | N/A |
| `proxyConfigMap.create` | When `true`, render a ConfigMap from `.Values.proxy_config` and mount it. | `true` |
| `proxyConfigMap.name` | When `create=false`, name of the existing ConfigMap to mount. | `""` |
| `proxyConfigMap.key` | Key in the ConfigMap that contains the proxy config file. | `"config.yaml"` |
| `proxy_config.*` | See [values.yaml](./values.yaml) for default settings. Rendered into the ConfigMap’s `config.yaml` only when `proxyConfigMap.create=true`. See [example_config_yaml](../../../litellm/proxy/example_config_yaml/) for configuration examples. | `N/A` |
| `extraContainers[]` | An array of additional containers to be deployed as sidecars alongside the LiteLLM Proxy.
| `pdb.enabled` | Enable a PodDisruptionBudget for the LiteLLM proxy Deployment | `false` |
| `pdb.minAvailable` | Minimum number/percentage of pods that must be available during **voluntary** disruptions (choose **one** of minAvailable/maxUnavailable) | `null` |
| `pdb.maxUnavailable` | Maximum number/percentage of pods that can be unavailable during **voluntary** disruptions (choose **one** of minAvailable/maxUnavailable) | `null` |
| `pdb.annotations` | Extra metadata annotations to add to the PDB | `{}` |
| `pdb.labels` | Extra metadata labels to add to the PDB | `{}` |
| Name | Description | Value |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| `replicaCount` | The number of LiteLLM Proxy pods to be deployed | `1` |
| `masterkeySecretName` | The name of the Kubernetes Secret that contains the Master API Key for LiteLLM. If not specified, use the generated secret name. | N/A |
| `masterkeySecretKey` | The key within the Kubernetes Secret that contains the Master API Key for LiteLLM. If not specified, use `masterkey` as the key. | N/A |
| `masterkey` | The Master API Key for LiteLLM. If not specified, a random key in the `sk-...` format is generated. | N/A |
| `environmentSecrets` | An optional array of Secret object names. The keys and values in these secrets will be presented to the LiteLLM proxy pod as environment variables. See below for an example Secret object. | `[]` |
| `environmentConfigMaps` | An optional array of ConfigMap object names. The keys and values in these configmaps will be presented to the LiteLLM proxy pod as environment variables. See below for an example Secret object. | `[]` |
| `image.repository` | LiteLLM Proxy image repository | `ghcr.io/berriai/litellm` |
| `image.pullPolicy` | LiteLLM Proxy image pull policy | `IfNotPresent` |
| `image.tag` | Overrides the image tag whose default the latest version of LiteLLM at the time this chart was published. | `""` |
| `imagePullSecrets` | Registry credentials for the LiteLLM and initContainer images. | `[]` |
| `serviceAccount.create` | Whether or not to create a Kubernetes Service Account for this deployment. The default is `false` because LiteLLM has no need to access the Kubernetes API. | `false` |
| `service.type` | Kubernetes Service type (e.g. `LoadBalancer`, `ClusterIP`, etc.) | `ClusterIP` |
| `service.port` | TCP port that the Kubernetes Service will listen on. Also the TCP port within the Pod that the proxy will listen on. | `4000` |
| `service.loadBalancerClass` | Optional LoadBalancer implementation class (only used when `service.type` is `LoadBalancer`) | `""` |
| `ingress.labels` | Additional labels for the Ingress resource | `{}` |
| `ingress.*` | See [values.yaml](./values.yaml) for example settings | N/A |
| `proxyConfigMap.create` | When `true`, render a ConfigMap from `.Values.proxy_config` and mount it. | `true` |
| `proxyConfigMap.name` | When `create=false`, name of the existing ConfigMap to mount. | `""` |
| `proxyConfigMap.key` | Key in the ConfigMap that contains the proxy config file. | `"config.yaml"` |
| `proxy_config.*` | See [values.yaml](./values.yaml) for default settings. Rendered into the ConfigMap’s `config.yaml` only when `proxyConfigMap.create=true`. See [example_config_yaml](../../../litellm/proxy/example_config_yaml/) for configuration examples. | `N/A` |
| `extraContainers[]` | An array of additional containers to be deployed as sidecars alongside the LiteLLM Proxy. |
| `pdb.enabled` | Enable a PodDisruptionBudget for the LiteLLM proxy Deployment | `false` |
| `pdb.minAvailable` | Minimum number/percentage of pods that must be available during **voluntary** disruptions (choose **one** of minAvailable/maxUnavailable) | `null` |
| `pdb.maxUnavailable` | Maximum number/percentage of pods that can be unavailable during **voluntary** disruptions (choose **one** of minAvailable/maxUnavailable) | `null` |
| `pdb.annotations` | Extra metadata annotations to add to the PDB | `{}` |
| `pdb.labels` | Extra metadata labels to add to the PDB | `{}` |
#### Example `proxy_config` ConfigMap from values (default):
```
proxyConfigMap:
create: true
@ -67,7 +69,6 @@ proxy_config:
#### Example using existing `proxyConfigMap` instead of creating it:
```
proxyConfigMap:
create: false
@ -77,8 +78,7 @@ proxyConfigMap:
# proxy_config is ignored in this mode
```
#### Example `environmentSecrets` Secret
#### Example `environmentSecrets` Secret
```
apiVersion: v1
@ -91,21 +91,23 @@ type: Opaque
```
### Database Settings
| Name | Description | Value |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
| `db.useExisting` | Use an existing Postgres database. A Kubernetes Secret object must exist that contains credentials for connecting to the database. An example secret object definition is provided below. | `false` |
| `db.endpoint` | If `db.useExisting` is `true`, this is the IP, Hostname or Service Name of the Postgres server to connect to. | `localhost` |
| `db.database` | If `db.useExisting` is `true`, the name of the existing database to connect to. | `litellm` |
| `db.url` | If `db.useExisting` is `true`, the connection url of the existing database to connect to can be overwritten with this value. | `postgresql://$(DATABASE_USERNAME):$(DATABASE_PASSWORD)@$(DATABASE_HOST)/$(DATABASE_NAME)` |
| `db.secret.name` | If `db.useExisting` is `true`, the name of the Kubernetes Secret that contains credentials. | `postgres` |
| `db.secret.usernameKey` | If `db.useExisting` is `true`, the name of the key within the Kubernetes Secret that holds the username for authenticating with the Postgres instance. | `username` |
| `db.secret.passwordKey` | If `db.useExisting` is `true`, the name of the key within the Kubernetes Secret that holds the password associates with the above user. | `password` |
| `db.useStackgresOperator` | Not yet implemented. | `false` |
| `db.deployStandalone` | Deploy a standalone, single instance deployment of Postgres, using the Bitnami postgresql chart. This is useful for getting started but doesn't provide HA or (by default) data backups. | `true` |
| `postgresql.*` | If `db.deployStandalone` is `true`, configuration passed to the Bitnami postgresql chart. See the [Bitnami Documentation](https://github.com/bitnami/charts/tree/main/bitnami/postgresql) for full configuration details. See [values.yaml](./values.yaml) for the default configuration. | See [values.yaml](./values.yaml) |
| `postgresql.auth.*` | If `db.deployStandalone` is `true`, care should be taken to ensure the default `password` and `postgres-password` values are **NOT** used. | `NoTaGrEaTpAsSwOrD` |
| Name | Description | Value |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| `db.useExisting` | Use an existing Postgres database. A Kubernetes Secret object must exist that contains credentials for connecting to the database. An example secret object definition is provided below. | `false` |
| `db.endpoint` | If `db.useExisting` is `true`, this is the IP, Hostname or Service Name of the Postgres server to connect to. | `localhost` |
| `db.database` | If `db.useExisting` is `true`, the name of the existing database to connect to. | `litellm` |
| `db.url` | If `db.useExisting` is `true`, the connection url of the existing database to connect to can be overwritten with this value. | `postgresql://$(DATABASE_USERNAME):$(DATABASE_PASSWORD)@$(DATABASE_HOST)/$(DATABASE_NAME)` |
| `db.secret.name` | If `db.useExisting` is `true`, the name of the Kubernetes Secret that contains credentials. | `postgres` |
| `db.secret.usernameKey` | If `db.useExisting` is `true`, the name of the key within the Kubernetes Secret that holds the username for authenticating with the Postgres instance. | `username` |
| `db.secret.passwordKey` | If `db.useExisting` is `true`, the name of the key within the Kubernetes Secret that holds the password associates with the above user. | `password` |
| `db.useStackgresOperator` | Not yet implemented. | `false` |
| `db.deployStandalone` | Deploy a standalone, single instance deployment of Postgres, using the Bitnami postgresql chart. This is useful for getting started but doesn't provide HA or (by default) data backups. | `true` |
| `postgresql.*` | If `db.deployStandalone` is `true`, configuration passed to the Bitnami postgresql chart. See the [Bitnami Documentation](https://github.com/bitnami/charts/tree/main/bitnami/postgresql) for full configuration details. See [values.yaml](./values.yaml) for the default configuration. | See [values.yaml](./values.yaml) |
| `postgresql.auth.*` | If `db.deployStandalone` is `true`, care should be taken to ensure the default `password` and `postgres-password` values are **NOT** used. | `NoTaGrEaTpAsSwOrD` |
#### Example Postgres `db.useExisting` Secret
```yaml
apiVersion: v1
kind: Secret
@ -143,7 +145,7 @@ metadata:
name: litellm-env-secret
type: Opaque
data:
SOME_PASSWORD: cDZbUGVXeU5e0ZW # base64 encoded
SOME_PASSWORD: cDZbUGVXeU5e0ZW # base64 encoded
ANOTHER_PASSWORD: AAZbUGVXeU5e0ZB # base64 encoded
```
@ -153,23 +155,23 @@ Source: [GitHub Gist from troyharvey](https://gist.github.com/troyharvey/4506472
The migration job supports both ArgoCD and Helm hooks to ensure database migrations run at the appropriate time during deployments.
| Name | Description | Value |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
| `migrationJob.enabled` | Enable or disable the schema migration Job | `true` |
| `migrationJob.backoffLimit` | Backoff limit for Job restarts | `4` |
| `migrationJob.ttlSecondsAfterFinished` | TTL for completed migration jobs | `120` |
| `migrationJob.annotations` | Additional annotations for the migration job pod | `{}` |
| `migrationJob.extraContainers` | Additional containers to run alongside the migration job | `[]` |
| `migrationJob.hooks.argocd.enabled` | Enable ArgoCD hooks for the migration job (uses PreSync hook with BeforeHookCreation delete policy) | `true` |
| `migrationJob.hooks.helm.enabled` | Enable Helm hooks for the migration job (uses pre-install,pre-upgrade hooks with before-hook-creation delete policy) | `false` |
| `migrationJob.hooks.helm.weight` | Helm hook execution order (lower weights executed first). Optional - defaults to "1" if not specified. | N/A |
| Name | Description | Value |
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ------- |
| `migrationJob.enabled` | Enable or disable the schema migration Job | `true` |
| `migrationJob.backoffLimit` | Backoff limit for Job restarts | `4` |
| `migrationJob.ttlSecondsAfterFinished` | TTL for completed migration jobs | `120` |
| `migrationJob.annotations` | Additional annotations for the migration job pod | `{}` |
| `migrationJob.extraContainers` | Additional containers to run alongside the migration job | `[]` |
| `migrationJob.hooks.argocd.enabled` | Enable ArgoCD hooks for the migration job (uses PreSync hook with BeforeHookCreation delete policy) | `true` |
| `migrationJob.hooks.helm.enabled` | Enable Helm hooks for the migration job (uses pre-install,pre-upgrade hooks with before-hook-creation delete policy) | `false` |
| `migrationJob.hooks.helm.weight` | Helm hook execution order (lower weights executed first). Optional - defaults to "1" if not specified. | N/A |
## Accessing the Admin UI
When browsing to the URL published per the settings in `ingress.*`, you will
be prompted for **Admin Configuration**. The **Proxy Endpoint** is the internal
be prompted for **Admin Configuration**. The **Proxy Endpoint** is the internal
(from the `litellm` pod's perspective) URL published by the `<RELEASE>-litellm`
Kubernetes Service. If the deployment uses the default settings for this
Kubernetes Service. If the deployment uses the default settings for this
service, the **Proxy Endpoint** should be set to `http://<RELEASE>-litellm:4000`.
The **Proxy Key** is the value specified for `masterkey` or, if a `masterkey`
@ -181,7 +183,8 @@ kubectl -n litellm get secret <RELEASE>-litellm-masterkey -o jsonpath="{.data.ma
```
## Admin UI Limitations
At the time of writing, the Admin UI is unable to add models. This is because
At the time of writing, the Admin UI is unable to add models. This is because
it would need to update the `config.yaml` file which is a exposed ConfigMap, and
therefore, read-only. This is a limitation of this helm chart, not the Admin UI
therefore, read-only. This is a limitation of this helm chart, not the Admin UI
itself.

3
deploy/charts/litellm-helm/templates/ingress.yaml
View file

@ -18,6 +18,9 @@ metadata:
name: {{ $fullName }}
labels:
{{- include "litellm.labels" . | nindent 4 }}
{{- with .Values.ingress.labels }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}

45
deploy/charts/litellm-helm/tests/ingress_tests.yaml Normal file
View file

@ -0,0 +1,45 @@
suite: Ingress Configuration Tests
templates:
- ingress.yaml
tests:
- it: should not create Ingress by default
asserts:
- hasDocuments:
count: 0
- it: should create Ingress when enabled
set:
ingress.enabled: true
asserts:
- hasDocuments:
count: 1
- isKind:
of: Ingress
- it: should add custom labels
set:
ingress.enabled: true
ingress.labels:
custom-label: "true"
another-label: "value"
asserts:
- isKind:
of: Ingress
- equal:
path: metadata.labels.custom-label
value: "true"
- equal:
path: metadata.labels.another-label
value: "value"
- it: should add annotations
set:
ingress.enabled: true
ingress.annotations:
kubernetes.io/ingress.class: "nginx"
asserts:
- isKind:
of: Ingress
- equal:
path: metadata.annotations["kubernetes.io/ingress.class"]
value: "nginx"

45
deploy/charts/litellm-helm/values.yaml
View file

@ -35,7 +35,8 @@ podAnnotations: {}
podLabels: {}
terminationGracePeriodSeconds: 90
topologySpreadConstraints: []
topologySpreadConstraints:
[]
# - maxSkew: 1
# topologyKey: kubernetes.io/hostname
# whenUnsatisfiable: DoNotSchedule
@ -46,7 +47,8 @@ topologySpreadConstraints: []
# At the time of writing, the litellm docker image requires write access to the
# filesystem on startup so that prisma can install some dependencies.
podSecurityContext: {}
securityContext: {}
securityContext:
{}
# capabilities:
# drop:
# - ALL
@ -57,13 +59,15 @@ securityContext: {}
# A list of Kubernetes Secret objects that will be exported to the LiteLLM proxy
# pod as environment variables. These secrets can then be referenced in the
# configuration file (or "litellm" ConfigMap) with `os.environ/<Env Var Name>`
environmentSecrets: []
environmentSecrets:
[]
# - litellm-env-secret
# A list of Kubernetes ConfigMap objects that will be exported to the LiteLLM proxy
# pod as environment variables. The ConfigMap kv-pairs can then be referenced in the
# configuration file (or "litellm" ConfigMap) with `os.environ/<Env Var Name>`
environmentConfigMaps: []
environmentConfigMaps:
[]
# - litellm-env-configmap
service:
@ -82,7 +86,9 @@ separateHealthPort: 8081
ingress:
enabled: false
className: "nginx"
annotations: {}
labels: {}
annotations:
{}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts:
@ -129,7 +135,8 @@ proxy_config:
general_settings:
master_key: os.environ/PROXY_MASTER_KEY
resources: {}
resources:
{}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
@ -231,7 +238,7 @@ migrationJob:
# cpu: 100m
# memory: 100Mi
extraContainers: []
# Hook configuration
hooks:
argocd:
@ -240,30 +247,30 @@ migrationJob:
enabled: false
# Additional environment variables to be added to the deployment as a map of key-value pairs
envVars: {
# USE_DDTRACE: "true"
}
envVars: {}
# USE_DDTRACE: "true"
# Additional environment variables to be added to the deployment as a list of k8s env vars
extraEnvVars: {
# - name: EXTRA_ENV_VAR
# value: EXTRA_ENV_VAR_VALUE
}
extraEnvVars: {}
# - name: EXTRA_ENV_VAR
# value: EXTRA_ENV_VAR_VALUE
# Pod Disruption Budget
pdb:
enabled: false
# Set exactly one of the following. If both are set, minAvailable takes precedence.
minAvailable: null # e.g. "50%" or 1
maxUnavailable: null # e.g. 1 or "20%"
minAvailable: null # e.g. "50%" or 1
maxUnavailable: null # e.g. 1 or "20%"
annotations: {}
labels: {}
serviceMonitor:
enabled: false
labels: {}
labels:
{}
# test: test
annotations: {}
annotations:
{}
# kubernetes.io/test: test
interval: 15s
scrapeTimeout: 10s
@ -273,4 +280,4 @@ serviceMonitor:
# action: replace
namespaceSelector:
matchNames: []
# - test-namespace
# - test-namespace

16
docker/Dockerfile.database
View file

@ -1,8 +1,8 @@
# Base image for building
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/python:latest-dev
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base
# Runtime image
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/python:latest-dev
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base
# Builder stage
FROM $LITELLM_BUILD_IMAGE AS builder
@ -13,13 +13,15 @@ USER root
# Install build dependencies
RUN apk add --no-cache \
build-base \
bash \
gcc \
py3-pip \
python3 \
python3-dev \
openssl \
openssl-dev
RUN pip install --upgrade pip && \
pip install build
RUN python -m pip install build
# Copy the current directory contents into the container at /app
COPY . .
@ -46,7 +48,7 @@ FROM $LITELLM_RUNTIME_IMAGE AS runtime
USER root
# Install runtime dependencies
RUN apk add --no-cache openssl
RUN apk add --no-cache bash openssl tzdata nodejs npm python3 py3-pip
WORKDIR /app
# Copy the current directory contents into the container at /app

19
docker/Dockerfile.non_root
View file

@ -1,6 +1,6 @@
# Base images
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/python:latest-dev
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/python:latest-dev
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base
# -----------------
# Builder Stage
@ -10,7 +10,18 @@ WORKDIR /app
# Install build dependencies including Node.js for UI build
USER root
RUN apk add --no-cache build-base bash nodejs npm \
RUN apk add --no-cache \
python3 \
py3-pip \
clang \
llvm \
lld \
gcc \
linux-headers \
build-base \
bash \
nodejs \
npm \
&& pip install --no-cache-dir --upgrade pip build
# Copy project files
@ -62,7 +73,7 @@ WORKDIR /app
# Install runtime dependencies
USER root
RUN apk upgrade --no-cache && \
apk add --no-cache bash libstdc++ ca-certificates openssl supervisor
apk add --no-cache python3 py3-pip bash openssl tzdata nodejs npm supervisor
# Copy only necessary artifacts from builder stage for runtime
COPY . .

204
docs/my-website/docs/a2a.md Normal file
View file

@ -0,0 +1,204 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import Image from '@theme/IdealImage';
# /a2a - Agent Gateway (A2A Protocol)
| Feature | Supported |
|---------|-----------|
| Logging | ✅ |
| Load Balancing | ✅ |
| Streaming | ✅ |
:::tip
LiteLLM follows the [A2A (Agent-to-Agent) Protocol](https://github.com/google/A2A) for invoking agents.
:::
## Adding your Agent
You can add A2A-compatible agents through the LiteLLM Admin UI.
1. Navigate to the **Agents** tab
2. Click **Add Agent**
3. Enter the agent name (e.g., `ij-local`) and the URL of your A2A agent
<Image
img={require('../img/add_agent_1.png')}
style={{width: '80%', display: 'block', margin: '0'}}
/>
The URL should be the invocation URL for your A2A agent (e.g., `http://localhost:10001`).
## Invoking your Agents
Use the [A2A Python SDK](https://pypi.org/project/a2a/) to invoke agents through LiteLLM:
- `base_url`: Your LiteLLM proxy URL + `/a2a/{agent_name}`
- `headers`: Include your LiteLLM Virtual Key for authentication
```python showLineNumbers title="invoke_a2a_agent.py"
from uuid import uuid4
import httpx
import asyncio
from a2a.client import A2ACardResolver, A2AClient
from a2a.types import MessageSendParams, SendMessageRequest
# === CONFIGURE THESE ===
LITELLM_BASE_URL = "http://localhost:4000" # Your LiteLLM proxy URL
LITELLM_VIRTUAL_KEY = "sk-1234" # Your LiteLLM Virtual Key
LITELLM_AGENT_NAME = "ij-local" # Agent name registered in LiteLLM
# =======================
async def main():
base_url = f"{LITELLM_BASE_URL}/a2a/{LITELLM_AGENT_NAME}"
headers = {"Authorization": f"Bearer {LITELLM_VIRTUAL_KEY}"}
async with httpx.AsyncClient(headers=headers) as httpx_client:
# Resolve agent card and create client
resolver = A2ACardResolver(httpx_client=httpx_client, base_url=base_url)
agent_card = await resolver.get_agent_card()
client = A2AClient(httpx_client=httpx_client, agent_card=agent_card)
# Send a message
request = SendMessageRequest(
id=str(uuid4()),
params=MessageSendParams(
message={
"role": "user",
"parts": [{"kind": "text", "text": "Hello, what can you do?"}],
"messageId": uuid4().hex,
}
),
)
response = await client.send_message(request)
print(response.model_dump(mode="json", exclude_none=True))
if __name__ == "__main__":
asyncio.run(main())
```
### Streaming Responses
For streaming responses, use `send_message_streaming`:
```python showLineNumbers title="invoke_a2a_agent_streaming.py"
from uuid import uuid4
import httpx
import asyncio
from a2a.client import A2ACardResolver, A2AClient
from a2a.types import MessageSendParams, SendStreamingMessageRequest
# === CONFIGURE THESE ===
LITELLM_BASE_URL = "http://localhost:4000" # Your LiteLLM proxy URL
LITELLM_VIRTUAL_KEY = "sk-1234" # Your LiteLLM Virtual Key
LITELLM_AGENT_NAME = "ij-local" # Agent name registered in LiteLLM
# =======================
async def main():
base_url = f"{LITELLM_BASE_URL}/a2a/{LITELLM_AGENT_NAME}"
headers = {"Authorization": f"Bearer {LITELLM_VIRTUAL_KEY}"}
async with httpx.AsyncClient(headers=headers) as httpx_client:
# Resolve agent card and create client
resolver = A2ACardResolver(httpx_client=httpx_client, base_url=base_url)
agent_card = await resolver.get_agent_card()
client = A2AClient(httpx_client=httpx_client, agent_card=agent_card)
# Send a streaming message
request = SendStreamingMessageRequest(
id=str(uuid4()),
params=MessageSendParams(
message={
"role": "user",
"parts": [{"kind": "text", "text": "Hello, what can you do?"}],
"messageId": uuid4().hex,
}
),
)
# Stream the response
async for chunk in client.send_message_streaming(request):
print(chunk.model_dump(mode="json", exclude_none=True))
if __name__ == "__main__":
asyncio.run(main())
```
## Tracking Agent Logs
After invoking an agent, you can view the request logs in the LiteLLM **Logs** tab.
The logs show:
- **Request/Response content** sent to and received from the agent
- **User, Key, Team** information for tracking who made the request
- **Latency and cost** metrics
<Image
img={require('../img/agent2.png')}
style={{width: '100%', display: 'block', margin: '2rem auto'}}
/>
## API Reference
### Endpoint
```
POST /a2a/{agent_name}/message/send
```
### Authentication
Include your LiteLLM Virtual Key in the `Authorization` header:
```
Authorization: Bearer sk-your-litellm-key
```
### Request Format
LiteLLM follows the [A2A JSON-RPC 2.0 specification](https://github.com/google/A2A):
```json title="Request Body"
{
"jsonrpc": "2.0",
"id": "unique-request-id",
"method": "message/send",
"params": {
"message": {
"role": "user",
"parts": [{"kind": "text", "text": "Your message here"}],
"messageId": "unique-message-id"
}
}
}
```
### Response Format
```json title="Response"
{
"jsonrpc": "2.0",
"id": "unique-request-id",
"result": {
"kind": "task",
"id": "task-id",
"contextId": "context-id",
"status": {"state": "completed", "timestamp": "2025-01-01T00:00:00Z"},
"artifacts": [
{
"artifactId": "artifact-id",
"name": "response",
"parts": [{"kind": "text", "text": "Agent response here"}]
}
]
}
}
```
## Agent Registry
Want to create a central registry so your team can discover what agents are available within your company?
Use the [AI Hub](./proxy/ai_hub) to make agents public and discoverable across your organization. This allows developers to browse available agents without needing to rebuild them.

87
docs/my-website/docs/adding_provider/generic_guardrail_api.md
View file

@ -21,6 +21,20 @@ The **Generic Guardrail API** lets you integrate with LiteLLM **instantly** by i
5. **Custom Parameters** - Pass provider-specific params via config
6. **Full Control** - You own and maintain your guardrail API
## Supported Endpoints
The Generic Guardrail API works with the following LiteLLM endpoints:
- `/v1/chat/completions` - OpenAI Chat Completions
- `/v1/completions` - OpenAI Text Completions
- `/v1/responses` - OpenAI Responses API
- `/v1/images/generations` - OpenAI Image Generation
- `/v1/audio/transcriptions` - OpenAI Audio Transcriptions
- `/v1/audio/speech` - OpenAI Text-to-Speech
- `/v1/messages` - Anthropic Messages
- `/v1/rerank` - Cohere Rerank
- Pass-through endpoints
## How It Works
1. LiteLLM extracts text and images from any request (chat messages, embeddings, image prompts, etc.)
@ -40,6 +54,21 @@ Implement `POST /beta/litellm_basic_guardrail_api`
{
"texts": ["extracted text from the request"], // array of text strings
"images": ["base64_encoded_image_data"], // optional array of images
"tools": [ // optional array of tools (OpenAI ChatCompletionToolParam format)
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string"}
}
}
}
}
],
"request_data": {
"user_api_key_hash": "hash of the litellm virtual key used",
"user_api_key_alias": "alias of the litellm virtual key used",
@ -75,6 +104,49 @@ Implement `POST /beta/litellm_basic_guardrail_api`
- `NONE` - Request proceeds unchanged
- `GUARDRAIL_INTERVENED` - Request proceeds with modified texts/images (provide `texts` and/or `images` fields)
## Parameters
### `tools` Parameter
The `tools` parameter provides information about available function/tool definitions in the request.
**Format:** OpenAI `ChatCompletionToolParam` format (see [OpenAI API reference](https://platform.openai.com/docs/api-reference/chat/create#chat-create-tools))
**Example:**
```json
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather in a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "City and state, e.g. San Francisco, CA"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"]
}
},
"required": ["location"]
}
}
}
```
**Limitations:**
- **Input only:** Tools are only passed for `input_type="request"` (pre-call guardrails). Output/response guardrails do not currently receive tool information.
- **Supported endpoints:** The `tools` parameter is supported on: `/v1/chat/completions`, `/v1/responses`, and `/v1/messages`. Other endpoints do not have tool support.
**Use cases:**
- Enforce tool permission policies (e.g., only allow certain users/teams to access specific tools)
- Validate tool schemas before sending to LLM
- Log tool usage for audit purposes
- Block sensitive tools based on user context
## LiteLLM Configuration
Add to `config.yaml`:
@ -138,6 +210,7 @@ app = FastAPI()
class GuardrailRequest(BaseModel):
texts: List[str]
images: Optional[List[str]] = None
tools: Optional[List[Dict[str, Any]]] = None # OpenAI ChatCompletionToolParam format
request_data: Dict[str, Any]
input_type: str # "request" or "response"
litellm_call_id: Optional[str] = None
@ -153,6 +226,8 @@ class GuardrailResponse(BaseModel):
@app.post("/beta/litellm_basic_guardrail_api")
async def apply_guardrail(request: GuardrailRequest):
# Your guardrail logic here
# Example: Check text content
for text in request.texts:
if "badword" in text.lower():
return GuardrailResponse(
@ -160,6 +235,18 @@ async def apply_guardrail(request: GuardrailRequest):
blocked_reason="Content contains prohibited terms"
)
# Example: Check tools (if present in request)
if request.tools:
for tool in request.tools:
if tool.get("type") == "function":
function_name = tool.get("function", {}).get("name", "")
# Block sensitive tools
if function_name in ["delete_data", "access_admin_panel"]:
return GuardrailResponse(
action="BLOCKED",
blocked_reason=f"Tool '{function_name}' is not allowed"
)
return GuardrailResponse(action="NONE")
```

1
docs/my-website/docs/completion/knowledgebase.md
View file

@ -21,6 +21,7 @@ LiteLLM integrates with vector stores, allowing your models to access your organ
- [Azure Vector Stores](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/file-search?tabs=python#vector-stores) (Cannot be directly queried. Only available for calling in Assistants messages. We will be adding Azure AI Search Vector Store API support soon.)
- [Vertex AI RAG API](https://cloud.google.com/vertex-ai/generative-ai/docs/rag-overview)
- [Gemini File Search](https://ai.google.dev/gemini-api/docs/file-search)
- [RAGFlow Datasets](/docs/providers/ragflow_vector_store.md) (Dataset management only, search not supported)
## Quick Start

13
docs/my-website/docs/files_endpoints.md
View file

@ -301,6 +301,17 @@ content = await litellm.afile_content(
print("file content=", content)
```
**Get File Content (Bedrock)**
```python
# For Bedrock batch output files stored in S3
content = await litellm.afile_content(
file_id="s3://bucket-name/path/to/file.jsonl", # S3 URI or unified file ID
custom_llm_provider="bedrock",
aws_region_name="us-west-2"
)
print("file content=", content.text)
```
</TabItem>
</Tabs>
@ -313,4 +324,6 @@ print("file content=", content)
### [Vertex AI](./providers/vertex#batch-apis)
### [Bedrock](./providers/bedrock_batches#4-retrieve-batch-results)
## [Swagger API Reference](https://litellm-api.up.railway.app/#/files)

108
docs/my-website/docs/getting_started.md
View file

@ -1,108 +0,0 @@
# Getting Started
import QuickStart from '../src/components/QuickStart.js'
LiteLLM simplifies LLM API calls by mapping them all to the [OpenAI ChatCompletion format](https://platform.openai.com/docs/api-reference/chat).
## basic usage
By default we provide a free $10 community-key to try all providers supported on LiteLLM.
```python
from litellm import completion
## set ENV variables
os.environ["OPENAI_API_KEY"] = "your-api-key"
os.environ["COHERE_API_KEY"] = "your-api-key"
messages = [{ "content": "Hello, how are you?","role": "user"}]
# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)
# cohere call
response = completion("command-nightly", messages)
```
**Need a dedicated key?**
Email us @ krrish@berri.ai
Next Steps 👉 [Call all supported models - e.g. Claude-2, Llama2-70b, etc.](./proxy_api.md#supported-models)
More details 👉
- [Completion() function details](./completion/)
- [Overview of supported models / providers on LiteLLM](./providers/)
- [Search all models / providers](https://models.litellm.ai/)
- [Build your own OpenAI proxy](https://github.com/BerriAI/liteLLM-proxy/tree/main)
## streaming
Same example from before. Just pass in `stream=True` in the completion args.
```python
from litellm import completion
## set ENV variables
os.environ["OPENAI_API_KEY"] = "openai key"
os.environ["COHERE_API_KEY"] = "cohere key"
messages = [{ "content": "Hello, how are you?","role": "user"}]
# openai call
response = completion(model="gpt-3.5-turbo", messages=messages, stream=True)
# cohere call
response = completion("command-nightly", messages, stream=True)
print(response)
```
More details 👉
- [streaming + async](./completion/stream.md)
- [tutorial for streaming Llama2 on TogetherAI](./tutorials/TogetherAI_liteLLM.md)
## exception handling
LiteLLM maps exceptions across all supported providers to the OpenAI exceptions. All our exceptions inherit from OpenAI's exception types, so any error-handling you have for that, should work out of the box with LiteLLM.
```python
from openai.error import OpenAIError
from litellm import completion
os.environ["ANTHROPIC_API_KEY"] = "bad-key"
try:
# some code
completion(model="claude-instant-1", messages=[{"role": "user", "content": "Hey, how's it going?"}])
except OpenAIError as e:
print(e)
```
## Logging Observability - Log LLM Input/Output ([Docs](https://docs.litellm.ai/docs/observability/callbacks))
LiteLLM exposes pre defined callbacks to send data to MLflow, Lunary, Langfuse, Helicone, Promptlayer, Traceloop, Slack
```python
from litellm import completion
## set env variables for logging tools (API key set up is not required when using MLflow)
os.environ["LUNARY_PUBLIC_KEY"] = "your-lunary-public-key" # get your public key at https://app.lunary.ai/settings
os.environ["HELICONE_API_KEY"] = "your-helicone-key"
os.environ["LANGFUSE_PUBLIC_KEY"] = ""
os.environ["LANGFUSE_SECRET_KEY"] = ""
os.environ["OPENAI_API_KEY"]
# set callbacks
litellm.success_callback = ["lunary", "mlflow", "langfuse", "helicone"] # log input/output to MLflow, langfuse, lunary, helicone
#openai call
response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hi 👋 - i'm openai"}])
```
More details 👉
- [exception mapping](./exception_mapping.md)
- [retries + model fallbacks for completion()](./completion/reliable_completions.md)
- [tutorial for model fallbacks with completion()](./tutorials/fallbacks.md)

18
docs/my-website/docs/observability/datadog.md
View file

@ -71,17 +71,19 @@ DD_SOURCE="litellm_dev" # [OPTIONAL] your datadog source. use to different
Send logs through a local DataDog agent (useful for containerized environments):
```shell
DD_AGENT_HOST="localhost" # hostname or IP of DataDog agent
DD_AGENT_PORT="10518" # [OPTIONAL] port of DataDog agent (default: 10518)
DD_API_KEY="5f2d0f310***********" # [OPTIONAL] your datadog API Key (agent handles auth)
DD_SOURCE="litellm_dev" # [OPTIONAL] your datadog source
LITELLM_DD_AGENT_HOST="localhost" # hostname or IP of DataDog agent
LITELLM_DD_AGENT_PORT="10518" # [OPTIONAL] port of DataDog agent (default: 10518)
DD_API_KEY="5f2d0f310***********" # [OPTIONAL] your datadog API Key (agent handles auth)
DD_SOURCE="litellm_dev" # [OPTIONAL] your datadog source
```
When `DD_AGENT_HOST` is set, logs are sent to the agent instead of directly to DataDog API. This is useful for:
When `LITELLM_DD_AGENT_HOST` is set, logs are sent to the agent instead of directly to DataDog API. This is useful for:
- Centralized log shipping in containerized environments
- Reducing direct API calls from multiple services
- Leveraging agent-side processing and filtering
**Note:** We use `LITELLM_DD_AGENT_HOST` instead of `DD_AGENT_HOST` to avoid conflicts with `ddtrace` which automatically sets `DD_AGENT_HOST` for APM tracing.
**Step 3**: Start the proxy, make a test request
Start proxy
@ -191,8 +193,8 @@ LiteLLM supports customizing the following Datadog environment variables
|---------------------|-------------|---------------|----------|
| `DD_API_KEY` | Your Datadog API key for authentication (required for direct API, optional for agent) | None | Conditional* |
| `DD_SITE` | Your Datadog site (e.g., "us5.datadoghq.com") (required for direct API) | None | Conditional* |
| `DD_AGENT_HOST` | Hostname or IP of DataDog agent (e.g., "localhost"). When set, logs are sent to agent instead of direct API | None | ❌ No |
| `DD_AGENT_PORT` | Port of DataDog agent for log intake | "10518" | ❌ No |
| `LITELLM_DD_AGENT_HOST` | Hostname or IP of DataDog agent (e.g., "localhost"). When set, logs are sent to agent instead of direct API | None | ❌ No |
| `LITELLM_DD_AGENT_PORT` | Port of DataDog agent for log intake | "10518" | ❌ No |
| `DD_ENV` | Environment tag for your logs (e.g., "production", "staging") | "unknown" | ❌ No |
| `DD_SERVICE` | Service name for your logs | "litellm-server" | ❌ No |
| `DD_SOURCE` | Source name for your logs | "litellm" | ❌ No |
@ -201,5 +203,5 @@ LiteLLM supports customizing the following Datadog environment variables
| `POD_NAME` | Pod name tag (useful for Kubernetes deployments) | "unknown" | ❌ No |
\* **Required when using Direct API** (default): `DD_API_KEY` and `DD_SITE` are required
\* **Optional when using DataDog Agent**: Set `DD_AGENT_HOST` to use agent mode; `DD_API_KEY` and `DD_SITE` are not required
\* **Optional when using DataDog Agent**: Set `LITELLM_DD_AGENT_HOST` to use agent mode; `DD_API_KEY` and `DD_SITE` are not required

79
docs/my-website/docs/observability/phoenix_integration.md
View file

@ -6,7 +6,7 @@ Open source tracing and evaluation platform
:::tip
This is community maintained, Please make an issue if you run into a bug
This is community maintained. Please make an issue if you run into a bug:
https://github.com/BerriAI/litellm
:::
@ -31,19 +31,16 @@ litellm.callbacks = ["arize_phoenix"]
import litellm
import os
os.environ["PHOENIX_API_KEY"] = "" # Necessary only using Phoenix Cloud
os.environ["PHOENIX_COLLECTOR_HTTP_ENDPOINT"] = "" # The URL of your Phoenix OSS instance e.g. http://localhost:6006/v1/traces
os.environ["PHOENIX_PROJECT_NAME"]="litellm" # OPTIONAL: you can configure project names, otherwise traces would go to "default" project
# Set env variables
os.environ["PHOENIX_API_KEY"] = "d0*****" # Set the Phoenix API key here. It is necessary only when using Phoenix Cloud.
os.environ["PHOENIX_COLLECTOR_HTTP_ENDPOINT"] = "https://app.phoenix.arize.com/s/<space-name>/v1/traces" # Set the URL of your Phoenix OSS instance, otherwise tracer would use https://app.phoenix.arize.com/v1/traces for Phoenix Cloud.
os.environ["PHOENIX_PROJECT_NAME"] = "litellm" # Configure the project name, otherwise traces would go to "default" project.
os.environ['OPENAI_API_KEY'] = "fake-key" # Set the OpenAI API key here.
# This defaults to https://app.phoenix.arize.com/v1/traces for Phoenix Cloud
# LLM API Keys
os.environ['OPENAI_API_KEY']=""
# set arize as a callback, litellm will send the data to arize
# Set arize_phoenix as a callback & LiteLLM will send the data to Phoenix.
litellm.callbacks = ["arize_phoenix"]
# openai call
# OpenAI call
response = litellm.completion(
model="gpt-3.5-turbo",
messages=[
@ -52,8 +49,9 @@ response = litellm.completion(
)
```
### Using with LiteLLM Proxy
## Using with LiteLLM Proxy
1. Setup config.yaml
```yaml
model_list:
@ -66,12 +64,63 @@ model_list:
litellm_settings:
callbacks: ["arize_phoenix"]
general_settings:
master_key: "sk-1234"
environment_variables:
PHOENIX_API_KEY: "d0*****"
PHOENIX_COLLECTOR_ENDPOINT: "https://app.phoenix.arize.com/v1/traces" # OPTIONAL, for setting the GRPC endpoint
PHOENIX_COLLECTOR_HTTP_ENDPOINT: "https://app.phoenix.arize.com/v1/traces" # OPTIONAL, for setting the HTTP endpoint
PHOENIX_COLLECTOR_ENDPOINT: "https://app.phoenix.arize.com/s/<space-name>/v1/traces" # OPTIONAL - For setting the gRPC endpoint
PHOENIX_COLLECTOR_HTTP_ENDPOINT: "https://app.phoenix.arize.com/s/<space-name>/v1/traces" # OPTIONAL - For setting the HTTP endpoint
```
2. Start the proxy
```bash
litellm --config config.yaml
```
3. Test it!
```bash
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Hi 👋 - i'm openai"}]}'
```
## Supported Phoenix Endpoints
Phoenix now supports multiple deployment types. The correct endpoint depends on which version of Phoenix Cloud you are using.
**Phoenix Cloud (With Spaces - New Version)**
Use this if your Phoenix URL contains `/s/<space-name>` path.
```bash
https://app.phoenix.arize.com/s/<space-name>/v1/traces
```
**Phoenix Cloud (Legacy - Deprecated)**
Use this only if your deployment still shows the `/legacy` pattern.
```bash
https://app.phoenix.arize.com/legacy/v1/traces
```
**Phoenix Cloud (Without Spaces - Old Version)**
Use this if your Phoenix Cloud URL does not contain `/s/<space-name>` or `/legacy` path.
```bash
https://app.phoenix.arize.com/v1/traces
```
**Self-Hosted Phoenix (Local Instance)**
Use this when running Phoenix on your machine or a private server.
```bash
http://localhost:6006/v1/traces
```
Depending on which Phoenix Cloud version or deployment you are using, you should set the corresponding endpoint in `PHOENIX_COLLECTOR_HTTP_ENDPOINT` or `PHOENIX_COLLECTOR_ENDPOINT`.
## Support & Talk to Founders
- [Schedule Demo 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)

10
docs/my-website/docs/projects/Agent Lightning.md Normal file
View file

@ -0,0 +1,10 @@
# Agent Lightning
[Agent Lightning](https://github.com/microsoft/agent-lightning) is Microsoft's open-source framework for training and optimizing AI agents with Reinforcement Learning, Automatic Prompt Optimization, and Supervised Fine-tuning — with almost zero code changes.
It works with any agent framework including LangChain, OpenAI Agents SDK, AutoGen, and CrewAI. Agent Lightning uses LiteLLM Proxy under the hood to route LLM requests and collect traces that power its training algorithms.
- [GitHub](https://github.com/microsoft/agent-lightning)
- [Docs](https://microsoft.github.io/agent-lightning/)
- [arXiv Paper](https://arxiv.org/abs/2508.03680)

21
docs/my-website/docs/projects/Google ADK.md Normal file
View file

@ -0,0 +1,21 @@
# Google ADK (Agent Development Kit)
[Google ADK](https://github.com/google/adk-python) is an open-source, code-first Python framework for building, evaluating, and deploying sophisticated AI agents. While optimized for Gemini, ADK is model-agnostic and supports LiteLLM for using 100+ providers.
```python
from google.adk.agents.llm_agent import Agent
from google.adk.models.lite_llm import LiteLlm
root_agent = Agent(
model=LiteLlm(model="openai/gpt-4o"), # Or any LiteLLM-supported model
name="my_agent",
description="An agent using LiteLLM",
instruction="You are a helpful assistant.",
tools=[your_tools],
)
```
- [GitHub](https://github.com/google/adk-python)
- [Documentation](https://google.github.io/adk-docs)
- [LiteLLM Samples](https://github.com/google/adk-python/tree/main/contributing/samples/hello_world_litellm)

24
docs/my-website/docs/projects/Harbor.md Normal file
View file

@ -0,0 +1,24 @@
# Harbor
[Harbor](https://github.com/laude-institute/harbor) is a framework from the creators of Terminal-Bench for evaluating and optimizing agents and language models. It uses LiteLLM to call 100+ LLM providers.
```bash
# Install
pip install harbor
# Run a benchmark with any LiteLLM-supported model
harbor run --dataset terminal-bench@2.0 \
--agent claude-code \
--model anthropic/claude-opus-4-1 \
--n-concurrent 4
```
Key features:
- Evaluate agents like Claude Code, OpenHands, Codex CLI
- Build and share benchmarks and environments
- Run experiments in parallel across cloud providers (Daytona, Modal)
- Generate rollouts for RL optimization
- [GitHub](https://github.com/laude-institute/harbor)
- [Documentation](https://harborframework.com/docs)

24
docs/my-website/docs/providers/anthropic.md
View file

@ -201,6 +201,30 @@ Without `LITELLM_ANTHROPIC_DISABLE_URL_SUFFIX`:
With `LITELLM_ANTHROPIC_DISABLE_URL_SUFFIX=true`:
- Base URL `https://my-proxy.com/custom/path` → `https://my-proxy.com/custom/path` (unchanged)
### Azure AI Foundry (Alternative Method)
:::tip Recommended Method
For full Azure support including Azure AD authentication, use the dedicated [Azure Anthropic provider](./azure/azure_anthropic) with `azure_ai/` prefix.
:::
As an alternative, you can use the `anthropic/` provider directly with your Azure endpoint since Azure exposes Claude using Anthropic's native API.
```python
from litellm import completion
response = completion(
model="anthropic/claude-sonnet-4-5",
api_base="https://<your-resource>.services.ai.azure.com/anthropic",
api_key="<your-azure-api-key>",
messages=[{"role": "user", "content": "Hello!"}],
)
print(response)
```
:::info
**Finding your Azure endpoint:** Go to Azure AI Foundry → Your deployment → Overview. Your base URL will be `https://<resource-name>.services.ai.azure.com/anthropic`
:::
## Usage
```python

16
docs/my-website/docs/providers/bedrock.md
View file

@ -7,7 +7,7 @@ ALL Bedrock models (Anthropic, Meta, Deepseek, Mistral, Amazon, etc.) are Suppor
| Property | Details |
|-------|-------|
| Description | Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models (FMs). |
| Provider Route on LiteLLM | `bedrock/`, [`bedrock/converse/`](#set-converse--invoke-route), [`bedrock/invoke/`](#set-invoke-route), [`bedrock/converse_like/`](#calling-via-internal-proxy), [`bedrock/llama/`](#deepseek-not-r1), [`bedrock/deepseek_r1/`](#deepseek-r1), [`bedrock/qwen3/`](#qwen3-imported-models), [`bedrock/openai/`](./bedrock_imported.md#openai-compatible-imported-models-qwen-25-vl-etc) |
| Provider Route on LiteLLM | `bedrock/`, [`bedrock/converse/`](#set-converse--invoke-route), [`bedrock/invoke/`](#set-invoke-route), [`bedrock/converse_like/`](#calling-via-internal-proxy), [`bedrock/llama/`](#deepseek-not-r1), [`bedrock/deepseek_r1/`](#deepseek-r1), [`bedrock/qwen3/`](#qwen3-imported-models), [`bedrock/qwen2/`](./bedrock_imported.md#qwen2-imported-models), [`bedrock/openai/`](./bedrock_imported.md#openai-compatible-imported-models-qwen-25-vl-etc) |
| Provider Doc | [Amazon Bedrock ↗](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) |
| Supported OpenAI Endpoints | `/chat/completions`, `/completions`, `/embeddings`, `/images/generations` |
| Rerank Endpoint | `/rerank` |
@ -43,6 +43,8 @@ export AWS_BEARER_TOKEN_BEDROCK="your-api-key"
Option 2: use the api_key parameter to pass in API key for completion, embedding, image_generation API calls.
<Tabs>
<TabItem value="sdk" label="SDK">
```python
response = completion(
model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
@ -50,7 +52,17 @@ response = completion(
api_key="your-api-key"
)
```
</TabItem>
<TabItem value="proxy" label="PROXY">
```yaml
model_list:
- model_name: bedrock-claude-3-sonnet
litellm_params:
model: bedrock/anthropic.claude-3-sonnet-20240229-v1:0
api_key: os.environ/AWS_BEARER_TOKEN_BEDROCK
```
</TabItem>
</Tabs>
## Usage

91
docs/my-website/docs/providers/bedrock_batches.md
View file

@ -172,6 +172,97 @@ curl http://localhost:4000/v1/batches \
</TabItem>
</Tabs>
### 4. Retrieve batch results
Once the batch job is completed, download the results from S3:
<Tabs>
<TabItem value="python" label="Python">
```python showLineNumbers title="bedrock_batch.py"
...
# Wait for batch completion (check status periodically)
batch_status = client.batches.retrieve(batch_id=batch.id)
if batch_status.status == "completed":
# Download the output file
result = client.files.content(
file_id=batch_status.output_file_id,
extra_headers={"custom-llm-provider": "bedrock"}
)
# Save or process the results
with open("batch_output.jsonl", "wb") as f:
f.write(result.content)
# Parse JSONL results
for line in result.text.strip().split('\n'):
record = json.loads(line)
print(f"Record ID: {record['recordId']}")
print(f"Output: {record.get('modelOutput', {})}")
```
</TabItem>
<TabItem value="curl" label="Curl">
```bash showLineNumbers title="Download Batch Results"
# First retrieve batch to get output_file_id
curl http://localhost:4000/v1/batches/batch_abc123 \
-H "Authorization: Bearer sk-1234"
# Then download the output file
curl http://localhost:4000/v1/files/{output_file_id}/content \
-H "Authorization: Bearer sk-1234" \
-H "custom-llm-provider: bedrock" \
-o batch_output.jsonl
```
</TabItem>
<TabItem value="litellm-direct" label="LiteLLM Direct">
```python showLineNumbers title="bedrock_batch.py"
import litellm
from litellm import file_content
# Download using litellm directly (bypasses proxy managed files)
result = file_content(
file_id=batch_status.output_file_id, # Can be S3 URI or unified file ID
custom_llm_provider="bedrock",
aws_region_name="us-west-2",
)
# Process results
print(result.text)
```
</TabItem>
</Tabs>
**Output Format:**
The batch output file is in JSONL format with each line containing:
```json
{
"recordId": "request-1",
"modelInput": {
"messages": [...],
"max_tokens": 1000
},
"modelOutput": {
"content": [...],
"id": "msg_abc123",
"model": "claude-3-5-sonnet-20240620-v1:0",
"role": "assistant",
"stop_reason": "end_turn",
"usage": {
"input_tokens": 15,
"output_tokens": 10
}
}
}
```
## FAQ
### Where are my files written?

65
docs/my-website/docs/providers/bedrock_imported.md
View file

@ -203,6 +203,71 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
</TabItem>
</Tabs>
### Qwen2 Imported Models
| Property | Details |
|----------|---------|
| Provider Route | `bedrock/qwen2/{model_arn}` |
| Provider Documentation | [Bedrock Imported Models](https://docs.aws.amazon.com/bedrock/latest/userguide/model-customization-import-model.html) |
| Note | Qwen2 and Qwen3 architectures are mostly similar. The main difference is in the response format: Qwen2 uses "text" field while Qwen3 uses "generation" field. |
<Tabs>
<TabItem value="sdk" label="SDK">
```python
from litellm import completion
import os
response = completion(
model="bedrock/qwen2/arn:aws:bedrock:us-east-1:086734376398:imported-model/your-qwen2-model", # bedrock/qwen2/{your-model-arn}
messages=[{"role": "user", "content": "Tell me a joke"}],
max_tokens=100,
temperature=0.7
)
```
</TabItem>
<TabItem value="proxy" label="Proxy">
**1. Add to config**
```yaml
model_list:
- model_name: Qwen2-72B
litellm_params:
model: bedrock/qwen2/arn:aws:bedrock:us-east-1:086734376398:imported-model/your-qwen2-model
```
**2. Start proxy**
```bash
litellm --config /path/to/config.yaml
# RUNNING at http://0.0.0.0:4000
```
**3. Test it!**
```bash
curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Authorization: Bearer sk-1234' \
--header 'Content-Type: application/json' \
--data '{
"model": "Qwen2-72B", # 👈 the 'model_name' in config
"messages": [
{
"role": "user",
"content": "what llm are you"
}
],
}'
```
</TabItem>
</Tabs>
### OpenAI-Compatible Imported Models (Qwen 2.5 VL, etc.)
Use this route for Bedrock imported models that follow the **OpenAI Chat Completions API spec**. This includes models like Qwen 2.5 VL that accept OpenAI-formatted messages with support for vision (images), tool calling, and other OpenAI features.

42
docs/my-website/docs/providers/github_copilot.md
View file

@ -15,7 +15,7 @@ https://docs.github.com/en/copilot
|-------|-------|
| Description | GitHub Copilot Chat API provides access to GitHub's AI-powered coding assistant. |
| Provider Route on LiteLLM | `github_copilot/` |
| Supported Endpoints | `/chat/completions` |
| Supported Endpoints | `/chat/completions`, `/embeddings` |
| API Reference | [GitHub Copilot docs](https://docs.github.com/en/copilot) |
## Authentication
@ -62,6 +62,34 @@ for chunk in stream:
print(chunk.choices[0].delta.content, end="")
```
### Responses
For GPT Codex models, only responses API is supported.
```python showLineNumbers title="GitHub Copilot Responses"
import litellm
response = await litellm.aresponses(
model="github_copilot/gpt-5.1-codex",
input="Write a Python hello world",
max_output_tokens=500
)
print(response)
```
### Embedding
```python showLineNumbers title="GitHub Copilot Embedding"
import litellm
response = litellm.embedding(
model="github_copilot/text-embedding-3-small",
input=["good morning from litellm"]
)
print(response)
```
## Usage - LiteLLM Proxy
Add the following to your LiteLLM Proxy configuration file:
@ -71,6 +99,16 @@ model_list:
- model_name: github_copilot/gpt-4
litellm_params:
model: github_copilot/gpt-4
- model_name: github_copilot/gpt-5.1-codex
model_info:
mode: responses
litellm_params:
model: github_copilot/gpt-5.1-codex
- model_name: github_copilot/text-embedding-ada-002
model_info:
mode: embedding
litellm_params:
model: github_copilot/text-embedding-ada-002
```
Start your LiteLLM Proxy server:
@ -180,7 +218,7 @@ extra_headers = {
"editor-version": "vscode/1.85.1", # Editor version
"editor-plugin-version": "copilot/1.155.0", # Plugin version
"Copilot-Integration-Id": "vscode-chat", # Integration ID
"user-agent": "GithubCopilot/1.155.0" # User agent
"user-agent": "GithubCopilot/1.155.0" # User agent
}
```

244
docs/my-website/docs/providers/ragflow.md Normal file
View file

@ -0,0 +1,244 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# RAGFlow
Litellm supports Ragflow's chat completions APIs
## Supported Features
- ✅ Chat completions
- ✅ Streaming responses
- ✅ Both chat and agent endpoints
- ✅ Multiple credential sources (params, env vars, litellm_params)
- ✅ OpenAI-compatible API format
## API Key
```python
# env variable
os.environ['RAGFLOW_API_KEY']
```
## API Base
```python
# env variable
os.environ['RAGFLOW_API_BASE']
```
## Overview
RAGFlow provides OpenAI-compatible APIs with unique path structures that include chat and agent IDs:
- **Chat endpoint**: `/api/v1/chats_openai/{chat_id}/chat/completions`
- **Agent endpoint**: `/api/v1/agents_openai/{agent_id}/chat/completions`
The model name format embeds the endpoint type and ID:
- Chat: `ragflow/chat/{chat_id}/{model_name}`
- Agent: `ragflow/agent/{agent_id}/{model_name}`
## Sample Usage - Chat Endpoint
```python
from litellm import completion
import os
os.environ['RAGFLOW_API_KEY'] = "your-ragflow-api-key"
os.environ['RAGFLOW_API_BASE'] = "http://localhost:9380" # or your hosted URL
response = completion(
model="ragflow/chat/my-chat-id/gpt-4o-mini",
messages=[{"role": "user", "content": "How does the deep doc understanding work?"}]
)
print(response)
```
## Sample Usage - Agent Endpoint
```python
from litellm import completion
import os
os.environ['RAGFLOW_API_KEY'] = "your-ragflow-api-key"
os.environ['RAGFLOW_API_BASE'] = "http://localhost:9380" # or your hosted URL
response = completion(
model="ragflow/agent/my-agent-id/gpt-4o-mini",
messages=[{"role": "user", "content": "What are the key features?"}]
)
print(response)
```
## Sample Usage - With Parameters
You can also pass `api_key` and `api_base` directly as parameters:
```python
from litellm import completion
response = completion(
model="ragflow/chat/my-chat-id/gpt-4o-mini",
messages=[{"role": "user", "content": "Hello!"}],
api_key="your-ragflow-api-key",
api_base="http://localhost:9380"
)
print(response)
```
## Sample Usage - Streaming
```python
from litellm import completion
import os
os.environ['RAGFLOW_API_KEY'] = "your-ragflow-api-key"
os.environ['RAGFLOW_API_BASE'] = "http://localhost:9380"
response = completion(
model="ragflow/agent/my-agent-id/gpt-4o-mini",
messages=[{"role": "user", "content": "Explain RAGFlow"}],
stream=True
)
for chunk in response:
print(chunk)
```
## Model Name Format
The model name must follow one of these formats:
### Chat Endpoint
```
ragflow/chat/{chat_id}/{model_name}
```
Example: `ragflow/chat/my-chat-id/gpt-4o-mini`
### Agent Endpoint
```
ragflow/agent/{agent_id}/{model_name}
```
Example: `ragflow/agent/my-agent-id/gpt-4o-mini`
Where:
- `{chat_id}` or `{agent_id}` is the ID of your chat or agent in RAGFlow
- `{model_name}` is the actual model name (e.g., `gpt-4o-mini`, `gpt-4o`, etc.)
## Configuration Sources
LiteLLM supports multiple ways to provide credentials, checked in this order:
1. **Function parameters**: `api_key="..."`, `api_base="..."`
2. **litellm_params**: `litellm_params={"api_key": "...", "api_base": "..."}`
3. **Environment variables**: `RAGFLOW_API_KEY`, `RAGFLOW_API_BASE`
4. **Global litellm settings**: `litellm.api_key`, `litellm.api_base`
## Usage - LiteLLM Proxy Server
### 1. Save key in your environment
```bash
export RAGFLOW_API_KEY="your-ragflow-api-key"
export RAGFLOW_API_BASE="http://localhost:9380"
```
### 2. Start the proxy
<Tabs>
<TabItem value="config" label="config.yaml">
```yaml
model_list:
- model_name: ragflow-chat-gpt4
litellm_params:
model: ragflow/chat/my-chat-id/gpt-4o-mini
api_key: os.environ/RAGFLOW_API_KEY
api_base: os.environ/RAGFLOW_API_BASE
- model_name: ragflow-agent-gpt4
litellm_params:
model: ragflow/agent/my-agent-id/gpt-4o-mini
api_key: os.environ/RAGFLOW_API_KEY
api_base: os.environ/RAGFLOW_API_BASE
```
</TabItem>
<TabItem value="cli" label="CLI">
```bash
$ litellm --config /path/to/config.yaml
# Server running on http://0.0.0.0:4000
```
</TabItem>
</Tabs>
### 3. Test it
<Tabs>
<TabItem value="Curl" label="Curl Request">
```bash
curl http://0.0.0.0:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "ragflow-chat-gpt4",
"messages": [
{"role": "user", "content": "How does RAGFlow work?"}
]
}'
```
</TabItem>
<TabItem value="Python" label="Python SDK">
```python
from openai import OpenAI
client = OpenAI(
api_key="sk-1234", # Your LiteLLM proxy key
base_url="http://0.0.0.0:4000"
)
response = client.chat.completions.create(
model="ragflow-chat-gpt4",
messages=[
{"role": "user", "content": "How does RAGFlow work?"}
]
)
print(response)
```
</TabItem>
</Tabs>
## API Base URL Handling
The `api_base` parameter can be provided with or without `/v1` suffix. LiteLLM will automatically handle it:
- `http://localhost:9380` → `http://localhost:9380/api/v1/chats_openai/{chat_id}/chat/completions`
- `http://localhost:9380/v1` → `http://localhost:9380/api/v1/chats_openai/{chat_id}/chat/completions`
- `http://localhost:9380/api/v1` → `http://localhost:9380/api/v1/chats_openai/{chat_id}/chat/completions`
All three formats will work correctly.
## Error Handling
If you encounter errors:
1. **Invalid model format**: Ensure your model name follows `ragflow/{chat|agent}/{id}/{model_name}` format
2. **Missing api_base**: Provide `api_base` via parameter, environment variable, or litellm_params
3. **Connection errors**: Verify your RAGFlow server is running and accessible at the provided `api_base`
:::info
For more information about passing provider-specific parameters, [go here](../completion/provider_specific_params.md)
:::

349
docs/my-website/docs/providers/ragflow_vector_store.md Normal file
View file

@ -0,0 +1,349 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import Image from '@theme/IdealImage';
# RAGFlow Vector Stores
Litellm support creation and management of datasets for document processing and knowledge base management in Ragflow.
| Property | Details |
|----------|---------|
| Description | RAGFlow datasets enable document processing, chunking, and knowledge base management for RAG applications. |
| Provider Route on LiteLLM | `ragflow` in the litellm vector_store_registry |
| Provider Doc | [RAGFlow API Documentation ↗](https://ragflow.io/docs) |
| Supported Operations | Dataset Management (Create, List, Update, Delete) |
| Search/Retrieval | ❌ Not supported (management only) |
## Quick Start
### LiteLLM Python SDK
```python showLineNumbers title="Example using LiteLLM Python SDK"
import os
import litellm
# Set RAGFlow credentials
os.environ["RAGFLOW_API_KEY"] = "your-ragflow-api-key"
os.environ["RAGFLOW_API_BASE"] = "http://localhost:9380" # Optional, defaults to localhost:9380
# Create a RAGFlow dataset
response = litellm.vector_stores.create(
name="my-dataset",
custom_llm_provider="ragflow",
metadata={
"description": "My knowledge base dataset",
"embedding_model": "BAAI/bge-large-zh-v1.5@BAAI",
"chunk_method": "naive"
}
)
print(f"Created dataset ID: {response.id}")
print(f"Dataset name: {response.name}")
```
### LiteLLM Proxy
#### 1. Configure your vector_store_registry
<Tabs>
<TabItem value="config-yaml" label="config.yaml">
```yaml
model_list:
- model_name: gpt-4o-mini
litellm_params:
model: gpt-4o-mini
api_key: os.environ/OPENAI_API_KEY
vector_store_registry:
- vector_store_name: "ragflow-knowledge-base"
litellm_params:
vector_store_id: "your-dataset-id"
custom_llm_provider: "ragflow"
api_key: os.environ/RAGFLOW_API_KEY
api_base: os.environ/RAGFLOW_API_BASE # Optional
vector_store_description: "RAGFlow dataset for knowledge base"
vector_store_metadata:
source: "Company documentation"
```
</TabItem>
<TabItem value="litellm-ui" label="LiteLLM UI">
On the LiteLLM UI, Navigate to Experimental > Vector Stores > Create Vector Store. On this page you can create a vector store with a name, vector store id and credentials.
<Image
img={require('../../img/kb_2.png')}
style={{width: '50%'}}
/>
</TabItem>
</Tabs>
#### 2. Create a dataset via Proxy
<Tabs>
<TabItem value="curl" label="Curl">
```bash
curl http://localhost:4000/v1/vector_stores \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-d '{
"name": "my-ragflow-dataset",
"custom_llm_provider": "ragflow",
"metadata": {
"description": "Test dataset",
"chunk_method": "naive"
}
}'
```
</TabItem>
<TabItem value="openai-sdk" label="OpenAI Python SDK">
```python
from openai import OpenAI
# Initialize client with your LiteLLM proxy URL
client = OpenAI(
base_url="http://localhost:4000",
api_key="your-litellm-api-key"
)
# Create a RAGFlow dataset
response = client.vector_stores.create(
name="my-ragflow-dataset",
custom_llm_provider="ragflow",
metadata={
"description": "Test dataset",
"chunk_method": "naive"
}
)
print(f"Created dataset: {response.id}")
```
</TabItem>
</Tabs>
## Configuration
### Environment Variables
RAGFlow vector stores support configuration via environment variables:
- `RAGFLOW_API_KEY` - Your RAGFlow API key (required)
- `RAGFLOW_API_BASE` - RAGFlow API base URL (optional, defaults to `http://localhost:9380`)
### Parameters
You can also pass these via `litellm_params`:
- `api_key` - RAGFlow API key (overrides `RAGFLOW_API_KEY` env var)
- `api_base` - RAGFlow API base URL (overrides `RAGFLOW_API_BASE` env var)
## Dataset Creation Options
### Basic Dataset Creation
```python
response = litellm.vector_stores.create(
name="basic-dataset",
custom_llm_provider="ragflow"
)
```
### Dataset with Chunk Method
RAGFlow supports various chunk methods for different document types:
<Tabs>
<TabItem value="naive" label="Naive (General)">
```python
response = litellm.vector_stores.create(
name="general-dataset",
custom_llm_provider="ragflow",
metadata={
"chunk_method": "naive",
"parser_config": {
"chunk_token_num": 512,
"delimiter": "\n",
"html4excel": False,
"layout_recognize": "DeepDOC"
}
}
)
```
</TabItem>
<TabItem value="book" label="Book">
```python
response = litellm.vector_stores.create(
name="book-dataset",
custom_llm_provider="ragflow",
metadata={
"chunk_method": "book",
"parser_config": {
"raptor": {
"use_raptor": False
}
}
}
)
```
</TabItem>
<TabItem value="qa" label="Q&A">
```python
response = litellm.vector_stores.create(
name="qa-dataset",
custom_llm_provider="ragflow",
metadata={
"chunk_method": "qa",
"parser_config": {
"raptor": {
"use_raptor": False
}
}
}
)
```
</TabItem>
<TabItem value="paper" label="Paper">
```python
response = litellm.vector_stores.create(
name="paper-dataset",
custom_llm_provider="ragflow",
metadata={
"chunk_method": "paper",
"parser_config": {
"raptor": {
"use_raptor": False
}
}
}
)
```
</TabItem>
</Tabs>
### Dataset with Ingestion Pipeline
Instead of using a chunk method, you can use an ingestion pipeline:
```python
response = litellm.vector_stores.create(
name="pipeline-dataset",
custom_llm_provider="ragflow",
metadata={
"parse_type": 2, # Number of parsers in your pipeline
"pipeline_id": "d0bebe30ae2211f0970942010a8e0005" # 32-character hex ID
}
)
```
**Note**: `chunk_method` and `pipeline_id` are mutually exclusive. Use one or the other.
### Advanced Parser Configuration
```python
response = litellm.vector_stores.create(
name="advanced-dataset",
custom_llm_provider="ragflow",
metadata={
"chunk_method": "naive",
"description": "Advanced dataset with custom parser config",
"embedding_model": "BAAI/bge-large-zh-v1.5@BAAI",
"permission": "me", # or "team"
"parser_config": {
"chunk_token_num": 1024,
"delimiter": "\n!?;。；！？",
"html4excel": True,
"layout_recognize": "DeepDOC",
"auto_keywords": 5,
"auto_questions": 3,
"task_page_size": 12,
"raptor": {
"use_raptor": True
},
"graphrag": {
"use_graphrag": False
}
}
}
)
```
## Supported Chunk Methods
RAGFlow supports the following chunk methods:
- `naive` - General purpose (default)
- `book` - For book documents
- `email` - For email documents
- `laws` - For legal documents
- `manual` - Manual chunking
- `one` - Single chunk
- `paper` - For academic papers
- `picture` - For image documents
- `presentation` - For presentation documents
- `qa` - Q&A format
- `table` - For table documents
- `tag` - Tag-based chunking
## RAGFlow-Specific Parameters
All RAGFlow-specific parameters should be passed via the `metadata` field:
| Parameter | Type | Description |
|-----------|------|-------------|
| `avatar` | string | Base64 encoding of the avatar (max 65535 chars) |
| `description` | string | Brief description of the dataset (max 65535 chars) |
| `embedding_model` | string | Embedding model name (e.g., "BAAI/bge-large-zh-v1.5@BAAI") |
| `permission` | string | Access permission: "me" (default) or "team" |
| `chunk_method` | string | Chunking method (see supported methods above) |
| `parser_config` | object | Parser configuration (varies by chunk_method) |
| `parse_type` | int | Number of parsers in pipeline (required with pipeline_id) |
| `pipeline_id` | string | 32-character hex pipeline ID (required with parse_type) |
## Error Handling
RAGFlow returns error responses in the following format:
```json
{
"code": 101,
"message": "Dataset name 'my-dataset' already exists"
}
```
LiteLLM automatically maps these to appropriate exceptions:
- `code != 0` → Raises exception with the error message
- Missing required fields → Raises `ValueError`
- Mutually exclusive parameters → Raises `ValueError`
## Limitations
- **Search/Retrieval**: RAGFlow vector stores support dataset management only. Search operations are not supported and will raise `NotImplementedError`.
- **List/Update/Delete**: These operations are not yet implemented through the standard vector store API. Use RAGFlow's native API endpoints directly.
## Further Reading
Vector Stores:
- [Vector Store Creation](../vector_stores/create.md)
- [Using Vector Stores with Completions](../completion/knowledgebase.md)
- [Vector Store Registry](../completion/knowledgebase.md#vectorstoreregistry)

349
docs/my-website/docs/providers/vertex.md
View file

@ -2550,355 +2550,6 @@ print(response)
</TabItem>
</Tabs>
## **Gemini TTS (Text-to-Speech) Audio Output**
:::info
LiteLLM supports Gemini TTS models on Vertex AI that can generate audio responses using the OpenAI-compatible `audio` parameter format.
:::
### Supported Models
LiteLLM supports Gemini TTS models with audio capabilities on Vertex AI (e.g. `vertex_ai/gemini-2.5-flash-preview-tts` and `vertex_ai/gemini-2.5-pro-preview-tts`). For the complete list of available TTS models and voices, see the [official Gemini TTS documentation](https://ai.google.dev/gemini-api/docs/speech-generation).
### Limitations
:::warning
**Important Limitations**:
- Gemini TTS models only support the `pcm16` audio format
- **Streaming support has not been added** to TTS models yet
- The `modalities` parameter must be set to `['audio']` for TTS requests
:::
### Quick Start
<Tabs>
<TabItem value="sdk" label="SDK">
```python
from litellm import completion
import json
## GET CREDENTIALS
file_path = 'path/to/vertex_ai_service_account.json'
# Load the JSON file
with open(file_path, 'r') as file:
vertex_credentials = json.load(file)
# Convert to JSON string
vertex_credentials_json = json.dumps(vertex_credentials)
response = completion(
model="vertex_ai/gemini-2.5-flash-preview-tts",
messages=[{"role": "user", "content": "Say hello in a friendly voice"}],
modalities=["audio"], # Required for TTS models
audio={
"voice": "Kore",
"format": "pcm16" # Required: must be "pcm16"
},
vertex_credentials=vertex_credentials_json
)
print(response)
```
</TabItem>
<TabItem value="proxy" label="PROXY">
1. Setup config.yaml
```yaml
model_list:
- model_name: gemini-tts-flash
litellm_params:
model: vertex_ai/gemini-2.5-flash-preview-tts
vertex_project: "your-project-id"
vertex_location: "us-central1"
vertex_credentials: "/path/to/service_account.json"
- model_name: gemini-tts-pro
litellm_params:
model: vertex_ai/gemini-2.5-pro-preview-tts
vertex_project: "your-project-id"
vertex_location: "us-central1"
vertex_credentials: "/path/to/service_account.json"
```
2. Start proxy
```bash
litellm --config /path/to/config.yaml
```
3. Make TTS request
```bash
curl http://0.0.0.0:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <YOUR-LITELLM-KEY>" \
-d '{
"model": "gemini-tts-flash",
"messages": [{"role": "user", "content": "Say hello in a friendly voice"}],
"modalities": ["audio"],
"audio": {
"voice": "Kore",
"format": "pcm16"
}
}'
```
</TabItem>
</Tabs>
### Advanced Usage
You can combine TTS with other Gemini features:
```python
response = completion(
model="vertex_ai/gemini-2.5-pro-preview-tts",
messages=[
{"role": "system", "content": "You are a helpful assistant that speaks clearly."},
{"role": "user", "content": "Explain quantum computing in simple terms"}
],
modalities=["audio"],
audio={
"voice": "Charon",
"format": "pcm16"
},
temperature=0.7,
max_tokens=150,
vertex_credentials=vertex_credentials_json
)
```
For more information about Gemini's TTS capabilities and available voices, see the [official Gemini TTS documentation](https://ai.google.dev/gemini-api/docs/speech-generation).
## **Text to Speech APIs**
:::info
LiteLLM supports calling [Vertex AI Text to Speech API](https://console.cloud.google.com/vertex-ai/generative/speech/text-to-speech) in the OpenAI text to speech API format
:::
### Usage - Basic
<Tabs>
<TabItem value="sdk" label="SDK">
Vertex AI does not support passing a `model` param - so passing `model=vertex_ai/` is the only required param
**Sync Usage**
```python
speech_file_path = Path(__file__).parent / "speech_vertex.mp3"
response = litellm.speech(
model="vertex_ai/",
input="hello what llm guardrail do you have",
)
response.stream_to_file(speech_file_path)
```
**Async Usage**
```python
speech_file_path = Path(__file__).parent / "speech_vertex.mp3"
response = litellm.aspeech(
model="vertex_ai/",
input="hello what llm guardrail do you have",
)
response.stream_to_file(speech_file_path)
```
</TabItem>
<TabItem value="proxy" label="LiteLLM PROXY (Unified Endpoint)">
1. Add model to config.yaml
```yaml
model_list:
- model_name: vertex-tts
litellm_params:
model: vertex_ai/ # Vertex AI does not support passing a `model` param - so passing `model=vertex_ai/` is the only required param
vertex_project: "adroit-crow-413218"
vertex_location: "us-central1"
vertex_credentials: adroit-crow-413218-a956eef1a2a8.json
litellm_settings:
drop_params: True
```
2. Start Proxy
```
$ litellm --config /path/to/config.yaml
```
3. Make Request use OpenAI Python SDK
```python
import openai
client = openai.OpenAI(api_key="sk-1234", base_url="http://0.0.0.0:4000")
# see supported values for "voice" on vertex here:
# https://console.cloud.google.com/vertex-ai/generative/speech/text-to-speech
response = client.audio.speech.create(
model = "vertex-tts",
input="the quick brown fox jumped over the lazy dogs",
voice={'languageCode': 'en-US', 'name': 'en-US-Studio-O'}
)
print("response from proxy", response)
```
</TabItem>
</Tabs>
### Usage - `ssml` as input
Pass your `ssml` as input to the `input` param, if it contains `<speak>`, it will be automatically detected and passed as `ssml` to the Vertex AI API
If you need to force your `input` to be passed as `ssml`, set `use_ssml=True`
<Tabs>
<TabItem value="sdk" label="SDK">
Vertex AI does not support passing a `model` param - so passing `model=vertex_ai/` is the only required param
```python
speech_file_path = Path(__file__).parent / "speech_vertex.mp3"
ssml = """
<speak>
<p>Hello, world!</p>
<p>This is a test of the <break strength="medium" /> text-to-speech API.</p>
</speak>
"""
response = litellm.speech(
input=ssml,
model="vertex_ai/test",
voice={
"languageCode": "en-UK",
"name": "en-UK-Studio-O",
},
audioConfig={
"audioEncoding": "LINEAR22",
"speakingRate": "10",
},
)
response.stream_to_file(speech_file_path)
```
</TabItem>
<TabItem value="proxy" label="LiteLLM PROXY (Unified Endpoint)">
```python
import openai
client = openai.OpenAI(api_key="sk-1234", base_url="http://0.0.0.0:4000")
ssml = """
<speak>
<p>Hello, world!</p>
<p>This is a test of the <break strength="medium" /> text-to-speech API.</p>
</speak>
"""
# see supported values for "voice" on vertex here:
# https://console.cloud.google.com/vertex-ai/generative/speech/text-to-speech
response = client.audio.speech.create(
model = "vertex-tts",
input=ssml,
voice={'languageCode': 'en-US', 'name': 'en-US-Studio-O'},
)
print("response from proxy", response)
```
</TabItem>
</Tabs>
### Forcing SSML Usage
You can force the use of SSML by setting the `use_ssml` parameter to `True`. This is useful when you want to ensure that your input is treated as SSML, even if it doesn't contain the `<speak>` tags.
Here are examples of how to force SSML usage:
<Tabs>
<TabItem value="sdk" label="SDK">
Vertex AI does not support passing a `model` param - so passing `model=vertex_ai/` is the only required param
```python
speech_file_path = Path(__file__).parent / "speech_vertex.mp3"
ssml = """
<speak>
<p>Hello, world!</p>
<p>This is a test of the <break strength="medium" /> text-to-speech API.</p>
</speak>
"""
response = litellm.speech(
input=ssml,
use_ssml=True,
model="vertex_ai/test",
voice={
"languageCode": "en-UK",
"name": "en-UK-Studio-O",
},
audioConfig={
"audioEncoding": "LINEAR22",
"speakingRate": "10",
},
)
response.stream_to_file(speech_file_path)
```
</TabItem>
<TabItem value="proxy" label="LiteLLM PROXY (Unified Endpoint)">
```python
import openai
client = openai.OpenAI(api_key="sk-1234", base_url="http://0.0.0.0:4000")
ssml = """
<speak>
<p>Hello, world!</p>
<p>This is a test of the <break strength="medium" /> text-to-speech API.</p>
</speak>
"""
# see supported values for "voice" on vertex here:
# https://console.cloud.google.com/vertex-ai/generative/speech/text-to-speech
response = client.audio.speech.create(
model = "vertex-tts",
input=ssml, # pass as None since OpenAI SDK requires this param
voice={'languageCode': 'en-US', 'name': 'en-US-Studio-O'},
extra_body={"use_ssml": True},
)
print("response from proxy", response)
```
</TabItem>
</Tabs>
## **Fine Tuning APIs**

423
docs/my-website/docs/providers/vertex_speech.md Normal file
View file

@ -0,0 +1,423 @@
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# Vertex AI Text to Speech
| Property | Details |
|-------|-------|
| Description | Google Cloud Text-to-Speech with Chirp3 HD voices and Gemini TTS |
| Provider Route on LiteLLM | `vertex_ai/chirp` (Chirp), `vertex_ai/gemini-*-tts` (Gemini) |
## Chirp3 HD Voices
Google Cloud Text-to-Speech API with high-quality Chirp3 HD voices.
### Quick Start
#### LiteLLM Python SDK
```python showLineNumbers title="Chirp3 Quick Start"
from litellm import speech
from pathlib import Path
speech_file_path = Path(__file__).parent / "speech.mp3"
response = speech(
model="vertex_ai/chirp",
voice="alloy", # OpenAI voice name - automatically mapped
input="Hello, this is Vertex AI Text to Speech",
vertex_project="your-project-id",
vertex_location="us-central1",
)
response.stream_to_file(speech_file_path)
```
#### LiteLLM AI Gateway
**1. Setup config.yaml**
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: vertex-tts
litellm_params:
model: vertex_ai/chirp
vertex_project: "your-project-id"
vertex_location: "us-central1"
vertex_credentials: "/path/to/service_account.json"
```
**2. Start the proxy**
```bash title="Start LiteLLM Proxy"
litellm --config /path/to/config.yaml
```
**3. Make requests**
<Tabs>
<TabItem value="curl" label="curl">
```bash showLineNumbers title="Chirp3 Quick Start"
curl http://0.0.0.0:4000/v1/audio/speech \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
"model": "vertex-tts",
"voice": "alloy",
"input": "Hello, this is Vertex AI Text to Speech"
}' \
--output speech.mp3
```
</TabItem>
<TabItem value="openai-sdk" label="OpenAI Python SDK">
```python showLineNumbers title="Chirp3 Quick Start"
import openai
client = openai.OpenAI(api_key="sk-1234", base_url="http://0.0.0.0:4000")
response = client.audio.speech.create(
model="vertex-tts",
voice="alloy",
input="Hello, this is Vertex AI Text to Speech",
)
response.stream_to_file("speech.mp3")
```
</TabItem>
</Tabs>
### Voice Mapping
LiteLLM maps OpenAI voice names to Google Cloud voices. You can use either OpenAI voices or Google Cloud voices directly.
| OpenAI Voice | Google Cloud Voice |
|-------------|-------------------|
| `alloy` | en-US-Studio-O |
| `echo` | en-US-Studio-M |
| `fable` | en-GB-Studio-B |
| `onyx` | en-US-Wavenet-D |
| `nova` | en-US-Studio-O |
| `shimmer` | en-US-Wavenet-F |
### Using Google Cloud Voices Directly
#### LiteLLM Python SDK
```python showLineNumbers title="Chirp3 HD Voice"
from litellm import speech
# Pass Chirp3 HD voice name directly
response = speech(
model="vertex_ai/chirp",
voice="en-US-Chirp3-HD-Charon",
input="Hello with a Chirp3 HD voice",
vertex_project="your-project-id",
)
response.stream_to_file("speech.mp3")
```
```python showLineNumbers title="Voice as Dict (Multilingual)"
from litellm import speech
# Pass as dict for full control over language and voice
response = speech(
model="vertex_ai/chirp",
voice={
"languageCode": "de-DE",
"name": "de-DE-Chirp3-HD-Charon",
},
input="Hallo, dies ist ein Test",
vertex_project="your-project-id",
)
response.stream_to_file("speech.mp3")
```
#### LiteLLM AI Gateway
<Tabs>
<TabItem value="curl" label="curl">
```bash showLineNumbers title="Chirp3 HD Voice"
curl http://0.0.0.0:4000/v1/audio/speech \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
"model": "vertex-tts",
"voice": "en-US-Chirp3-HD-Charon",
"input": "Hello with a Chirp3 HD voice"
}' \
--output speech.mp3
```
```bash showLineNumbers title="Voice as Dict (Multilingual)"
curl http://0.0.0.0:4000/v1/audio/speech \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
"model": "vertex-tts",
"voice": {"languageCode": "de-DE", "name": "de-DE-Chirp3-HD-Charon"},
"input": "Hallo, dies ist ein Test"
}' \
--output speech.mp3
```
</TabItem>
<TabItem value="openai-sdk" label="OpenAI Python SDK">
```python showLineNumbers title="Chirp3 HD Voice"
import openai
client = openai.OpenAI(api_key="sk-1234", base_url="http://0.0.0.0:4000")
response = client.audio.speech.create(
model="vertex-tts",
voice="en-US-Chirp3-HD-Charon",
input="Hello with a Chirp3 HD voice",
)
response.stream_to_file("speech.mp3")
```
```python showLineNumbers title="Voice as Dict (Multilingual)"
import openai
client = openai.OpenAI(api_key="sk-1234", base_url="http://0.0.0.0:4000")
response = client.audio.speech.create(
model="vertex-tts",
voice={"languageCode": "de-DE", "name": "de-DE-Chirp3-HD-Charon"},
input="Hallo, dies ist ein Test",
)
response.stream_to_file("speech.mp3")
```
</TabItem>
</Tabs>
Browse available voices: [Google Cloud Text-to-Speech Console](https://console.cloud.google.com/vertex-ai/generative/speech/text-to-speech)
### Passing Raw SSML
LiteLLM auto-detects SSML when your input contains `<speak>` tags and passes it through unchanged.
#### LiteLLM Python SDK
```python showLineNumbers title="SSML Input"
from litellm import speech
ssml = """
<speak>
<p>Hello, world!</p>
<p>This is a test of the <break strength="medium" /> text-to-speech API.</p>
</speak>
"""
response = speech(
model="vertex_ai/chirp",
voice="en-US-Studio-O",
input=ssml, # Auto-detected as SSML
vertex_project="your-project-id",
)
response.stream_to_file("speech.mp3")
```
```python showLineNumbers title="Force SSML Mode"
from litellm import speech
# Force SSML mode with use_ssml=True
response = speech(
model="vertex_ai/chirp",
voice="en-US-Studio-O",
input="<speak><prosody rate='slow'>Speaking slowly</prosody></speak>",
use_ssml=True,
vertex_project="your-project-id",
)
response.stream_to_file("speech.mp3")
```
#### LiteLLM AI Gateway
<Tabs>
<TabItem value="curl" label="curl">
```bash showLineNumbers title="SSML Input"
curl http://0.0.0.0:4000/v1/audio/speech \
-H "Authorization: Bearer sk-1234" \
-H "Content-Type: application/json" \
-d '{
"model": "vertex-tts",
"voice": "en-US-Studio-O",
"input": "<speak><p>Hello!</p><break time=\"500ms\"/><p>How are you?</p></speak>"
}' \
--output speech.mp3
```
</TabItem>
<TabItem value="openai-sdk" label="OpenAI Python SDK">
```python showLineNumbers title="SSML Input"
import openai
client = openai.OpenAI(api_key="sk-1234", base_url="http://0.0.0.0:4000")
ssml = """<speak><p>Hello!</p><break time="500ms"/><p>How are you?</p></speak>"""
response = client.audio.speech.create(
model="vertex-tts",
voice="en-US-Studio-O",
input=ssml,
)
response.stream_to_file("speech.mp3")
```
</TabItem>
</Tabs>
### Supported Parameters
| Parameter | Description | Values |
|-----------|-------------|--------|
| `voice` | Voice selection | OpenAI voice, Google Cloud voice name, or dict |
| `input` | Text to convert | Plain text or SSML |
| `speed` | Speaking rate | 0.25 to 4.0 (default: 1.0) |
| `response_format` | Audio format | `mp3`, `opus`, `wav`, `pcm`, `flac` |
| `use_ssml` | Force SSML mode | `True` / `False` |
### Async Usage
```python showLineNumbers title="Async Speech Generation"
import asyncio
from litellm import aspeech
async def main():
response = await aspeech(
model="vertex_ai/chirp",
voice="alloy",
input="Hello from async",
vertex_project="your-project-id",
)
response.stream_to_file("speech.mp3")
asyncio.run(main())
```
---
## Gemini TTS
Gemini models with audio output capabilities using the chat completions API.
:::warning
**Limitations:**
- Only supports `pcm16` audio format
- Streaming not yet supported
- Must set `modalities: ["audio"]`
:::
### Quick Start
#### LiteLLM Python SDK
```python showLineNumbers title="Gemini TTS Quick Start"
from litellm import completion
import json
# Load credentials
with open('path/to/service_account.json', 'r') as file:
vertex_credentials = json.dumps(json.load(file))
response = completion(
model="vertex_ai/gemini-2.5-flash-preview-tts",
messages=[{"role": "user", "content": "Say hello in a friendly voice"}],
modalities=["audio"],
audio={
"voice": "Kore",
"format": "pcm16"
},
vertex_credentials=vertex_credentials
)
print(response)
```
#### LiteLLM AI Gateway
**1. Setup config.yaml**
```yaml showLineNumbers title="config.yaml"
model_list:
- model_name: gemini-tts
litellm_params:
model: vertex_ai/gemini-2.5-flash-preview-tts
vertex_project: "your-project-id"
vertex_location: "us-central1"
vertex_credentials: "/path/to/service_account.json"
```
**2. Start the proxy**
```bash title="Start LiteLLM Proxy"
litellm --config /path/to/config.yaml
```
**3. Make requests**
<Tabs>
<TabItem value="curl" label="curl">
```bash showLineNumbers title="Gemini TTS Request"
curl http://0.0.0.0:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-1234" \
-d '{
"model": "gemini-tts",
"messages": [{"role": "user", "content": "Say hello in a friendly voice"}],
"modalities": ["audio"],
"audio": {"voice": "Kore", "format": "pcm16"}
}'
```
</TabItem>
<TabItem value="openai-sdk" label="OpenAI Python SDK">
```python showLineNumbers title="Gemini TTS Request"
import openai
client = openai.OpenAI(api_key="sk-1234", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
model="gemini-tts",
messages=[{"role": "user", "content": "Say hello in a friendly voice"}],
modalities=["audio"],
audio={"voice": "Kore", "format": "pcm16"},
)
print(response)
```
</TabItem>
</Tabs>
### Supported Models
- `vertex_ai/gemini-2.5-flash-preview-tts`
- `vertex_ai/gemini-2.5-pro-preview-tts`
See [Gemini TTS documentation](https://ai.google.dev/gemini-api/docs/speech-generation) for available voices.
### Advanced Usage
```python showLineNumbers title="Gemini TTS with System Prompt"
from litellm import completion
response = completion(
model="vertex_ai/gemini-2.5-pro-preview-tts",
messages=[
{"role": "system", "content": "You are a helpful assistant that speaks clearly."},
{"role": "user", "content": "Explain quantum computing in simple terms"}
],
modalities=["audio"],
audio={"voice": "Charon", "format": "pcm16"},
temperature=0.7,
max_tokens=150,
vertex_credentials=vertex_credentials
)
```

7
docs/my-website/docs/proxy/config_settings.md
View file

@ -113,7 +113,7 @@ general_settings:
# Database Settings
database_url: string
database_connection_pool_limit: 0 # default 100
database_connection_pool_limit: 0 # default 10
database_connection_timeout: 0 # default 60s
allow_requests_on_db_unavailable: boolean # if true, will allow requests that can not connect to the DB to verify Virtual Key to still work
@ -234,7 +234,7 @@ router_settings:
| max_response_size_mb | int | The maximum size for responses in MB. LLM Responses above this size will not be sent. |
| proxy_budget_rescheduler_min_time | int | The minimum time (in seconds) to wait before checking db for budget resets. **Default is 597 seconds** |
| proxy_budget_rescheduler_max_time | int | The maximum time (in seconds) to wait before checking db for budget resets. **Default is 605 seconds** |
| proxy_batch_write_at | int | Time (in seconds) to wait before batch writing spend logs to the db. **Default is 30 seconds** |
| proxy_batch_write_at | int | Time (in seconds) to wait before batch writing spend logs to the db. **Default is 10 seconds** |
| proxy_batch_polling_interval | int | Time (in seconds) to wait before polling a batch, to check if it's completed. **Default is 6000 seconds (1 hour)** |
| alerting_args | dict | Args for Slack Alerting [Doc on Slack Alerting](./alerting.md) |
| custom_key_generate | str | Custom function for key generation [Doc on custom key generation](./virtual_keys.md#custom--key-generate) |
@ -377,6 +377,7 @@ router_settings:
| ATHINA_API_KEY | API key for Athina service
| ATHINA_BASE_URL | Base URL for Athina service (defaults to `https://log.athina.ai`)
| AUTH_STRATEGY | Strategy used for authentication (e.g., OAuth, API key)
| AUTO_REDIRECT_UI_LOGIN_TO_SSO | Flag to enable automatic redirect of UI login page to SSO when SSO is configured. Default is **true**
| ANTHROPIC_API_KEY | API key for Anthropic service
| ANTHROPIC_API_BASE | Base URL for Anthropic API. Default is https://api.anthropic.com
| AWS_ACCESS_KEY_ID | Access Key ID for AWS services
@ -763,7 +764,7 @@ router_settings:
| PROMPTLAYER_API_KEY | API key for PromptLayer integration
| PROXY_ADMIN_ID | Admin identifier for proxy server
| PROXY_BASE_URL | Base URL for proxy service
| PROXY_BATCH_WRITE_AT | Time in seconds to wait before batch writing spend logs to the database. Default is 30
| PROXY_BATCH_WRITE_AT | Time in seconds to wait before batch writing spend logs to the database. Default is 10
| PROXY_BATCH_POLLING_INTERVAL | Time in seconds to wait before polling a batch, to check if it's completed. Default is 6000s (1 hour)
| PROXY_BUDGET_RESCHEDULER_MAX_TIME | Maximum time in seconds to wait before checking database for budget resets. Default is 605
| PROXY_BUDGET_RESCHEDULER_MIN_TIME | Minimum time in seconds to wait before checking database for budget resets. Default is 597

2
docs/my-website/docs/proxy/configs.md
View file

@ -576,7 +576,7 @@ custom_tokenizer:
```yaml
general_settings:
database_connection_pool_limit: 100 # sets connection pool for prisma client to postgres db at 100
database_connection_pool_limit: 10 # sets connection pool for prisma client to postgres db (default: 10, recommended: 10-20)
database_connection_timeout: 60 # sets a 60s timeout for any connection call to the db
```

90
docs/my-website/docs/proxy/error_diagnosis.md Normal file
View file

@ -0,0 +1,90 @@
# Diagnosing Errors - Provider vs Gateway
Having trouble diagnosing if an error is from the **LLM Provider** (OpenAI, Anthropic, etc.) or from the **LiteLLM AI Gateway** itself? Here's how to tell.
## Quick Rule
**If the error contains `<Provider>Exception`, it's from the provider.**
| Error Contains | Error Source |
|----------------|--------------|
| `AnthropicException` | Anthropic |
| `OpenAIException` | OpenAI |
| `AzureException` | Azure |
| `BedrockException` | AWS Bedrock |
| `VertexAIException` | Google Vertex AI |
| No provider name | LiteLLM AI Gateway |
## Examples
### Provider Error (from AWS Bedrock)
```
{
"error": {
"message": "litellm.BadRequestError: BedrockException - {\"message\":\"The model returned the following errors: messages.1.content.0.type: Expected `thinking` or `redacted_thinking`, but found `text`.\"}",
"type": "invalid_request_error",
"param": null,
"code": "400"
}
}
```
This error is from **AWS Bedrock** (notice `BedrockException`). The Bedrock API is rejecting the request due to invalid message format - this is not a LiteLLM issue.
### Provider Error (from OpenAI)
```
{
"error": {
"message": "litellm.AuthenticationError: OpenAIException - Incorrect API key provided: <my-key>. You can find your API key at https://platform.openai.com/account/api-keys.",
"type": "invalid_request_error",
"param": null,
"code": "invalid_api_key"
}
}
```
This error is from **OpenAI** (notice `OpenAIException`). The OpenAI API key configured in LiteLLM is invalid.
### Provider Error (from Anthropic)
```
{
"error": {
"message": "litellm.InternalServerError: AnthropicException - Overloaded. Handle with `litellm.InternalServerError`.",
"type": "internal_server_error",
"param": null,
"code": "500"
}
}
```
This error is from **Anthropic** (notice `AnthropicException`). The Anthropic API is overloaded - this is not a LiteLLM issue.
### Gateway Error (from LiteLLM)
```
{
"error": {
"message": "Invalid API Key. Please check your LiteLLM API key.",
"type": "auth_error",
"param": null,
"code": "401"
}
}
```
This error is from the **LiteLLM AI Gateway** (no provider name). Your LiteLLM virtual key is invalid.
## What to do?
| Error Source | Action |
|--------------|--------|
| Provider Error | Check the provider's status page, adjust rate limits, or retry later |
| Gateway Error | Check your LiteLLM configuration, API keys, or [open an issue](https://github.com/BerriAI/litellm/issues) |
## See Also
- [Debugging](/docs/proxy/debugging) - Enable debug logs to see detailed request/response info
- [Exception Mapping](/docs/exception_mapping) - Full list of LiteLLM exception types

14
docs/my-website/docs/proxy/pass_through.md
View file

@ -275,6 +275,20 @@ In this video, we'll add the Azure OpenAI Assistants API as a pass through endpo
- Check LiteLLM proxy logs for error details
- Verify the target API's expected request format
### Allowing Team JWTs to use pass-through routes
If you are using pass-through provider routes (e.g., `/anthropic/*`) and want your JWT team tokens to access these routes, add `mapped_pass_through_routes` to the `team_allowed_routes` in `litellm_jwtauth` or explicitly add the relevant route(s).
Example (`proxy_server_config.yaml`):
```yaml
general_settings:
enable_jwt_auth: True
litellm_jwtauth:
team_ids_jwt_field: "team_ids"
team_allowed_routes: ["openai_routes","info_routes","mapped_pass_through_routes"]
```
### Getting Help
[Schedule Demo 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)

52
docs/my-website/docs/proxy/token_auth.md
View file

@ -338,6 +338,58 @@ general_settings:
team_allowed_routes: ["/v1/chat/completions"] # 👈 Set accepted routes
```
### Allowing other provider routes for Teams
To enable team JWT tokens to access Anthropic-style endpoints such as `/v1/messages`, update `team_allowed_routes` in your `litellm_jwtauth` configuration. `team_allowed_routes` supports the following values:
- Named route groups from `LiteLLMRoutes` (e.g., `openai_routes`, `anthropic_routes`, `info_routes`, `mapped_pass_through_routes`).
Below is a quick reference for the route groups you can use and example representative routes from each group. If you need the exhaustive list, see the `LiteLLMRoutes` enum in `litellm/proxy/_types.py` for the authoritative list.
| Route Group | What it contains | Representative routes |
|-------------|------------------|-----------------------|
| `openai_routes` | OpenAI-compatible REST endpoints (chat, completion, embeddings, images, responses, models, etc.) | `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/images/generations`, `/v1/models` |
| `anthropic_routes` | Anthropic-style endpoints (`/v1/messages` and related) | `/v1/messages`, `/v1/messages/count_tokens`, `/v1/skills` |
| `mapped_pass_through_routes` | Provider-specific pass-through route prefixes (e.g., Anthropic when proxied via `/anthropic`). Use with `mapped_pass_through_routes` for provider wildcard mapping | `/anthropic/*`, `/vertex-ai/*`, `/bedrock/*` |
| `passthrough_routes_wildcard` | Wildcard mapping for providers (e.g., `/anthropic/*`) - precomputed wildcard list used by the proxy | `/anthropic/*`, `/vllm/*` |
| `google_routes` | Google-specific (e.g., Vertex / Batching endpoints) | `/v1beta/models/{model_name}:generateContent` |
| `mcp_routes` | Internal MCP management endpoints | `/mcp/tools`, `/mcp/tools/call` |
| `info_routes` | Read-only & info endpoints used by the UI | `/key/info`, `/team/info`, `/v1/models` |
| `management_routes` | Admin-only management endpoints (create/update/delete user/team/model) | `/team/new`, `/key/generate`, `/model/new` |
| `spend_tracking_routes` | Budget/spend related endpoints | `/spend/logs`, `/spend/keys` |
| `public_routes` | Public and unauthenticated endpoints | `/`, `/routes`, `/.well-known/litellm-ui-config` |
Note: `llm_api_routes` is the union of OpenAI, Anthropic, Google, pass-through and other LLM routes (`openai_routes + anthropic_routes + google_routes + mapped_pass_through_routes + passthrough_routes_wildcard + apply_guardrail_routes + mcp_routes + litellm_native_routes`).
Defaults (what the proxy uses if you don't override them in `litellm_jwtauth`):
- `admin_jwt_scope`: `litellm_proxy_admin`
- `admin_allowed_routes` (default): `management_routes`, `spend_tracking_routes`, `global_spend_tracking_routes`, `info_routes`
- `team_allowed_routes` (default): `openai_routes`, `info_routes`
- `public_allowed_routes` (default): `public_routes`
Example: Allow team JWTs to call Anthropic `/v1/messages` (either by route group or by explicit route string):
```yaml
general_settings:
enable_jwt_auth: True
litellm_jwtauth:
team_ids_jwt_field: "team_ids"
team_allowed_routes: ["openai_routes", "info_routes", "anthropic_routes"]
```
Or selectively allow the exact Anthropic message endpoint only:
```yaml
general_settings:
enable_jwt_auth: True
litellm_jwtauth:
team_ids_jwt_field: "team_ids"
team_allowed_routes: ["/v1/messages", "info_routes"]
```
### Caching Public Keys
Control how long public keys are cached for (in seconds).

19
docs/my-website/docs/secret_managers/cyberark.md
View file

@ -41,6 +41,7 @@ CYBERARK_CLIENT_KEY="path/to/client.key"
# OPTIONAL
CYBERARK_REFRESH_INTERVAL="300" # defaults to 300 seconds (5 minutes), frequency of token refresh
CYBERARK_SSL_VERIFY="true" # defaults to true, set to "false" to disable SSL verification (for self-signed certificates)
```
**Step 2.** Add to proxy config.yaml
@ -172,6 +173,24 @@ If these commands work successfully against your CyberArk instance, then CyberAr
- The `CYBERARK_API_BASE` URL is accessible from your LiteLLM instance
- Your API key or certificates have the necessary permissions in CyberArk
### SSL Certificate Errors
If you encounter SSL certificate verification errors like:
```
RuntimeError: Could not authenticate to CyberArk Conjur: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain
```
This typically occurs when your CyberArk Conjur instance uses a self-signed certificate. You can disable SSL verification by setting:
```bash
CYBERARK_SSL_VERIFY="false"
```
:::warning
Disabling SSL verification is insecure and should only be used for testing or development environments with self-signed certificates. For production, configure your certificate chain properly or use certificate-based authentication with `CYBERARK_CLIENT_CERT` and `CYBERARK_CLIENT_KEY`.
:::
## Video Walkthrough
This video walks through using CyberArk Conjur as a secret manager with LiteLLM. We create a virtual key in the LiteLLM Admin UI and verify it exists in CyberArk. Then we rotate the secret key and verify it exists in CyberArk.

1
docs/my-website/docs/vector_stores/create.md
View file

@ -14,6 +14,7 @@ Create a vector store which can be used to store and search document chunks for
| End-user Tracking | ✅ | |
| Support LLM Providers (OpenAI `/vector_stores` API) | **OpenAI** | Full vector stores API support across providers |
| Support LLM Providers (Passthrough API) | [**Azure AI**](/docs/providers/azure_ai/azure_ai_vector_stores_passthrough) | Full vector stores API support across providers |
| Support LLM Providers (Dataset Management) | [**RAGFlow**](/docs/providers/ragflow_vector_store.md) | Dataset creation and management (search not supported) |
## Usage

0
docs/my-website/img/add_agent1.png Normal file
View file

BIN
docs/my-website/img/add_agent_1.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 288 KiB

BIN
docs/my-website/img/agent2.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 806 KiB

4
docs/my-website/release_notes/v1.80.5-stable/index.md
View file

@ -1,5 +1,5 @@
---
title: "[PREVIEW] v1.80.5.rc.2 - Gemini 3.0 Support"
title: "v1.80.5-stable - Gemini 3.0 Support"
slug: "v1-80-5"
date: 2025-11-22T10:00:00
authors:
@ -27,7 +27,7 @@ import TabItem from '@theme/TabItem';
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.80.5.rc.2
ghcr.io/berriai/litellm:v1.80.5-stable
```
</TabItem>

9
docs/my-website/sidebars.js
View file

@ -141,6 +141,7 @@ const sidebars = {
"proxy/quick_start",
"proxy/cli",
"proxy/debugging",
"proxy/error_diagnosis",
"proxy/deploy",
"proxy/health",
"proxy/master_key_rotations",
@ -315,6 +316,7 @@ const sidebars = {
slug: "/supported_endpoints",
},
items: [
"a2a",
"assistants",
{
type: "category",
@ -520,6 +522,7 @@ const sidebars = {
"providers/vertex_partner",
"providers/vertex_self_deployed",
"providers/vertex_image",
"providers/vertex_speech",
"providers/vertex_batch",
"providers/vertex_ocr",
]
@ -625,6 +628,7 @@ const sidebars = {
"providers/petals",
"providers/publicai",
"providers/predibase",
"providers/ragflow",
"providers/recraft",
"providers/replicate",
{
@ -818,10 +822,13 @@ const sidebars = {
"Learn how to deploy + call models from different providers on LiteLLM",
slug: "/project",
},
items: [
items: [
"projects/smolagents",
"projects/mini-swe-agent",
"projects/openai-agents",
"projects/Google ADK",
"projects/Agent Lightning",
"projects/Harbor",
"projects/Docq.AI",
"projects/PDL",
"projects/OpenInterpreter",

47
docs/my-website/src/pages/intro.md
View file

@ -1,47 +0,0 @@
---
sidebar_position: 1
---
# Tutorial Intro
Let's discover **Docusaurus in less than 5 minutes**.
## Getting Started
Get started by **creating a new site**.
Or **try Docusaurus immediately** with **[docusaurus.new](https://docusaurus.new)**.
### What you'll need
- [Node.js](https://nodejs.org/en/download/) version 16.14 or above:
- When installing Node.js, you are recommended to check all checkboxes related to dependencies.
## Generate a new site
Generate a new Docusaurus site using the **classic template**.
The classic template will automatically be added to your project after you run the command:
```bash
npm init docusaurus@latest my-website classic
```
You can type this command into Command Prompt, Powershell, Terminal, or any other integrated terminal of your code editor.
The command also installs all necessary dependencies you need to run Docusaurus.
## Start your site
Run the development server:
```bash
cd my-website
npm run start
```
The `cd` command changes the directory you're working with. In order to work with your newly created Docusaurus site, you'll need to navigate the terminal there.
The `npm run start` command builds your website locally and serves it through a development server, ready for you to view at http://localhost:3000/.
Open `docs/intro.md` (this page) and edit some lines: the site **reloads automatically** and displays your changes.

8
docs/my-website/src/pages/tutorial-basics/_category_.json
View file

@ -1,8 +0,0 @@
{
"label": "Tutorial - Basics",
"position": 2,
"link": {
"type": "generated-index",
"description": "5 minutes to learn the most important Docusaurus concepts."
}
}

23
docs/my-website/src/pages/tutorial-basics/congratulations.md
View file

@ -1,23 +0,0 @@
---
sidebar_position: 6
---
# Congratulations!
You have just learned the **basics of Docusaurus** and made some changes to the **initial template**.
Docusaurus has **much more to offer**!
Have **5 more minutes**? Take a look at **[versioning](../tutorial-extras/manage-docs-versions.md)** and **[i18n](../tutorial-extras/translate-your-site.md)**.
Anything **unclear** or **buggy** in this tutorial? [Please report it!](https://github.com/facebook/docusaurus/discussions/4610)
## What's next?
- Read the [official documentation](https://docusaurus.io/)
- Modify your site configuration with [`docusaurus.config.js`](https://docusaurus.io/docs/api/docusaurus-config)
- Add navbar and footer items with [`themeConfig`](https://docusaurus.io/docs/api/themes/configuration)
- Add a custom [Design and Layout](https://docusaurus.io/docs/styling-layout)
- Add a [search bar](https://docusaurus.io/docs/search)
- Find inspirations in the [Docusaurus showcase](https://docusaurus.io/showcase)
- Get involved in the [Docusaurus Community](https://docusaurus.io/community/support)

34
docs/my-website/src/pages/tutorial-basics/create-a-blog-post.md
View file

@ -1,34 +0,0 @@
---
sidebar_position: 3
---
# Create a Blog Post
Docusaurus creates a **page for each blog post**, but also a **blog index page**, a **tag system**, an **RSS** feed...
## Create your first Post
Create a file at `blog/2021-02-28-greetings.md`:
```md title="blog/2021-02-28-greetings.md"
---
slug: greetings
title: Greetings!
authors:
- name: Joel Marcey
title: Co-creator of Docusaurus 1
url: https://github.com/JoelMarcey
image_url: https://github.com/JoelMarcey.png
- name: Sébastien Lorber
title: Docusaurus maintainer
url: https://sebastienlorber.com
image_url: https://github.com/slorber.png
tags: [greetings]
---
Congratulations, you have made your first post!
Feel free to play around and edit this post as much you like.
```
A new blog post is now available at [http://localhost:3000/blog/greetings](http://localhost:3000/blog/greetings).

57
docs/my-website/src/pages/tutorial-basics/create-a-document.md
View file

@ -1,57 +0,0 @@
---
sidebar_position: 2
---
# Create a Document
Documents are **groups of pages** connected through:
- a **sidebar**
- **previous/next navigation**
- **versioning**
## Create your first Doc
Create a Markdown file at `docs/hello.md`:
```md title="docs/hello.md"
# Hello
This is my **first Docusaurus document**!
```
A new document is now available at [http://localhost:3000/docs/hello](http://localhost:3000/docs/hello).
## Configure the Sidebar
Docusaurus automatically **creates a sidebar** from the `docs` folder.
Add metadata to customize the sidebar label and position:
```md title="docs/hello.md" {1-4}
---
sidebar_label: 'Hi!'
sidebar_position: 3
---
# Hello
This is my **first Docusaurus document**!
```
It is also possible to create your sidebar explicitly in `sidebars.js`:
```js title="sidebars.js"
module.exports = {
tutorialSidebar: [
'intro',
// highlight-next-line
'hello',
{
type: 'category',
label: 'Tutorial',
items: ['tutorial-basics/create-a-document'],
},
],
};
```

43
docs/my-website/src/pages/tutorial-basics/create-a-page.md
View file

@ -1,43 +0,0 @@
---
sidebar_position: 1
---
# Create a Page
Add **Markdown or React** files to `src/pages` to create a **standalone page**:
- `src/pages/index.js` → `localhost:3000/`
- `src/pages/foo.md` → `localhost:3000/foo`
- `src/pages/foo/bar.js` → `localhost:3000/foo/bar`
## Create your first React Page
Create a file at `src/pages/my-react-page.js`:
```jsx title="src/pages/my-react-page.js"
import React from 'react';
import Layout from '@theme/Layout';
export default function MyReactPage() {
return (
<Layout>
<h1>My React page</h1>
<p>This is a React page</p>
</Layout>
);
}
```
A new page is now available at [http://localhost:3000/my-react-page](http://localhost:3000/my-react-page).
## Create your first Markdown Page
Create a file at `src/pages/my-markdown-page.md`:
```mdx title="src/pages/my-markdown-page.md"
# My Markdown page
This is a Markdown page
```
A new page is now available at [http://localhost:3000/my-markdown-page](http://localhost:3000/my-markdown-page).

31
docs/my-website/src/pages/tutorial-basics/deploy-your-site.md
View file

@ -1,31 +0,0 @@
---
sidebar_position: 5
---
# Deploy your site
Docusaurus is a **static-site-generator** (also called **[Jamstack](https://jamstack.org/)**).
It builds your site as simple **static HTML, JavaScript and CSS files**.
## Build your site
Build your site **for production**:
```bash
npm run build
```
The static files are generated in the `build` folder.
## Deploy your site
Test your production build locally:
```bash
npm run serve
```
The `build` folder is now served at [http://localhost:3000/](http://localhost:3000/).
You can now deploy the `build` folder **almost anywhere** easily, **for free** or very small cost (read the **[Deployment Guide](https://docusaurus.io/docs/deployment)**).

150
docs/my-website/src/pages/tutorial-basics/markdown-features.mdx
View file

@ -1,150 +0,0 @@
---
sidebar_position: 4
---
# Markdown Features
Docusaurus supports **[Markdown](https://daringfireball.net/projects/markdown/syntax)** and a few **additional features**.
## Front Matter
Markdown documents have metadata at the top called [Front Matter](https://jekyllrb.com/docs/front-matter/):
```text title="my-doc.md"
// highlight-start
---
id: my-doc-id
title: My document title
description: My document description
slug: /my-custom-url
---
// highlight-end
## Markdown heading
Markdown text with [links](./hello.md)
```
## Links
Regular Markdown links are supported, using url paths or relative file paths.
```md
Let's see how to [Create a page](/create-a-page).
```
```md
Let's see how to [Create a page](./create-a-page.md).
```
**Result:** Let's see how to [Create a page](./create-a-page.md).
## Images
Regular Markdown images are supported.
You can use absolute paths to reference images in the static directory (`static/img/docusaurus.png`):
```md
![Docusaurus logo](/img/docusaurus.png)
```
![Docusaurus logo](/img/docusaurus.png)
You can reference images relative to the current file as well. This is particularly useful to colocate images close to the Markdown files using them:
```md
![Docusaurus logo](./img/docusaurus.png)
```
## Code Blocks
Markdown code blocks are supported with Syntax highlighting.
```jsx title="src/components/HelloDocusaurus.js"
function HelloDocusaurus() {
return (
<h1>Hello, Docusaurus!</h1>
)
}
```
```jsx title="src/components/HelloDocusaurus.js"
function HelloDocusaurus() {
return <h1>Hello, Docusaurus!</h1>;
}
```
## Admonitions
Docusaurus has a special syntax to create admonitions and callouts:
:::tip My tip
Use this awesome feature option
:::
:::danger Take care
This action is dangerous
:::
:::tip My tip
Use this awesome feature option
:::
:::danger Take care
This action is dangerous
:::
## MDX and React Components
[MDX](https://mdxjs.com/) can make your documentation more **interactive** and allows using any **React components inside Markdown**:
```jsx
export const Highlight = ({children, color}) => (
<span
style={{
backgroundColor: color,
borderRadius: '20px',
color: '#fff',
padding: '10px',
cursor: 'pointer',
}}
onClick={() => {
alert(`You clicked the color ${color} with label ${children}`)
}}>
{children}
</span>
);
This is <Highlight color="#25c2a0">Docusaurus green</Highlight> !
This is <Highlight color="#1877F2">Facebook blue</Highlight> !
```
export const Highlight = ({children, color}) => (
<span
style={{
backgroundColor: color,
borderRadius: '20px',
color: '#fff',
padding: '10px',
cursor: 'pointer',
}}
onClick={() => {
alert(`You clicked the color ${color} with label ${children}`);
}}>
{children}
</span>
);
This is <Highlight color="#25c2a0">Docusaurus green</Highlight> !
This is <Highlight color="#1877F2">Facebook blue</Highlight> !

7
docs/my-website/src/pages/tutorial-extras/_category_.json
View file

@ -1,7 +0,0 @@
{
"label": "Tutorial - Extras",
"position": 3,
"link": {
"type": "generated-index"
}
}

BIN
docs/my-website/src/pages/tutorial-extras/img/docsVersionDropdown.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 25 KiB

BIN
docs/my-website/src/pages/tutorial-extras/img/localeDropdown.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 27 KiB

55
docs/my-website/src/pages/tutorial-extras/manage-docs-versions.md
View file

@ -1,55 +0,0 @@
---
sidebar_position: 1
---
# Manage Docs Versions
Docusaurus can manage multiple versions of your docs.
## Create a docs version
Release a version 1.0 of your project:
```bash
npm run docusaurus docs:version 1.0
```
The `docs` folder is copied into `versioned_docs/version-1.0` and `versions.json` is created.
Your docs now have 2 versions:
- `1.0` at `http://localhost:3000/docs/` for the version 1.0 docs
- `current` at `http://localhost:3000/docs/next/` for the **upcoming, unreleased docs**
## Add a Version Dropdown
To navigate seamlessly across versions, add a version dropdown.
Modify the `docusaurus.config.js` file:
```js title="docusaurus.config.js"
module.exports = {
themeConfig: {
navbar: {
items: [
// highlight-start
{
type: 'docsVersionDropdown',
},
// highlight-end
],
},
},
};
```
The docs version dropdown appears in your navbar:
![Docs Version Dropdown](./img/docsVersionDropdown.png)
## Update an existing version
It is possible to edit versioned docs in their respective folder:
- `versioned_docs/version-1.0/hello.md` updates `http://localhost:3000/docs/hello`
- `docs/hello.md` updates `http://localhost:3000/docs/next/hello`

88
docs/my-website/src/pages/tutorial-extras/translate-your-site.md
View file

@ -1,88 +0,0 @@
---
sidebar_position: 2
---
# Translate your site
Let's translate `docs/intro.md` to French.
## Configure i18n
Modify `docusaurus.config.js` to add support for the `fr` locale:
```js title="docusaurus.config.js"
module.exports = {
i18n: {
defaultLocale: 'en',
locales: ['en', 'fr'],
},
};
```
## Translate a doc
Copy the `docs/intro.md` file to the `i18n/fr` folder:
```bash
mkdir -p i18n/fr/docusaurus-plugin-content-docs/current/
cp docs/intro.md i18n/fr/docusaurus-plugin-content-docs/current/intro.md
```
Translate `i18n/fr/docusaurus-plugin-content-docs/current/intro.md` in French.
## Start your localized site
Start your site on the French locale:
```bash
npm run start -- --locale fr
```
Your localized site is accessible at [http://localhost:3000/fr/](http://localhost:3000/fr/) and the `Getting Started` page is translated.
:::caution
In development, you can only use one locale at a same time.
:::
## Add a Locale Dropdown
To navigate seamlessly across languages, add a locale dropdown.
Modify the `docusaurus.config.js` file:
```js title="docusaurus.config.js"
module.exports = {
themeConfig: {
navbar: {
items: [
// highlight-start
{
type: 'localeDropdown',
},
// highlight-end
],
},
},
};
```
The locale dropdown now appears in your navbar:
![Locale Dropdown](./img/localeDropdown.png)
## Build your localized site
Build your site for a specific locale:
```bash
npm run build -- --locale fr
```
Or build your site to include all the locales at once:
```bash
npm run build
```

337
enterprise/litellm_enterprise/proxy/vector_stores/endpoints.py Normal file
View file

@ -0,0 +1,337 @@
"""
VECTOR STORE MANAGEMENT
All /vector_store management endpoints
/vector_store/new
/vector_store/delete
/vector_store/list
"""
import copy
import json
from typing import List, Optional
from fastapi import APIRouter, Depends, HTTPException
import litellm
from litellm._logging import verbose_proxy_logger
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.proxy._types import (
LiteLLM_ManagedVectorStoresTable,
ResponseLiteLLM_ManagedVectorStore,
UserAPIKeyAuth,
)
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.types.vector_stores import (
LiteLLM_ManagedVectorStore,
LiteLLM_ManagedVectorStoreListResponse,
VectorStoreDeleteRequest,
VectorStoreInfoRequest,
VectorStoreUpdateRequest,
)
from litellm.vector_stores.vector_store_registry import VectorStoreRegistry
router = APIRouter()
########################################################
# Management Endpoints
########################################################
@router.post(
"/vector_store/new",
tags=["vector store management"],
dependencies=[Depends(user_api_key_auth)],
)
async def new_vector_store(
vector_store: LiteLLM_ManagedVectorStore,
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
):
"""
Create a new vector store.
Parameters:
- vector_store_id: str - Unique identifier for the vector store
- custom_llm_provider: str - Provider of the vector store
- vector_store_name: Optional[str] - Name of the vector store
- vector_store_description: Optional[str] - Description of the vector store
- vector_store_metadata: Optional[Dict] - Additional metadata for the vector store
"""
from litellm.proxy.proxy_server import prisma_client
from litellm.types.router import GenericLiteLLMParams
if prisma_client is None:
raise HTTPException(status_code=500, detail="Database not connected")
try:
# Check if vector store already exists
existing_vector_store = (
await prisma_client.db.litellm_managedvectorstorestable.find_unique(
where={"vector_store_id": vector_store.get("vector_store_id")}
)
)
if existing_vector_store is not None:
raise HTTPException(
status_code=400,
detail=f"Vector store with ID {vector_store.get('vector_store_id')} already exists",
)
if vector_store.get("vector_store_metadata") is not None:
vector_store["vector_store_metadata"] = safe_dumps(
vector_store.get("vector_store_metadata")
)
# Safely handle JSON serialization of litellm_params
litellm_params_json: Optional[str] = None
_input_litellm_params: dict = vector_store.get("litellm_params", {}) or {}
if _input_litellm_params is not None:
litellm_params_dict = GenericLiteLLMParams(
**_input_litellm_params
).model_dump(exclude_none=True)
litellm_params_json = safe_dumps(litellm_params_dict)
del vector_store["litellm_params"]
_new_vector_store = (
await prisma_client.db.litellm_managedvectorstorestable.create(
data={
**vector_store,
"litellm_params": litellm_params_json,
}
)
)
new_vector_store: LiteLLM_ManagedVectorStore = LiteLLM_ManagedVectorStore(
**_new_vector_store.model_dump()
)
# Add vector store to registry
if litellm.vector_store_registry is not None:
litellm.vector_store_registry.add_vector_store_to_registry(
vector_store=new_vector_store
)
return {
"status": "success",
"message": f"Vector store {vector_store.get('vector_store_id')} created successfully",
"vector_store": new_vector_store,
}
except Exception as e:
verbose_proxy_logger.exception(f"Error creating vector store: {str(e)}")
raise HTTPException(status_code=500, detail=str(e))
@router.get(
"/vector_store/list",
tags=["vector store management"],
dependencies=[Depends(user_api_key_auth)],
response_model=LiteLLM_ManagedVectorStoreListResponse,
)
async def list_vector_stores(
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
page: int = 1,
page_size: int = 100,
):
"""
List all available vector stores with optional filtering and pagination.
Combines both in-memory vector stores and those stored in the database.
Parameters:
- page: int - Page number for pagination (default: 1)
- page_size: int - Number of items per page (default: 100)
"""
from litellm.proxy.proxy_server import prisma_client
seen_vector_store_ids = set()
try:
# Get in-memory vector stores
in_memory_vector_stores: List[LiteLLM_ManagedVectorStore] = []
if litellm.vector_store_registry is not None:
in_memory_vector_stores = copy.deepcopy(
litellm.vector_store_registry.vector_stores
)
# Get vector stores from database
vector_stores_from_db = await VectorStoreRegistry._get_vector_stores_from_db(
prisma_client=prisma_client
)
# Combine in-memory and database vector stores
combined_vector_stores: List[LiteLLM_ManagedVectorStore] = []
for vector_store in in_memory_vector_stores + vector_stores_from_db:
vector_store_id = vector_store.get("vector_store_id", None)
if vector_store_id not in seen_vector_store_ids:
combined_vector_stores.append(vector_store)
seen_vector_store_ids.add(vector_store_id)
total_count = len(combined_vector_stores)
total_pages = (total_count + page_size - 1) // page_size
# Format response using LiteLLM_ManagedVectorStoreListResponse
response = LiteLLM_ManagedVectorStoreListResponse(
object="list",
data=combined_vector_stores,
total_count=total_count,
current_page=page,
total_pages=total_pages,
)
return response
except Exception as e:
verbose_proxy_logger.exception(f"Error listing vector stores: {str(e)}")
raise HTTPException(status_code=500, detail=str(e))
@router.post(
"/vector_store/delete",
tags=["vector store management"],
dependencies=[Depends(user_api_key_auth)],
)
async def delete_vector_store(
data: VectorStoreDeleteRequest,
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
):
"""
Delete a vector store.
Parameters:
- vector_store_id: str - ID of the vector store to delete
"""
from litellm.proxy.proxy_server import prisma_client
if prisma_client is None:
raise HTTPException(status_code=500, detail="Database not connected")
try:
# Check if vector store exists
existing_vector_store = (
await prisma_client.db.litellm_managedvectorstorestable.find_unique(
where={"vector_store_id": data.vector_store_id}
)
)
if existing_vector_store is None:
raise HTTPException(
status_code=404,
detail=f"Vector store with ID {data.vector_store_id} not found",
)
# Delete vector store
await prisma_client.db.litellm_managedvectorstorestable.delete(
where={"vector_store_id": data.vector_store_id}
)
# Delete vector store from registry
if litellm.vector_store_registry is not None:
litellm.vector_store_registry.delete_vector_store_from_registry(
vector_store_id=data.vector_store_id
)
return {"message": f"Vector store {data.vector_store_id} deleted successfully"}
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
@router.post(
"/vector_store/info",
tags=["vector store management"],
dependencies=[Depends(user_api_key_auth)],
response_model=ResponseLiteLLM_ManagedVectorStore,
)
async def get_vector_store_info(
data: VectorStoreInfoRequest,
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
):
"""Return a single vector store's details"""
from litellm.proxy.proxy_server import prisma_client
if prisma_client is None:
raise HTTPException(status_code=500, detail="Database not connected")
try:
if litellm.vector_store_registry is not None:
vector_store = litellm.vector_store_registry.get_litellm_managed_vector_store_from_registry(
vector_store_id=data.vector_store_id
)
if vector_store is not None:
vector_store_metadata = vector_store.get("vector_store_metadata")
# Parse metadata if it's a JSON string
parsed_metadata: Optional[dict] = None
if isinstance(vector_store_metadata, str):
parsed_metadata = json.loads(vector_store_metadata)
elif isinstance(vector_store_metadata, dict):
parsed_metadata = vector_store_metadata
vector_store_pydantic_obj = LiteLLM_ManagedVectorStoresTable(
vector_store_id=vector_store.get("vector_store_id") or "",
custom_llm_provider=vector_store.get("custom_llm_provider") or "",
vector_store_name=vector_store.get("vector_store_name") or None,
vector_store_description=vector_store.get(
"vector_store_description"
)
or None,
vector_store_metadata=parsed_metadata,
created_at=vector_store.get("created_at") or None,
updated_at=vector_store.get("updated_at") or None,
litellm_credential_name=vector_store.get("litellm_credential_name"),
litellm_params=vector_store.get("litellm_params") or None,
)
return {"vector_store": vector_store_pydantic_obj}
vector_store = (
await prisma_client.db.litellm_managedvectorstorestable.find_unique(
where={"vector_store_id": data.vector_store_id}
)
)
if vector_store is None:
raise HTTPException(
status_code=404,
detail=f"Vector store with ID {data.vector_store_id} not found",
)
vector_store_dict = vector_store.model_dump() # type: ignore[attr-defined]
return {"vector_store": vector_store_dict}
except Exception as e:
verbose_proxy_logger.exception(f"Error getting vector store info: {str(e)}")
raise HTTPException(status_code=500, detail=str(e))
@router.post(
"/vector_store/update",
tags=["vector store management"],
dependencies=[Depends(user_api_key_auth)],
)
async def update_vector_store(
data: VectorStoreUpdateRequest,
user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth),
):
"""Update vector store details"""
from litellm.proxy.proxy_server import prisma_client
if prisma_client is None:
raise HTTPException(status_code=500, detail="Database not connected")
try:
update_data = data.model_dump(exclude_unset=True)
vector_store_id = update_data.pop("vector_store_id")
if update_data.get("vector_store_metadata") is not None:
update_data["vector_store_metadata"] = safe_dumps(
update_data["vector_store_metadata"]
)
updated = await prisma_client.db.litellm_managedvectorstorestable.update(
where={"vector_store_id": vector_store_id},
data=update_data,
)
updated_vs = LiteLLM_ManagedVectorStore(**updated.model_dump())
if litellm.vector_store_registry is not None:
litellm.vector_store_registry.update_vector_store_in_registry(
vector_store_id=vector_store_id,
updated_data=updated_vs,
)
return {"vector_store": updated_vs}
except Exception as e:
verbose_proxy_logger.exception(f"Error updating vector store: {str(e)}")
raise HTTPException(status_code=500, detail=str(e))

152
litellm/__init__.py
View file

@ -151,6 +151,7 @@ _custom_logger_compatible_callbacks_literal = Literal[
"mlflow",
"langfuse",
"langfuse_otel",
"weave_otel",
"pagerduty",
"humanloop",
"gcs_pubsub",
@ -1056,57 +1057,10 @@ from .timeout import timeout
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
from litellm.litellm_core_utils.core_helpers import remove_index_from_tool_calls
from litellm.litellm_core_utils.token_counter import get_modified_max_tokens
from .utils import (
client,
exception_type,
get_optional_params,
get_response_string,
token_counter,
create_pretrained_tokenizer,
create_tokenizer,
supports_function_calling,
supports_web_search,
supports_url_context,
supports_response_schema,
supports_parallel_function_calling,
supports_vision,
supports_audio_input,
supports_audio_output,
supports_system_messages,
supports_reasoning,
get_litellm_params,
acreate,
get_max_tokens,
get_model_info,
register_prompt_template,
validate_environment,
check_valid_key,
register_model,
encode,
decode,
_calculate_retry_after,
_should_retry,
get_supported_openai_params,
get_api_base,
get_first_chars_messages,
ModelResponse,
ModelResponseStream,
EmbeddingResponse,
ImageResponse,
TranscriptionResponse,
TextCompletionResponse,
get_provider_fields,
ModelResponseListIterator,
get_valid_models,
)
ALL_LITELLM_RESPONSE_TYPES = [
ModelResponse,
EmbeddingResponse,
ImageResponse,
TranscriptionResponse,
TextCompletionResponse,
]
# client must be imported immediately as it's used as a decorator at function definition time
from .utils import client
# Note: Most other utils imports are lazy-loaded via __getattr__ to avoid loading utils.py
# (which imports tiktoken) at import time
from .llms.bytez.chat.transformation import BytezChatConfig
from .llms.custom_llm import CustomLLM
@ -1210,6 +1164,9 @@ from .llms.bedrock.chat.invoke_transformations.amazon_ai21_transformation import
from .llms.bedrock.chat.invoke_transformations.amazon_nova_transformation import (
AmazonInvokeNovaConfig,
)
from .llms.bedrock.chat.invoke_transformations.amazon_qwen2_transformation import (
AmazonQwen2Config,
)
from .llms.bedrock.chat.invoke_transformations.amazon_qwen3_transformation import (
AmazonQwen3Config,
)
@ -1387,6 +1344,7 @@ from .llms.docker_model_runner.chat.transformation import DockerModelRunnerChatC
from .llms.v0.chat.transformation import V0ChatConfig
from .llms.oci.chat.transformation import OCIChatConfig
from .llms.morph.chat.transformation import MorphChatConfig
from .llms.ragflow.chat.transformation import RAGFlowConfig
from .llms.lambda_ai.chat.transformation import LambdaAIChatConfig
from .llms.hyperbolic.chat.transformation import HyperbolicChatConfig
from .llms.vercel_ai_gateway.chat.transformation import VercelAIGatewayConfig
@ -1537,56 +1495,6 @@ def set_global_gitlab_config(config: Dict[str, Any]) -> None:
# Lazy loading system for heavy modules to reduce initial import time and memory usage
def _lazy_import_cost_calculator(name: str) -> Any:
"""Lazy import for cost_calculator functions."""
from .cost_calculator import (
completion_cost as _completion_cost,
cost_per_token as _cost_per_token,
response_cost_calculator as _response_cost_calculator,
)
_cost_functions = {
"completion_cost": _completion_cost,
"cost_per_token": _cost_per_token,
"response_cost_calculator": _response_cost_calculator,
}
func = _cost_functions[name]
globals()[name] = func
return func
def _lazy_import_litellm_logging(name: str) -> Any:
"""Lazy import for litellm_logging module."""
try:
from litellm.litellm_core_utils.litellm_logging import (
Logging as _Logging,
modify_integration as _modify_integration,
)
_logging_objects = {
"Logging": _Logging,
"modify_integration": _modify_integration,
}
obj = _logging_objects[name]
globals()[name] = obj
return obj
except Exception as e:
raise AttributeError(
f"module {__name__!r} has no attribute {name!r}. "
f"Lazy import failed: {e}"
) from e
_LAZY_LOAD_REGISTRY: Dict[str, Callable[[str], Any]] = {
"completion_cost": _lazy_import_cost_calculator,
"cost_per_token": _lazy_import_cost_calculator,
"response_cost_calculator": _lazy_import_cost_calculator,
"Logging": _lazy_import_litellm_logging,
"modify_integration": _lazy_import_litellm_logging,
}
if TYPE_CHECKING:
cost_per_token: Callable[..., Tuple[float, float]]
@ -1597,7 +1505,45 @@ if TYPE_CHECKING:
def __getattr__(name: str) -> Any:
"""Lazy import handler for cost_calculator and litellm_logging functions."""
if name in _LAZY_LOAD_REGISTRY:
return _LAZY_LOAD_REGISTRY[name](name)
# Lazy load cost_calculator functions
_cost_calculator_names = (
"completion_cost",
"cost_per_token",
"response_cost_calculator",
)
if name in _cost_calculator_names:
from ._lazy_imports import _lazy_import_cost_calculator
return _lazy_import_cost_calculator(name)
# Lazy load litellm_logging functions
_litellm_logging_names = (
"Logging",
"modify_integration",
)
if name in _litellm_logging_names:
from ._lazy_imports import _lazy_import_litellm_logging
return _lazy_import_litellm_logging(name)
# Lazy load utils functions
_utils_names = (
"exception_type", "get_optional_params", "get_response_string", "token_counter",
"create_pretrained_tokenizer", "create_tokenizer", "supports_function_calling",
"supports_web_search", "supports_url_context", "supports_response_schema",
"supports_parallel_function_calling", "supports_vision", "supports_audio_input",
"supports_audio_output", "supports_system_messages", "supports_reasoning",
"get_litellm_params", "acreate", "get_max_tokens", "get_model_info",
"register_prompt_template", "validate_environment", "check_valid_key",
"register_model", "encode", "decode", "_calculate_retry_after", "_should_retry",
"get_supported_openai_params", "get_api_base", "get_first_chars_messages",
"ModelResponse", "ModelResponseStream", "EmbeddingResponse", "ImageResponse",
"TranscriptionResponse", "TextCompletionResponse", "get_provider_fields",
"ModelResponseListIterator", "get_valid_models",
)
if name in _utils_names:
from ._lazy_imports import _lazy_import_utils
return _lazy_import_utils(name)
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
# ALL_LITELLM_RESPONSE_TYPES is lazy-loaded via __getattr__ to avoid loading utils at import time

259
litellm/_lazy_imports.py Normal file
View file

@ -0,0 +1,259 @@
from typing import Any
import sys
def _get_litellm_globals() -> dict:
"""Helper to get the globals dictionary of the litellm module."""
return sys.modules["litellm"].__dict__
# Lazy import for utils module - imports only the requested item by name.
# Note: PLR0915 (too many statements) is suppressed because the many if statements
# are intentional - each attribute is imported individually only when requested,
# ensuring true lazy imports rather than importing the entire utils module.
def _lazy_import_utils(name: str) -> Any: # noqa: PLR0915
"""Lazy import for utils module - imports only the requested item by name."""
_globals = _get_litellm_globals()
if name == "exception_type":
from .utils import exception_type as _exception_type
_globals["exception_type"] = _exception_type
return _exception_type
if name == "get_optional_params":
from .utils import get_optional_params as _get_optional_params
_globals["get_optional_params"] = _get_optional_params
return _get_optional_params
if name == "get_response_string":
from .utils import get_response_string as _get_response_string
_globals["get_response_string"] = _get_response_string
return _get_response_string
if name == "token_counter":
from .utils import token_counter as _token_counter
_globals["token_counter"] = _token_counter
return _token_counter
if name == "create_pretrained_tokenizer":
from .utils import create_pretrained_tokenizer as _create_pretrained_tokenizer
_globals["create_pretrained_tokenizer"] = _create_pretrained_tokenizer
return _create_pretrained_tokenizer
if name == "create_tokenizer":
from .utils import create_tokenizer as _create_tokenizer
_globals["create_tokenizer"] = _create_tokenizer
return _create_tokenizer
if name == "supports_function_calling":
from .utils import supports_function_calling as _supports_function_calling
_globals["supports_function_calling"] = _supports_function_calling
return _supports_function_calling
if name == "supports_web_search":
from .utils import supports_web_search as _supports_web_search
_globals["supports_web_search"] = _supports_web_search
return _supports_web_search
if name == "supports_url_context":
from .utils import supports_url_context as _supports_url_context
_globals["supports_url_context"] = _supports_url_context
return _supports_url_context
if name == "supports_response_schema":
from .utils import supports_response_schema as _supports_response_schema
_globals["supports_response_schema"] = _supports_response_schema
return _supports_response_schema
if name == "supports_parallel_function_calling":
from .utils import supports_parallel_function_calling as _supports_parallel_function_calling
_globals["supports_parallel_function_calling"] = _supports_parallel_function_calling
return _supports_parallel_function_calling
if name == "supports_vision":
from .utils import supports_vision as _supports_vision
_globals["supports_vision"] = _supports_vision
return _supports_vision
if name == "supports_audio_input":
from .utils import supports_audio_input as _supports_audio_input
_globals["supports_audio_input"] = _supports_audio_input
return _supports_audio_input
if name == "supports_audio_output":
from .utils import supports_audio_output as _supports_audio_output
_globals["supports_audio_output"] = _supports_audio_output
return _supports_audio_output
if name == "supports_system_messages":
from .utils import supports_system_messages as _supports_system_messages
_globals["supports_system_messages"] = _supports_system_messages
return _supports_system_messages
if name == "supports_reasoning":
from .utils import supports_reasoning as _supports_reasoning
_globals["supports_reasoning"] = _supports_reasoning
return _supports_reasoning
if name == "get_litellm_params":
from .utils import get_litellm_params as _get_litellm_params
_globals["get_litellm_params"] = _get_litellm_params
return _get_litellm_params
if name == "acreate":
from .utils import acreate as _acreate
_globals["acreate"] = _acreate
return _acreate
if name == "get_max_tokens":
from .utils import get_max_tokens as _get_max_tokens
_globals["get_max_tokens"] = _get_max_tokens
return _get_max_tokens
if name == "get_model_info":
from .utils import get_model_info as _get_model_info
_globals["get_model_info"] = _get_model_info
return _get_model_info
if name == "register_prompt_template":
from .utils import register_prompt_template as _register_prompt_template
_globals["register_prompt_template"] = _register_prompt_template
return _register_prompt_template
if name == "validate_environment":
from .utils import validate_environment as _validate_environment
_globals["validate_environment"] = _validate_environment
return _validate_environment
if name == "check_valid_key":
from .utils import check_valid_key as _check_valid_key
_globals["check_valid_key"] = _check_valid_key
return _check_valid_key
if name == "register_model":
from .utils import register_model as _register_model
_globals["register_model"] = _register_model
return _register_model
if name == "encode":
from .utils import encode as _encode
_globals["encode"] = _encode
return _encode
if name == "decode":
from .utils import decode as _decode
_globals["decode"] = _decode
return _decode
if name == "_calculate_retry_after":
from .utils import _calculate_retry_after as __calculate_retry_after
_globals["_calculate_retry_after"] = __calculate_retry_after
return __calculate_retry_after
if name == "_should_retry":
from .utils import _should_retry as __should_retry
_globals["_should_retry"] = __should_retry
return __should_retry
if name == "get_supported_openai_params":
from .utils import get_supported_openai_params as _get_supported_openai_params
_globals["get_supported_openai_params"] = _get_supported_openai_params
return _get_supported_openai_params
if name == "get_api_base":
from .utils import get_api_base as _get_api_base
_globals["get_api_base"] = _get_api_base
return _get_api_base
if name == "get_first_chars_messages":
from .utils import get_first_chars_messages as _get_first_chars_messages
_globals["get_first_chars_messages"] = _get_first_chars_messages
return _get_first_chars_messages
if name == "ModelResponse":
from .utils import ModelResponse as _ModelResponse
_globals["ModelResponse"] = _ModelResponse
return _ModelResponse
if name == "ModelResponseStream":
from .utils import ModelResponseStream as _ModelResponseStream
_globals["ModelResponseStream"] = _ModelResponseStream
return _ModelResponseStream
if name == "EmbeddingResponse":
from .utils import EmbeddingResponse as _EmbeddingResponse
_globals["EmbeddingResponse"] = _EmbeddingResponse
return _EmbeddingResponse
if name == "ImageResponse":
from .utils import ImageResponse as _ImageResponse
_globals["ImageResponse"] = _ImageResponse
return _ImageResponse
if name == "TranscriptionResponse":
from .utils import TranscriptionResponse as _TranscriptionResponse
_globals["TranscriptionResponse"] = _TranscriptionResponse
return _TranscriptionResponse
if name == "TextCompletionResponse":
from .utils import TextCompletionResponse as _TextCompletionResponse
_globals["TextCompletionResponse"] = _TextCompletionResponse
return _TextCompletionResponse
if name == "get_provider_fields":
from .utils import get_provider_fields as _get_provider_fields
_globals["get_provider_fields"] = _get_provider_fields
return _get_provider_fields
if name == "ModelResponseListIterator":
from .utils import ModelResponseListIterator as _ModelResponseListIterator
_globals["ModelResponseListIterator"] = _ModelResponseListIterator
return _ModelResponseListIterator
if name == "get_valid_models":
from .utils import get_valid_models as _get_valid_models
_globals["get_valid_models"] = _get_valid_models
return _get_valid_models
raise AttributeError(f"Utils lazy import: unknown attribute {name!r}")
def _lazy_import_cost_calculator(name: str) -> Any:
"""Lazy import for cost_calculator functions."""
_globals = _get_litellm_globals()
from .cost_calculator import (
completion_cost as _completion_cost,
cost_per_token as _cost_per_token,
response_cost_calculator as _response_cost_calculator,
)
_cost_functions = {
"completion_cost": _completion_cost,
"cost_per_token": _cost_per_token,
"response_cost_calculator": _response_cost_calculator,
}
func = _cost_functions[name]
_globals[name] = func
return func
def _lazy_import_litellm_logging(name: str) -> Any:
"""Lazy import for litellm_logging module."""
_globals = _get_litellm_globals()
try:
from litellm.litellm_core_utils.litellm_logging import (
Logging as _Logging,
modify_integration as _modify_integration,
)
_logging_objects = {
"Logging": _Logging,
"modify_integration": _modify_integration,
}
obj = _logging_objects[name]
_globals[name] = obj
return obj
except Exception as e:
raise AttributeError(
f"module 'litellm' has no attribute {name!r}. "
f"Lazy import failed: {e}"
) from e

59
litellm/a2a_protocol/__init__.py Normal file
View file

@ -0,0 +1,59 @@
"""
LiteLLM A2A - Wrapper for invoking A2A protocol agents.
This module provides a thin wrapper around the official `a2a` SDK that:
- Handles httpx client creation and agent card resolution
- Adds LiteLLM logging via @client decorator
- Matches the A2A SDK interface (SendMessageRequest, SendMessageResponse, etc.)
Example usage (standalone functions with @client decorator):
```python
from litellm.a2a_protocol import asend_message
from a2a.types import SendMessageRequest, MessageSendParams
from uuid import uuid4
request = SendMessageRequest(
id=str(uuid4()),
params=MessageSendParams(
message={
"role": "user",
"parts": [{"kind": "text", "text": "Hello!"}],
"messageId": uuid4().hex,
}
)
)
response = await asend_message(
base_url="http://localhost:10001",
request=request,
)
print(response.model_dump(mode='json', exclude_none=True))
```
Example usage (class-based):
```python
from litellm.a2a_protocol import A2AClient
client = A2AClient(base_url="http://localhost:10001")
response = await client.send_message(request)
```
"""
from litellm.a2a_protocol.client import A2AClient
from litellm.a2a_protocol.main import (
aget_agent_card,
asend_message,
asend_message_streaming,
create_a2a_client,
send_message,
)
from litellm.types.agents import LiteLLMSendMessageResponse
__all__ = [
"A2AClient",
"asend_message",
"send_message",
"asend_message_streaming",
"aget_agent_card",
"create_a2a_client",
"LiteLLMSendMessageResponse",
]

107
litellm/a2a_protocol/client.py Normal file
View file

@ -0,0 +1,107 @@
"""
LiteLLM A2A Client class.
Provides a class-based interface for A2A agent invocation.
"""
from typing import TYPE_CHECKING, AsyncIterator, Dict, Optional
from litellm.types.agents import LiteLLMSendMessageResponse
if TYPE_CHECKING:
from a2a.client import A2AClient as A2AClientType
from a2a.types import (
AgentCard,
SendMessageRequest,
SendStreamingMessageRequest,
SendStreamingMessageResponse,
)
class A2AClient:
"""
LiteLLM wrapper for A2A agent invocation.
Creates the underlying A2A client once on first use and reuses it.
Example:
```python
from litellm.a2a_protocol import A2AClient
from a2a.types import SendMessageRequest, MessageSendParams
from uuid import uuid4
client = A2AClient(base_url="http://localhost:10001")
request = SendMessageRequest(
id=str(uuid4()),
params=MessageSendParams(
message={
"role": "user",
"parts": [{"kind": "text", "text": "Hello!"}],
"messageId": uuid4().hex,
}
)
)
response = await client.send_message(request)
```
"""
def __init__(
self,
base_url: str,
timeout: float = 60.0,
extra_headers: Optional[Dict[str, str]] = None,
):
"""
Initialize the A2A client wrapper.
Args:
base_url: The base URL of the A2A agent (e.g., "http://localhost:10001")
timeout: Request timeout in seconds (default: 60.0)
extra_headers: Optional additional headers to include in requests
"""
self.base_url = base_url
self.timeout = timeout
self.extra_headers = extra_headers
self._a2a_client: Optional["A2AClientType"] = None
async def _get_client(self) -> "A2AClientType":
"""Get or create the underlying A2A client."""
if self._a2a_client is None:
from litellm.a2a_protocol.main import create_a2a_client
self._a2a_client = await create_a2a_client(
base_url=self.base_url,
timeout=self.timeout,
extra_headers=self.extra_headers,
)
return self._a2a_client
async def get_agent_card(self) -> "AgentCard":
"""Fetch the agent card from the server."""
from litellm.a2a_protocol.main import aget_agent_card
return await aget_agent_card(
base_url=self.base_url,
timeout=self.timeout,
extra_headers=self.extra_headers,
)
async def send_message(
self, request: "SendMessageRequest"
) -> LiteLLMSendMessageResponse:
"""Send a message to the A2A agent."""
from litellm.a2a_protocol.main import asend_message
a2a_client = await self._get_client()
return await asend_message(a2a_client=a2a_client, request=request)
async def send_message_streaming(
self, request: "SendStreamingMessageRequest"
) -> AsyncIterator["SendStreamingMessageResponse"]:
"""Send a streaming message to the A2A agent."""
from litellm.a2a_protocol.main import asend_message_streaming
a2a_client = await self._get_client()
async for chunk in asend_message_streaming(a2a_client=a2a_client, request=request):
yield chunk

36
litellm/a2a_protocol/cost_calculator.py Normal file
View file

@ -0,0 +1,36 @@
"""
Cost calculator for A2A (Agent-to-Agent) calls.
"""
from typing import TYPE_CHECKING, Any, Optional
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import (
Logging as LitellmLoggingObject,
)
else:
LitellmLoggingObject = Any
class A2ACostCalculator:
@staticmethod
def calculate_a2a_cost(
litellm_logging_obj: Optional[LitellmLoggingObject],
) -> float:
"""
Calculate the cost of an A2A send_message call.
Default is 0.0. In the future, users can configure cost per agent call.
"""
if litellm_logging_obj is None:
return 0.0
# Check if user set a custom response cost
response_cost = litellm_logging_obj.model_call_details.get(
"response_cost", None
)
if response_cost is not None:
return response_cost
# Default to 0.0 for A2A calls
return 0.0

298
litellm/a2a_protocol/main.py Normal file
View file

@ -0,0 +1,298 @@
"""
LiteLLM A2A SDK functions.
Provides standalone functions with @client decorator for LiteLLM logging integration.
"""
import asyncio
from typing import TYPE_CHECKING, Any, AsyncIterator, Coroutine, Dict, Optional, Union
from litellm._logging import verbose_logger
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
)
from litellm.types.agents import LiteLLMSendMessageResponse
from litellm.utils import client
if TYPE_CHECKING:
from a2a.client import A2AClient as A2AClientType
from a2a.types import (
AgentCard,
SendMessageRequest,
SendStreamingMessageRequest,
SendStreamingMessageResponse,
)
# Runtime imports with availability check
A2A_SDK_AVAILABLE = False
A2ACardResolver: Any = None
_A2AClient: Any = None
try:
from a2a.client import A2ACardResolver # type: ignore[no-redef]
from a2a.client import A2AClient as _A2AClient # type: ignore[no-redef]
A2A_SDK_AVAILABLE = True
except ImportError:
pass
def _get_a2a_model_info(a2a_client: Any, kwargs: Dict[str, Any]) -> str:
"""
Extract agent info and set model/custom_llm_provider for cost tracking.
Sets model info on the litellm_logging_obj if available.
Returns the agent name for logging.
"""
agent_name = "unknown"
# Try to get agent card from our stored attribute first, then fallback to SDK attribute
agent_card = getattr(a2a_client, "_litellm_agent_card", None)
if agent_card is None:
agent_card = getattr(a2a_client, "agent_card", None)
if agent_card is not None:
agent_name = getattr(agent_card, "name", "unknown") or "unknown"
# Build model string
model = f"a2a_agent/{agent_name}"
custom_llm_provider = "a2a_agent"
# Set on litellm_logging_obj if available (for standard logging payload)
litellm_logging_obj = kwargs.get("litellm_logging_obj")
if litellm_logging_obj is not None:
litellm_logging_obj.model = model
litellm_logging_obj.custom_llm_provider = custom_llm_provider
litellm_logging_obj.model_call_details["model"] = model
litellm_logging_obj.model_call_details["custom_llm_provider"] = custom_llm_provider
return agent_name
@client
async def asend_message(
a2a_client: "A2AClientType",
request: "SendMessageRequest",
**kwargs: Any,
) -> LiteLLMSendMessageResponse:
"""
Async: Send a message to an A2A agent.
Uses the @client decorator for LiteLLM logging and tracking.
Args:
a2a_client: An initialized a2a.client.A2AClient instance
request: SendMessageRequest from a2a.types
**kwargs: Additional arguments passed to the client decorator
Returns:
LiteLLMSendMessageResponse (wraps a2a SendMessageResponse with _hidden_params)
Example:
```python
from litellm.a2a_protocol import asend_message, create_a2a_client
from a2a.types import SendMessageRequest, MessageSendParams
from uuid import uuid4
# Create client once
a2a_client = await create_a2a_client(base_url="http://localhost:10001")
# Use it for multiple requests
request = SendMessageRequest(
id=str(uuid4()),
params=MessageSendParams(
message={
"role": "user",
"parts": [{"kind": "text", "text": "Hello!"}],
"messageId": uuid4().hex,
}
)
)
response = await asend_message(a2a_client=a2a_client, request=request)
```
"""
agent_name = _get_a2a_model_info(a2a_client, kwargs)
verbose_logger.info(f"A2A send_message request_id={request.id}, agent={agent_name}")
a2a_response = await a2a_client.send_message(request)
verbose_logger.info(f"A2A send_message completed, request_id={request.id}")
# Wrap in LiteLLM response type for _hidden_params support
response = LiteLLMSendMessageResponse.from_a2a_response(a2a_response)
return response
@client
def send_message(
a2a_client: "A2AClientType",
request: "SendMessageRequest",
**kwargs: Any,
) -> Union[LiteLLMSendMessageResponse, Coroutine[Any, Any, LiteLLMSendMessageResponse]]:
"""
Sync: Send a message to an A2A agent.
Uses the @client decorator for LiteLLM logging and tracking.
Args:
a2a_client: An initialized a2a.client.A2AClient instance
request: SendMessageRequest from a2a.types
**kwargs: Additional arguments passed to the client decorator
Returns:
LiteLLMSendMessageResponse (wraps a2a SendMessageResponse with _hidden_params)
"""
try:
loop = asyncio.get_running_loop()
except RuntimeError:
loop = None
if loop is not None:
return asend_message(a2a_client=a2a_client, request=request, **kwargs)
else:
return asyncio.run(asend_message(a2a_client=a2a_client, request=request, **kwargs))
async def asend_message_streaming(
a2a_client: "A2AClientType",
request: "SendStreamingMessageRequest",
) -> AsyncIterator["SendStreamingMessageResponse"]:
"""
Async: Send a streaming message to an A2A agent.
Args:
a2a_client: An initialized a2a.client.A2AClient instance
request: SendStreamingMessageRequest from a2a.types
Yields:
SendStreamingMessageResponse chunks from the agent
"""
verbose_logger.info(f"A2A send_message_streaming request_id={request.id}")
stream = a2a_client.send_message_streaming(request)
chunk_count = 0
async for chunk in stream:
chunk_count += 1
yield chunk
verbose_logger.info(
f"A2A send_message_streaming completed, request_id={request.id}, chunks={chunk_count}"
)
async def create_a2a_client(
base_url: str,
timeout: float = 60.0,
extra_headers: Optional[Dict[str, str]] = None,
) -> "A2AClientType":
"""
Create an A2A client for the given agent URL.
This resolves the agent card and returns a ready-to-use A2A client.
The client can be reused for multiple requests.
Args:
base_url: The base URL of the A2A agent (e.g., "http://localhost:10001")
timeout: Request timeout in seconds (default: 60.0)
extra_headers: Optional additional headers to include in requests
Returns:
An initialized a2a.client.A2AClient instance
Example:
```python
from litellm.a2a_protocol import create_a2a_client, asend_message
# Create client once
client = await create_a2a_client(base_url="http://localhost:10001")
# Reuse for multiple requests
response1 = await asend_message(a2a_client=client, request=request1)
response2 = await asend_message(a2a_client=client, request=request2)
```
"""
if not A2A_SDK_AVAILABLE:
raise ImportError(
"The 'a2a' package is required for A2A agent invocation. "
"Install it with: pip install a2a"
)
verbose_logger.info(f"Creating A2A client for {base_url}")
# Use LiteLLM's cached httpx client
http_handler = get_async_httpx_client(
llm_provider=httpxSpecialProvider.A2A,
params={"timeout": timeout},
)
httpx_client = http_handler.client
# Resolve agent card
resolver = A2ACardResolver(
httpx_client=httpx_client,
base_url=base_url,
)
agent_card = await resolver.get_agent_card()
verbose_logger.debug(
f"Resolved agent card: {agent_card.name if hasattr(agent_card, 'name') else 'unknown'}"
)
# Create A2A client
a2a_client = _A2AClient(
httpx_client=httpx_client,
agent_card=agent_card,
)
# Store agent_card on client for later retrieval (SDK doesn't expose it)
a2a_client._litellm_agent_card = agent_card # type: ignore[attr-defined]
verbose_logger.info(f"A2A client created for {base_url}")
return a2a_client
async def aget_agent_card(
base_url: str,
timeout: float = 60.0,
extra_headers: Optional[Dict[str, str]] = None,
) -> "AgentCard":
"""
Fetch the agent card from an A2A agent.
Args:
base_url: The base URL of the A2A agent (e.g., "http://localhost:10001")
timeout: Request timeout in seconds (default: 60.0)
extra_headers: Optional additional headers to include in requests
Returns:
AgentCard from the A2A agent
"""
if not A2A_SDK_AVAILABLE:
raise ImportError(
"The 'a2a' package is required for A2A agent invocation. "
"Install it with: pip install a2a"
)
verbose_logger.info(f"Fetching agent card from {base_url}")
# Use LiteLLM's cached httpx client
http_handler = get_async_httpx_client(
llm_provider=httpxSpecialProvider.A2A,
params={"timeout": timeout},
)
httpx_client = http_handler.client
resolver = A2ACardResolver(
httpx_client=httpx_client,
base_url=base_url,
)
agent_card = await resolver.get_agent_card()
verbose_logger.info(
f"Fetched agent card: {agent_card.name if hasattr(agent_card, 'name') else 'unknown'}"
)
return agent_card

16
litellm/batches/batch_utils.py
View file

@ -14,8 +14,7 @@ from litellm.utils import token_counter
async def calculate_batch_cost_and_usage(
file_content_dictionary: List[dict],
custom_llm_provider: Literal["openai", "azure", "vertex_ai"],
model_name: Optional[str] = None,
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "hosted_vllm"],
) -> Tuple[float, Usage, List[str]]:
"""
Calculate the cost and usage of a batch
@ -37,8 +36,7 @@ async def calculate_batch_cost_and_usage(
async def _handle_completed_batch(
batch: Batch,
custom_llm_provider: Literal["openai", "azure", "vertex_ai"],
model_name: Optional[str] = None,
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "hosted_vllm"],
) -> Tuple[float, Usage, List[str]]:
"""Helper function to process a completed batch and handle logging"""
# Get batch results
@ -84,8 +82,7 @@ def _get_batch_models_from_file_content(
def _batch_cost_calculator(
file_content_dictionary: List[dict],
custom_llm_provider: Literal["openai", "azure", "vertex_ai"] = "openai",
model_name: Optional[str] = None,
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "hosted_vllm"] = "openai",
) -> float:
"""
Calculate the cost of a batch based on the output file id
@ -186,7 +183,7 @@ def calculate_vertex_ai_batch_cost_and_usage(
async def _get_batch_output_file_content_as_dictionary(
batch: Batch,
custom_llm_provider: Literal["openai", "azure", "vertex_ai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "hosted_vllm"] = "openai",
) -> List[dict]:
"""
Get the batch output file content as a list of dictionaries
@ -225,7 +222,7 @@ def _get_file_content_as_dictionary(file_content: bytes) -> List[dict]:
def _get_batch_job_cost_from_file_content(
file_content_dictionary: List[dict],
custom_llm_provider: Literal["openai", "azure", "vertex_ai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "hosted_vllm"] = "openai",
) -> float:
"""
Get the cost of a batch job from the file content
@ -253,8 +250,7 @@ def _get_batch_job_cost_from_file_content(
def _get_batch_job_total_usage_from_file_content(
file_content_dictionary: List[dict],
custom_llm_provider: Literal["openai", "azure", "vertex_ai"] = "openai",
model_name: Optional[str] = None,
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "hosted_vllm"] = "openai",
) -> Usage:
"""
Get the tokens of a batch job from the file content

40
litellm/batches/main.py
View file

@ -23,6 +23,7 @@ import litellm
from litellm._logging import verbose_logger
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.azure.batches.handler import AzureBatchesAPI
from litellm.llms.bedrock.batches.handler import BedrockBatchesHandler
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
from litellm.llms.custom_httpx.llm_http_handler import BaseLLMHTTPHandler
from litellm.llms.openai.openai import OpenAIBatchesAPI
@ -35,7 +36,11 @@ from litellm.types.llms.openai import (
RetrieveBatchRequest,
)
from litellm.types.router import GenericLiteLLMParams
from litellm.types.utils import LiteLLMBatch, LlmProviders
from litellm.types.utils import (
OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS,
LiteLLMBatch,
LlmProviders,
)
from litellm.utils import (
ProviderConfigManager,
client,
@ -100,7 +105,7 @@ async def acreate_batch(
completion_window: Literal["24h"],
endpoint: Literal["/v1/chat/completions", "/v1/embeddings", "/v1/completions"],
input_file_id: str,
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock"] = "openai",
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock", "hosted_vllm"] = "openai",
metadata: Optional[Dict[str, str]] = None,
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
@ -148,7 +153,7 @@ def create_batch(
completion_window: Literal["24h"],
endpoint: Literal["/v1/chat/completions", "/v1/embeddings", "/v1/completions"],
input_file_id: str,
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock"] = "openai",
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock", "hosted_vllm"] = "openai",
metadata: Optional[Dict[str, str]] = None,
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
@ -235,7 +240,7 @@ def create_batch(
)
return response
api_base: Optional[str] = None
if custom_llm_provider == "openai":
if custom_llm_provider in OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS:
# for deepinfra/perplexity/anyscale/groq we check in get_llm_provider and pass in the api base from there
api_base = (
optional_params.api_base
@ -350,7 +355,7 @@ def create_batch(
@client
async def aretrieve_batch(
batch_id: str,
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock"] = "openai",
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock", "hosted_vllm"] = "openai",
metadata: Optional[Dict[str, str]] = None,
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
@ -396,10 +401,10 @@ def _handle_retrieve_batch_providers_without_provider_config(
litellm_params: dict,
_retrieve_batch_request: RetrieveBatchRequest,
_is_async: bool,
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock"] = "openai",
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock", "hosted_vllm"] = "openai",
):
api_base: Optional[str] = None
if custom_llm_provider == "openai":
if custom_llm_provider in OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS:
# for deepinfra/perplexity/anyscale/groq we check in get_llm_provider and pass in the api base from there
api_base = (
optional_params.api_base
@ -512,7 +517,7 @@ def _handle_retrieve_batch_providers_without_provider_config(
@client
def retrieve_batch(
batch_id: str,
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock"] = "openai",
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock", "hosted_vllm"] = "openai",
metadata: Optional[Dict[str, str]] = None,
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
@ -576,7 +581,7 @@ def retrieve_batch(
async_kwargs = kwargs.copy()
async_kwargs.pop("aws_region_name", None)
return _handle_async_invoke_status(
return BedrockBatchesHandler._handle_async_invoke_status(
batch_id=batch_id,
aws_region_name=kwargs.get("aws_region_name", "us-east-1"),
logging_obj=litellm_logging_obj,
@ -644,7 +649,7 @@ def retrieve_batch(
async def alist_batches(
after: Optional[str] = None,
limit: Optional[int] = None,
custom_llm_provider: Literal["openai", "azure", "vertex_ai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "hosted_vllm", "vertex_ai"] = "openai",
metadata: Optional[Dict[str, str]] = None,
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
@ -687,7 +692,7 @@ async def alist_batches(
def list_batches(
after: Optional[str] = None,
limit: Optional[int] = None,
custom_llm_provider: Literal["openai", "azure", "vertex_ai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "hosted_vllm", "vertex_ai"] = "openai",
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
**kwargs,
@ -727,7 +732,7 @@ def list_batches(
timeout = 600.0
_is_async = kwargs.pop("alist_batches", False) is True
if custom_llm_provider == "openai":
if custom_llm_provider in OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS:
# for deepinfra/perplexity/anyscale/groq we check in get_llm_provider and pass in the api base from there
api_base = (
optional_params.api_base
@ -928,7 +933,7 @@ def cancel_batch(
_is_async = kwargs.pop("acancel_batch", False) is True
api_base: Optional[str] = None
if custom_llm_provider == "openai":
if custom_llm_provider in OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS:
api_base = (
optional_params.api_base
or litellm.api_base
@ -1043,19 +1048,20 @@ def _handle_async_invoke_status(
)
# Transform response to a LiteLLMBatch object
from litellm.types.llms.openai import BatchJobStatus
from litellm.types.utils import LiteLLMBatch
# Normalize status to lowercase (AWS returns 'Completed', 'Failed', etc.)
aws_status_raw = status_response.get("status", "")
aws_status_lower = aws_status_raw.lower()
# Map AWS status values to LiteLLM expected values
status_mapping = {
status_mapping: dict[str, BatchJobStatus] = {
"completed": "completed",
"failed": "failed",
"inprogress": "in_progress",
"in_progress": "in_progress",
}
normalized_status = status_mapping.get(aws_status_lower, aws_status_lower)
normalized_status: BatchJobStatus = status_mapping.get(aws_status_lower, "failed") # Default to "failed" if unknown status
# Get output S3 URI safely
output_s3_uri = ""
@ -1065,13 +1071,15 @@ def _handle_async_invoke_status(
pass
# Use BedrockBatchesConfig's timestamp parsing method (expects raw AWS status string)
import time
from litellm.llms.bedrock.batches.transformation import BedrockBatchesConfig
created_at, in_progress_at, completed_at, failed_at, _, _ = BedrockBatchesConfig()._parse_timestamps_and_status(status_response, aws_status_raw)
result = LiteLLMBatch(
id=status_response["invocationArn"],
object="batch",
status=normalized_status,
created_at=created_at,
created_at=created_at or int(time.time()), # Provide default timestamp if None
in_progress_at=in_progress_at,
completed_at=completed_at,
failed_at=failed_at,

35
litellm/constants.py
View file

@ -1,4 +1,5 @@
import os
import sys
from typing import List, Literal
DEFAULT_HEALTH_CHECK_PROMPT = str(
@ -99,10 +100,18 @@ RUNWAYML_POLLING_TIMEOUT = int(
########## Networking constants ##############################################################
_DEFAULT_TTL_FOR_HTTPX_CLIENTS = 3600 # 1 hour, re-use the same httpx client for 1 hour
# Aiohttp connection pooling constants
AIOHTTP_CONNECTOR_LIMIT = int(os.getenv("AIOHTTP_CONNECTOR_LIMIT", 0))
# Aiohttp connection pooling - prevents memory leaks from unbounded connection growth
# Set to 0 for unlimited (not recommended for production)
AIOHTTP_CONNECTOR_LIMIT = int(os.getenv("AIOHTTP_CONNECTOR_LIMIT", 300))
AIOHTTP_CONNECTOR_LIMIT_PER_HOST = int(os.getenv("AIOHTTP_CONNECTOR_LIMIT_PER_HOST", 50))
AIOHTTP_KEEPALIVE_TIMEOUT = int(os.getenv("AIOHTTP_KEEPALIVE_TIMEOUT", 120))
AIOHTTP_TTL_DNS_CACHE = int(os.getenv("AIOHTTP_TTL_DNS_CACHE", 300))
# enable_cleanup_closed is only needed for Python versions with the SSL leak bug
# Fixed in Python 3.12.7+ and 3.13.1+ (see https://github.com/python/cpython/pull/118960)
# Reference: https://github.com/aio-libs/aiohttp/blob/master/aiohttp/connector.py#L74-L78
AIOHTTP_NEEDS_CLEANUP_CLOSED = (
(3, 13, 0) <= sys.version_info < (3, 13, 1) or sys.version_info < (3, 12, 7)
)
# WebSocket constants
# Default to None (unlimited) to match OpenAI's official agents SDK behavior
@ -255,7 +264,9 @@ TOGETHER_AI_EMBEDDING_350_M = int(os.getenv("TOGETHER_AI_EMBEDDING_350_M", 350))
QDRANT_SCALAR_QUANTILE = float(os.getenv("QDRANT_SCALAR_QUANTILE", 0.99))
QDRANT_VECTOR_SIZE = int(os.getenv("QDRANT_VECTOR_SIZE", 1536))
CACHED_STREAMING_CHUNK_DELAY = float(os.getenv("CACHED_STREAMING_CHUNK_DELAY", 0.02))
AUDIO_SPEECH_CHUNK_SIZE = 8192 # chunk_size for audio speech streaming. Balance between latency and memory usage
AUDIO_SPEECH_CHUNK_SIZE = int(
os.getenv("AUDIO_SPEECH_CHUNK_SIZE", 8192)
) # chunk_size for audio speech streaming. Balance between latency and memory usage
MAX_SIZE_PER_ITEM_IN_MEMORY_CACHE_IN_KB = int(
os.getenv("MAX_SIZE_PER_ITEM_IN_MEMORY_CACHE_IN_KB", 512)
)
@ -278,10 +289,16 @@ REDACTED_BY_LITELM_STRING = "REDACTED_BY_LITELM"
MAX_LANGFUSE_INITIALIZED_CLIENTS = int(
os.getenv("MAX_LANGFUSE_INITIALIZED_CLIENTS", 50)
)
LOGGING_WORKER_CONCURRENCY = int(os.getenv("LOGGING_WORKER_CONCURRENCY", 100)) # Must be above 0
LOGGING_WORKER_CONCURRENCY = int(
os.getenv("LOGGING_WORKER_CONCURRENCY", 100)
) # Must be above 0
LOGGING_WORKER_MAX_QUEUE_SIZE = int(os.getenv("LOGGING_WORKER_MAX_QUEUE_SIZE", 50_000))
LOGGING_WORKER_MAX_TIME_PER_COROUTINE = float(os.getenv("LOGGING_WORKER_MAX_TIME_PER_COROUTINE", 20.0))
LOGGING_WORKER_CLEAR_PERCENTAGE = int(os.getenv("LOGGING_WORKER_CLEAR_PERCENTAGE", 50)) # Percentage of queue to clear (default: 50%)
LOGGING_WORKER_MAX_TIME_PER_COROUTINE = float(
os.getenv("LOGGING_WORKER_MAX_TIME_PER_COROUTINE", 20.0)
)
LOGGING_WORKER_CLEAR_PERCENTAGE = int(
os.getenv("LOGGING_WORKER_CLEAR_PERCENTAGE", 50)
) # Percentage of queue to clear (default: 50%)
MAX_ITERATIONS_TO_CLEAR_QUEUE = int(os.getenv("MAX_ITERATIONS_TO_CLEAR_QUEUE", 200))
MAX_TIME_TO_CLEAR_QUEUE = float(os.getenv("MAX_TIME_TO_CLEAR_QUEUE", 5.0))
LOGGING_WORKER_AGGRESSIVE_CLEAR_COOLDOWN_SECONDS = float(
@ -586,6 +603,7 @@ openai_compatible_providers: List = [
"cometapi",
"clarifai",
"docker_model_runner",
"ragflow",
]
openai_text_completion_compatible_providers: List = (
[ # providers that support `/v1/completions`
@ -858,7 +876,9 @@ BEDROCK_INVOKE_PROVIDERS_LITERAL = Literal[
"nova",
"deepseek_r1",
"qwen3",
"qwen2",
"twelvelabs",
"openai",
]
BEDROCK_EMBEDDING_PROVIDERS_LITERAL = Literal[
@ -908,6 +928,9 @@ BEDROCK_CONVERSE_MODELS = [
"meta.llama3-2-3b-instruct-v1:0",
"meta.llama3-2-11b-instruct-v1:0",
"meta.llama3-2-90b-instruct-v1:0",
"amazon.nova-lite-v1:0",
"amazon.nova-2-lite-v1:0",
"amazon.nova-pro-v1:0",
]

48
litellm/cost_calculator.py
View file

@ -95,6 +95,7 @@ from litellm.utils import (
EmbeddingResponse,
ImageResponse,
ModelResponse,
ModelResponseStream,
ProviderConfigManager,
TextCompletionResponse,
TranscriptionResponse,
@ -654,7 +655,9 @@ def _infer_call_type(
if completion_response is None:
return None
if isinstance(completion_response, ModelResponse):
if isinstance(completion_response, ModelResponse) or isinstance(
completion_response, ModelResponseStream
):
return "completion"
elif isinstance(completion_response, EmbeddingResponse):
return "embedding"
@ -934,6 +937,17 @@ def completion_cost( # noqa: PLR0915
prompt_tokens = token_counter(model=model, text=prompt)
completion_tokens = token_counter(model=model, text=completion)
# Handle A2A calls before model check - A2A doesn't require a model
if call_type in (
CallTypes.asend_message.value,
CallTypes.send_message.value,
):
from litellm.a2a_protocol.cost_calculator import A2ACostCalculator
return A2ACostCalculator.calculate_a2a_cost(
litellm_logging_obj=litellm_logging_obj
)
if model is None:
raise ValueError(
f"Model is None and does not exist in passed completion_response. Passed completion_response={completion_response}, model={model}"
@ -1046,29 +1060,33 @@ def completion_cost( # noqa: PLR0915
number_of_queries = len(query)
elif query is not None:
number_of_queries = 1
search_model = model or ""
if custom_llm_provider and "/" not in search_model:
# If model is like "tavily-search", construct "tavily/search" for cost lookup
search_model = f"{custom_llm_provider}/search"
prompt_cost, completion_cost_result = search_provider_cost_per_query(
model=search_model,
custom_llm_provider=custom_llm_provider,
number_of_queries=number_of_queries,
optional_params=optional_params,
prompt_cost, completion_cost_result = (
search_provider_cost_per_query(
model=search_model,
custom_llm_provider=custom_llm_provider,
number_of_queries=number_of_queries,
optional_params=optional_params,
)
)
# Return the total cost (prompt_cost + completion_cost, but for search it's just prompt_cost)
_final_cost = prompt_cost + completion_cost_result
# Apply discount
original_cost = _final_cost
_final_cost, discount_percent, discount_amount = _apply_cost_discount(
base_cost=_final_cost,
custom_llm_provider=custom_llm_provider,
_final_cost, discount_percent, discount_amount = (
_apply_cost_discount(
base_cost=_final_cost,
custom_llm_provider=custom_llm_provider,
)
)
# Store cost breakdown in logging object if available
_store_cost_breakdown_in_logging_obj(
litellm_logging_obj=litellm_logging_obj,
@ -1080,7 +1098,7 @@ def completion_cost( # noqa: PLR0915
discount_percent=discount_percent,
discount_amount=discount_amount,
)
return _final_cost
elif call_type == CallTypes.arealtime.value and isinstance(
completion_response, LiteLLMRealtimeStreamLoggingObject

42
litellm/files/main.py
View file

@ -18,6 +18,7 @@ from litellm import get_secret_str
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.llms.azure.files.handler import AzureOpenAIFilesAPI
from litellm.llms.bedrock.files.handler import BedrockFilesHandler
from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler, HTTPHandler
from litellm.llms.custom_httpx.llm_http_handler import BaseLLMHTTPHandler
from litellm.llms.openai.openai import FileDeleted, FileObject, OpenAIFilesAPI
@ -30,7 +31,10 @@ from litellm.types.llms.openai import (
OpenAIFileObject,
)
from litellm.types.router import *
from litellm.types.utils import LlmProviders
from litellm.types.utils import (
OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS,
LlmProviders,
)
from litellm.utils import (
ProviderConfigManager,
client,
@ -44,6 +48,7 @@ base_llm_http_handler = BaseLLMHTTPHandler()
openai_files_instance = OpenAIFilesAPI()
azure_files_instance = AzureOpenAIFilesAPI()
vertex_ai_files_instance = VertexAIFilesHandler()
bedrock_files_instance = BedrockFilesHandler()
#################################################
@ -51,7 +56,7 @@ vertex_ai_files_instance = VertexAIFilesHandler()
async def acreate_file(
file: FileTypes,
purpose: Literal["assistants", "batch", "fine-tune"],
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock"] = "openai",
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock", "hosted_vllm"] = "openai",
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
**kwargs,
@ -95,9 +100,7 @@ async def acreate_file(
def create_file(
file: FileTypes,
purpose: Literal["assistants", "batch", "fine-tune"],
custom_llm_provider: Optional[
Literal["openai", "azure", "vertex_ai", "bedrock"]
] = None,
custom_llm_provider: Optional[Literal["openai", "azure", "vertex_ai", "bedrock", "hosted_vllm"]] = None,
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
**kwargs,
@ -165,7 +168,7 @@ def create_file(
),
timeout=timeout,
)
elif custom_llm_provider == "openai":
elif custom_llm_provider in OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS:
# for deepinfra/perplexity/anyscale/groq we check in get_llm_provider and pass in the api base from there
api_base = (
optional_params.api_base
@ -276,7 +279,7 @@ def create_file(
@client
async def afile_retrieve(
file_id: str,
custom_llm_provider: Literal["openai", "azure"] = "openai",
custom_llm_provider: Literal["openai", "azure", "hosted_vllm"] = "openai",
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
**kwargs,
@ -317,7 +320,7 @@ async def afile_retrieve(
@client
def file_retrieve(
file_id: str,
custom_llm_provider: Literal["openai", "azure"] = "openai",
custom_llm_provider: Literal["openai", "azure", "hosted_vllm"] = "openai",
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
**kwargs,
@ -347,7 +350,7 @@ def file_retrieve(
_is_async = kwargs.pop("is_async", False) is True
if custom_llm_provider == "openai":
if custom_llm_provider in OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS:
# for deepinfra/perplexity/anyscale/groq we check in get_llm_provider and pass in the api base from there
api_base = (
optional_params.api_base
@ -514,7 +517,7 @@ def file_delete(
elif timeout is None:
timeout = 600.0
_is_async = kwargs.pop("is_async", False) is True
if custom_llm_provider == "openai":
if custom_llm_provider in OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS:
# for deepinfra/perplexity/anyscale/groq we check in get_llm_provider and pass in the api base from there
api_base = (
optional_params.api_base
@ -670,7 +673,7 @@ def file_list(
timeout = 600.0
_is_async = kwargs.pop("is_async", False) is True
if custom_llm_provider == "openai":
if custom_llm_provider in OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS:
# for deepinfra/perplexity/anyscale/groq we check in get_llm_provider and pass in the api base from there
api_base = (
optional_params.api_base
@ -754,7 +757,7 @@ def file_list(
@client
async def afile_content(
file_id: str,
custom_llm_provider: Literal["openai", "azure", "vertex_ai"] = "openai",
custom_llm_provider: Literal["openai", "azure", "vertex_ai", "bedrock", "hosted_vllm"] = "openai",
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
**kwargs,
@ -799,7 +802,7 @@ def file_content(
file_id: str,
model: Optional[str] = None,
custom_llm_provider: Optional[
Union[Literal["openai", "azure", "vertex_ai"], str]
Union[Literal["openai", "azure", "vertex_ai", "bedrock", "hosted_vllm"], str]
] = None,
extra_headers: Optional[Dict[str, str]] = None,
extra_body: Optional[Dict[str, str]] = None,
@ -846,7 +849,7 @@ def file_content(
_is_async = kwargs.pop("afile_content", False) is True
if custom_llm_provider == "openai":
if custom_llm_provider in OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS:
# for deepinfra/perplexity/anyscale/groq we check in get_llm_provider and pass in the api base from there
api_base = (
optional_params.api_base
@ -937,9 +940,18 @@ def file_content(
timeout=timeout,
max_retries=optional_params.max_retries,
)
elif custom_llm_provider == "bedrock":
response = bedrock_files_instance.file_content(
_is_async=_is_async,
file_content_request=_file_content_request,
api_base=optional_params.api_base,
optional_params=litellm_params_dict,
timeout=timeout,
max_retries=optional_params.max_retries,
)
else:
raise litellm.exceptions.BadRequestError(
message="LiteLLM doesn't support {} for 'custom_llm_provider'. Supported providers are 'openai', 'azure', 'vertex_ai'.".format(
message="LiteLLM doesn't support {} for 'custom_llm_provider'. Supported providers are 'openai', 'azure', 'vertex_ai', 'bedrock'.".format(
custom_llm_provider
),
model="n/a",

4
litellm/images/main.py
View file

@ -6,7 +6,9 @@ from typing import Any, Coroutine, Dict, List, Literal, Optional, Union, cast, o
import httpx
import litellm
from litellm import client, exception_type, get_litellm_params
from litellm.utils import exception_type, get_litellm_params
# client is imported from litellm as it's a decorator
from litellm import client
from litellm.constants import DEFAULT_IMAGE_ENDPOINT_MODEL
from litellm.constants import request_timeout as DEFAULT_REQUEST_TIMEOUT
from litellm.exceptions import LiteLLMUnknownProvider

10
litellm/integrations/SlackAlerting/budget_alert_types.py
View file

@ -50,6 +50,14 @@ class TeamBudgetAlert(BaseBudgetAlertType):
return user_info.team_id or "default_id"
class OrganizationBudgetAlert(BaseBudgetAlertType):
def get_event_message(self) -> str:
return "Organization Budget: "
def get_id(self, user_info: CallInfo) -> str:
return user_info.organization_id or "default_id"
class TokenBudgetAlert(BaseBudgetAlertType):
def get_event_message(self) -> str:
return "Key Budget: "
@ -72,6 +80,7 @@ def get_budget_alert_type(
"soft_budget",
"user_budget",
"team_budget",
"organization_budget",
"proxy_budget",
"projected_limit_exceeded",
],
@ -83,6 +92,7 @@ def get_budget_alert_type(
"soft_budget": SoftBudgetAlert(),
"user_budget": UserBudgetAlert(),
"team_budget": TeamBudgetAlert(),
"organization_budget": OrganizationBudgetAlert(),
"token_budget": TokenBudgetAlert(),
"projected_limit_exceeded": ProjectedLimitExceededAlert(),
}

19
litellm/integrations/SlackAlerting/slack_alerting.py
View file

@ -134,19 +134,25 @@ class SlackAlerting(CustomBatchLogger):
if llm_router is not None:
self.llm_router = llm_router
def _prepare_outage_value_for_cache(self, outage_value: Union[dict, ProviderRegionOutageModel, OutageModel]) -> dict:
def _prepare_outage_value_for_cache(
self, outage_value: Union[dict, ProviderRegionOutageModel, OutageModel]
) -> dict:
"""
Helper method to prepare outage value for Redis caching.
Converts set objects to lists for JSON serialization.
"""
# Convert to dict for processing
cache_value = dict(outage_value)
if "deployment_ids" in cache_value and isinstance(cache_value["deployment_ids"], set):
if "deployment_ids" in cache_value and isinstance(
cache_value["deployment_ids"], set
):
cache_value["deployment_ids"] = list(cache_value["deployment_ids"])
return cache_value
def _restore_outage_value_from_cache(self, outage_value: Optional[dict]) -> Optional[dict]:
def _restore_outage_value_from_cache(
self, outage_value: Optional[dict]
) -> Optional[dict]:
"""
Helper method to restore outage value after retrieving from cache.
Converts list objects back to sets for proper handling.
@ -528,6 +534,7 @@ class SlackAlerting(CustomBatchLogger):
"soft_budget",
"user_budget",
"team_budget",
"organization_budget",
"proxy_budget",
"projected_limit_exceeded",
],
@ -1338,7 +1345,7 @@ Model Info:
subject=email_event["subject"],
html=email_event["html"],
)
if webhook_event.event_group == "team":
if webhook_event.event_group == Litellm_EntityType.TEAM:
from litellm.integrations.email_alerting import send_team_budget_alert
await send_team_budget_alert(webhook_event=webhook_event)
@ -1399,7 +1406,7 @@ Model Info:
current_time = datetime.now().strftime("%H:%M:%S")
_proxy_base_url = os.getenv("PROXY_BASE_URL", None)
# Use .name if it's an enum, otherwise use as is
alert_type_name = getattr(alert_type, 'name', alert_type)
alert_type_name = getattr(alert_type, "name", alert_type)
alert_type_formatted = f"Alert type: `{alert_type_name}`"
if alert_type == "daily_reports" or alert_type == "new_model_added":
formatted_message = alert_type_formatted + message

57
litellm/integrations/arize/arize_phoenix.py
View file

@ -1,18 +1,20 @@
import os
from typing import TYPE_CHECKING, Any, Union
from typing import TYPE_CHECKING, Any, Optional, Union
from datetime import datetime
from litellm._logging import verbose_logger
from litellm.integrations.arize import _utils
from litellm.integrations.arize._utils import ArizeOTELAttributes
from litellm.types.integrations.arize_phoenix import ArizePhoenixConfig
from litellm.types.services import ServiceLoggerPayload
from litellm.integrations.opentelemetry import OpenTelemetry
if TYPE_CHECKING:
from opentelemetry.trace import Span as _Span
from litellm.integrations.opentelemetry import OpenTelemetryConfig as _OpenTelemetryConfig
from litellm.types.integrations.arize import Protocol as _Protocol
from .opentelemetry import OpenTelemetryConfig as _OpenTelemetryConfig
Protocol = _Protocol
OpenTelemetryConfig = _OpenTelemetryConfig
Span = Union[_Span, Any]
@ -25,7 +27,11 @@ else:
ARIZE_HOSTED_PHOENIX_ENDPOINT = "https://otlp.arize.com/v1/traces"
class ArizePhoenixLogger:
class ArizePhoenixLogger(OpenTelemetry):
def set_attributes(self, span: Span, kwargs, response_obj: Optional[Any]):
ArizePhoenixLogger.set_arize_phoenix_attributes(span, kwargs, response_obj)
return
@staticmethod
def set_arize_phoenix_attributes(span: Span, kwargs, response_obj):
_utils.set_attributes(span, kwargs, response_obj, ArizeOTELAttributes)
@ -97,3 +103,46 @@ class ArizePhoenixLogger:
endpoint=endpoint,
project_name=project_name,
)
async def async_service_success_hook(
self,
payload: ServiceLoggerPayload,
parent_otel_span: Optional[Span] = None,
start_time: Optional[Union[datetime, float]] = None,
end_time: Optional[Union[datetime, float]] = None,
event_metadata: Optional[dict] = None,
):
pass # suppress additional spans
async def async_service_failure_hook(
self,
payload: ServiceLoggerPayload,
error: Optional[str] = "",
parent_otel_span: Optional[Span] = None,
start_time: Optional[Union[datetime, float]] = None,
end_time: Optional[Union[float, datetime]] = None,
event_metadata: Optional[dict] = None,
):
pass # suppress additional spans
def create_litellm_proxy_request_started_span(
self,
start_time: datetime,
headers: dict,
):
pass # suppress additional spans
async def async_health_check(self):
config = self.get_arize_phoenix_config()
if not config.otlp_auth_headers:
return {
"status": "unhealthy",
"error_message": "PHOENIX_API_KEY environment variable not set",
}
return {
"status": "healthy",
"message": "Arize-Phoenix credentials are configured properly",
}

18
litellm/integrations/custom_guardrail.py
View file

@ -17,10 +17,10 @@ from litellm.caching import DualCache
from litellm.integrations.custom_logger import CustomLogger
from litellm.types.guardrails import (
DynamicGuardrailParams,
GenericGuardrailAPIInputs,
GuardrailEventHooks,
LitellmParams,
Mode,
PiiEntityType,
)
from litellm.types.llms.openai import AllMessageValues
from litellm.types.proxy.guardrails.guardrail_hooks.base import GuardrailConfigModel
@ -450,20 +450,22 @@ class CustomGuardrail(CustomLogger):
async def apply_guardrail(
self,
texts: List[str],
inputs: GenericGuardrailAPIInputs,
request_data: dict,
input_type: Literal["request", "response"],
logging_obj: Optional["LiteLLMLoggingObj"] = None,
images: Optional[List[str]] = None,
) -> Tuple[List[str], Optional[List[str]]]:
) -> GenericGuardrailAPIInputs:
"""
Apply your guardrail logic to the given text
Apply your guardrail logic to the given inputs
Args:
texts: The texts to apply the guardrail to
images: The images to apply the guardrail to
inputs: Dictionary containing:
- texts: List of texts to apply the guardrail to
- images: Optional list of images to apply the guardrail to
- tool_calls: Optional list of tool calls to apply the guardrail to
request_data: The request data dictionary - containing user api key metadata (e.g. user_id, team_id, etc.)
input_type: The type of input to apply the guardrail to - "request" or "response"
logging_obj: Optional logging object for tracking the guardrail execution
Any of the custom guardrails can override this method to provide custom guardrail logic
@ -474,7 +476,7 @@ class CustomGuardrail(CustomLogger):
- If the guardrail raises an exception
"""
return texts, images
return inputs
def _process_response(
self,

13
litellm/integrations/datadog/datadog.py
View file

@ -65,11 +65,11 @@ class DataDogLogger(
`DD_SITE` - your datadog site, example = `"us5.datadoghq.com"`
Optional environment variables (DataDog Agent):
`DD_AGENT_HOST` - hostname or IP of DataDog agent, example = `"localhost"`
`DD_AGENT_PORT` - port of DataDog agent (default: 10518 for logs)
`LITELLM_DD_AGENT_HOST` - hostname or IP of DataDog agent, example = `"localhost"`
`LITELLM_DD_AGENT_PORT` - port of DataDog agent (default: 10518 for logs)
Note: If DD_AGENT_HOST is set, logs will be sent to the agent instead of directly to DataDog API.
In this case, DD_API_KEY and DD_SITE are not required (agent handles authentication).
Note: We use LITELLM_DD_AGENT_HOST instead of DD_AGENT_HOST to avoid conflicts
with ddtrace which automatically sets DD_AGENT_HOST for APM tracing.
"""
try:
verbose_logger.debug("Datadog: in init datadog logger")
@ -85,7 +85,8 @@ class DataDogLogger(
)
# Configure DataDog endpoint (Agent or Direct API)
dd_agent_host = os.getenv("DD_AGENT_HOST")
# Use LITELLM_DD_AGENT_HOST to avoid conflicts with ddtrace's DD_AGENT_HOST
dd_agent_host = os.getenv("LITELLM_DD_AGENT_HOST")
if dd_agent_host:
self._configure_dd_agent(dd_agent_host=dd_agent_host)
else:
@ -127,7 +128,7 @@ class DataDogLogger(
Args:
dd_agent_host: Hostname or IP of DataDog agent
"""
dd_agent_port = os.getenv("DD_AGENT_PORT", "10518") # default port for logs
dd_agent_port = os.getenv("LITELLM_DD_AGENT_PORT", "10518") # default port for logs
self.intake_url = f"http://{dd_agent_host}:{dd_agent_port}/api/v2/logs"
self.DD_API_KEY = os.getenv("DD_API_KEY") # Optional when using agent
verbose_logger.debug(f"Datadog: Using DD Agent at {self.intake_url}")

14
litellm/integrations/opentelemetry.py
View file

@ -1065,14 +1065,7 @@ class OpenTelemetry(CustomLogger):
self, span: Span, kwargs, response_obj: Optional[Any]
):
try:
if self.callback_name == "arize_phoenix":
from litellm.integrations.arize.arize_phoenix import ArizePhoenixLogger
ArizePhoenixLogger.set_arize_phoenix_attributes(
span, kwargs, response_obj
)
return
elif self.callback_name == "langtrace":
if self.callback_name == "langtrace":
from litellm.integrations.langtrace import LangtraceAttributes
LangtraceAttributes().set_langtrace_attributes(
@ -1088,6 +1081,11 @@ class OpenTelemetry(CustomLogger):
span, kwargs, response_obj
)
return
elif self.callback_name == "weave_otel":
from litellm.integrations.weave.weave_otel import set_weave_otel_attributes
set_weave_otel_attributes(span, kwargs, response_obj)
return
from litellm.proxy._types import SpanAttributes
optional_params = kwargs.get("optional_params", {})

7
litellm/integrations/weave/__init__.py Normal file
View file

@ -0,0 +1,7 @@
"""
Weave (W&B) integration for LiteLLM via OpenTelemetry.
"""
from litellm.integrations.weave.weave_otel import WeaveOtelLogger
__all__ = ["WeaveOtelLogger"]

329
litellm/integrations/weave/weave_otel.py Normal file
View file

@ -0,0 +1,329 @@
from __future__ import annotations
import base64
import json
import os
from typing import TYPE_CHECKING, Any, Optional
from opentelemetry.trace import Status, StatusCode
from typing_extensions import override
from litellm._logging import verbose_logger
from litellm.integrations._types.open_inference import SpanAttributes as OpenInferenceSpanAttributes
from litellm.integrations.arize import _utils
from litellm.integrations.opentelemetry import OpenTelemetry, OpenTelemetryConfig
from litellm.integrations.opentelemetry_utils.base_otel_llm_obs_attributes import (
BaseLLMObsOTELAttributes,
safe_set_attribute,
)
from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
from litellm.types.integrations.weave_otel import WeaveOtelConfig, WeaveSpanAttributes
from litellm.types.utils import StandardCallbackDynamicParams
if TYPE_CHECKING:
from opentelemetry.trace import Span
# Weave OTEL endpoint
# Multi-tenant cloud: https://trace.wandb.ai/otel/v1/traces
# Dedicated cloud: https://<your-subdomain>.wandb.io/traces/otel/v1/traces
WEAVE_BASE_URL = "https://trace.wandb.ai"
WEAVE_OTEL_ENDPOINT = "/otel/v1/traces"
class WeaveLLMObsOTELAttributes(BaseLLMObsOTELAttributes):
"""
Weave-specific LLM observability OTEL attributes.
Weave automatically maps attributes from multiple frameworks including
GenAI, OpenInference, Langfuse, and others.
"""
@staticmethod
@override
def set_messages(span: "Span", kwargs: dict[str, Any]):
"""Set input messages as span attributes using OpenInference conventions."""
messages = kwargs.get("messages") or []
optional_params = kwargs.get("optional_params") or {}
prompt = {"messages": messages}
functions = optional_params.get("functions")
tools = optional_params.get("tools")
if functions is not None:
prompt["functions"] = functions
if tools is not None:
prompt["tools"] = tools
safe_set_attribute(span, OpenInferenceSpanAttributes.INPUT_VALUE, json.dumps(prompt))
def _set_weave_specific_attributes(span: Span, kwargs: dict[str, Any], response_obj: Any):
"""
Sets Weave-specific metadata attributes onto the OTEL span.
Based on Weave's OTEL attribute mappings from:
https://github.com/wandb/weave/blob/master/weave/trace_server/opentelemetry/constants.py
"""
# Extract all needed data upfront
litellm_params = kwargs.get("litellm_params") or {}
# optional_params = kwargs.get("optional_params") or {}
metadata = kwargs.get("metadata") or {}
model = kwargs.get("model") or ""
custom_llm_provider = litellm_params.get("custom_llm_provider") or ""
# Weave supports a custom display name and will default to the model name if not provided.
display_name = metadata.get("display_name")
if not display_name and model:
if custom_llm_provider:
display_name = f"{custom_llm_provider}/{model}"
else:
display_name = model
if display_name:
display_name = display_name.replace("/", "__")
safe_set_attribute(span, WeaveSpanAttributes.DISPLAY_NAME.value, display_name)
# Weave threads are OpenInference sessions.
if (session_id := metadata.get("session_id")) is not None:
if isinstance(session_id, (list, dict)):
session_id = safe_dumps(session_id)
safe_set_attribute(span, WeaveSpanAttributes.THREAD_ID.value, session_id)
safe_set_attribute(span, WeaveSpanAttributes.IS_TURN.value, True)
# Response attributes are already set by _utils.set_attributes,
# but we override them here to better match Weave's expectations
if response_obj:
output_dict = None
if hasattr(response_obj, "model_dump"):
output_dict = response_obj.model_dump()
elif hasattr(response_obj, "get"):
output_dict = response_obj
if output_dict:
safe_set_attribute(span, OpenInferenceSpanAttributes.OUTPUT_VALUE, safe_dumps(output_dict))
def _get_weave_authorization_header(api_key: str) -> str:
"""
Get the authorization header for Weave OpenTelemetry.
Weave uses Basic auth with format: api:<WANDB_API_KEY>
"""
auth_string = f"api:{api_key}"
auth_header = base64.b64encode(auth_string.encode()).decode()
return f"Basic {auth_header}"
def get_weave_otel_config() -> WeaveOtelConfig:
"""
Retrieves the Weave OpenTelemetry configuration based on environment variables.
Environment Variables:
WANDB_API_KEY: Required. W&B API key for authentication.
WANDB_PROJECT_ID: Required. Project ID in format <entity>/<project_name>.
WANDB_HOST: Optional. Custom Weave host URL. Defaults to cloud endpoint.
Returns:
WeaveOtelConfig: A Pydantic model containing Weave OTEL configuration.
Raises:
ValueError: If required environment variables are missing.
"""
api_key = os.getenv("WANDB_API_KEY")
project_id = os.getenv("WANDB_PROJECT_ID")
host = os.getenv("WANDB_HOST")
if not api_key:
raise ValueError("WANDB_API_KEY must be set for Weave OpenTelemetry integration.")
if not project_id:
raise ValueError(
"WANDB_PROJECT_ID must be set for Weave OpenTelemetry integration. Format: <entity>/<project_name>"
)
if host:
if not host.startswith("http"):
host = "https://" + host
# Self-managed instances use a different path
endpoint = host.rstrip("/") + WEAVE_OTEL_ENDPOINT
verbose_logger.debug(f"Using Weave OTEL endpoint from host: {endpoint}")
else:
endpoint = WEAVE_BASE_URL + WEAVE_OTEL_ENDPOINT
verbose_logger.debug(f"Using Weave cloud endpoint: {endpoint}")
# Weave uses Basic auth with format: api:<WANDB_API_KEY>
auth_header = _get_weave_authorization_header(api_key=api_key)
otlp_auth_headers = f"Authorization={auth_header},project_id={project_id}"
# Set standard OTEL environment variables
os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"] = endpoint
os.environ["OTEL_EXPORTER_OTLP_HEADERS"] = otlp_auth_headers
return WeaveOtelConfig(
otlp_auth_headers=otlp_auth_headers,
endpoint=endpoint,
project_id=project_id,
protocol="otlp_http",
)
def set_weave_otel_attributes(span: Span, kwargs: dict[str, Any], response_obj: Any):
"""
Sets OpenTelemetry span attributes for Weave observability.
Uses the same attribute setting logic as other OTEL integrations for consistency.
"""
_utils.set_attributes(span, kwargs, response_obj, WeaveLLMObsOTELAttributes)
_set_weave_specific_attributes(span=span, kwargs=kwargs, response_obj=response_obj)
class WeaveOtelLogger(OpenTelemetry):
"""
Weave (W&B) OpenTelemetry Logger for LiteLLM.
Sends LLM traces to Weave via the OpenTelemetry Protocol (OTLP).
Environment Variables:
WANDB_API_KEY: Required. Weights & Biases API key for authentication.
WANDB_PROJECT_ID: Required. Project ID in format <entity>/<project_name>.
WANDB_HOST: Optional. Custom Weave host URL. Defaults to cloud endpoint.
Usage:
litellm.callbacks = ["weave_otel"]
Or manually:
from litellm.integrations.weave.weave_otel import WeaveOtelLogger
weave_logger = WeaveOtelLogger(callback_name="weave_otel")
litellm.callbacks = [weave_logger]
Reference:
https://docs.wandb.ai/weave/guides/tracking/otel
"""
def __init__(
self,
config: Optional[OpenTelemetryConfig] = None,
callback_name: Optional[str] = "weave_otel",
**kwargs,
):
"""
Initialize WeaveOtelLogger.
If config is not provided, automatically configures from environment variables
(WANDB_API_KEY, WANDB_PROJECT_ID, WANDB_HOST) via get_weave_otel_config().
"""
if config is None:
# Auto-configure from Weave environment variables
weave_config = get_weave_otel_config()
config = OpenTelemetryConfig(
exporter=weave_config.protocol,
endpoint=weave_config.endpoint,
headers=weave_config.otlp_auth_headers,
)
super().__init__(config=config, callback_name=callback_name, **kwargs)
def _maybe_log_raw_request(self, kwargs, response_obj, start_time, end_time, parent_span):
"""
Override to skip creating the raw_gen_ai_request child span.
For Weave, we only want a single span per LLM call. The parent span
already contains all the necessary attributes, so the child span
is redundant.
"""
pass
def _start_primary_span(
self,
kwargs,
response_obj,
start_time,
end_time,
context,
parent_span=None,
):
"""
Override to always create a child span instead of reusing the parent span.
This ensures that wrapper spans (like "B", "C", "D", "E") remain separate
from the LiteLLM LLM call spans, creating proper nesting in Weave.
"""
otel_tracer = self.get_tracer_to_use_for_request(kwargs)
# Always create a new child span, even if parent_span is provided
# This ensures wrapper spans remain separate from LLM call spans
span = otel_tracer.start_span(
name=self._get_span_name(kwargs),
start_time=self._to_ns(start_time),
context=context,
)
span.set_status(Status(StatusCode.OK))
self.set_attributes(span, kwargs, response_obj)
span.end(end_time=self._to_ns(end_time))
return span
def _handle_success(self, kwargs, response_obj, start_time, end_time):
"""
Override to prevent ending externally created parent spans.
When wrapper spans (like "B", "C", "D", "E") are provided as parent spans,
they should be managed by the user code, not ended by LiteLLM.
"""
verbose_logger.debug(
"Weave OpenTelemetry Logger: Logging kwargs: %s, OTEL config settings=%s",
kwargs,
self.config,
)
ctx, parent_span = self._get_span_context(kwargs)
# Always create a child span (handled by _start_primary_span override)
primary_span_parent = None
# 1. Primary span
span = self._start_primary_span(kwargs, response_obj, start_time, end_time, ctx, primary_span_parent)
# 2. Raw-request sub-span (skipped for Weave via _maybe_log_raw_request override)
self._maybe_log_raw_request(kwargs, response_obj, start_time, end_time, span)
# 3. Guardrail span
self._create_guardrail_span(kwargs=kwargs, context=ctx)
# 4. Metrics & cost recording
self._record_metrics(kwargs, response_obj, start_time, end_time)
# 5. Semantic logs.
if self.config.enable_events:
self._emit_semantic_logs(kwargs, response_obj, span)
# 6. Don't end parent span - it's managed by user code
# Since we always create a child span (never reuse parent), the parent span
# lifecycle is owned by the user. This prevents double-ending of wrapper spans
# like "B", "C", "D", "E" that users create and manage themselves.
def construct_dynamic_otel_headers(
self, standard_callback_dynamic_params: StandardCallbackDynamicParams
) -> dict | None:
"""
Construct dynamic Weave headers from standard callback dynamic params.
This is used for team/key based logging.
Returns:
dict: A dictionary of dynamic Weave headers
"""
dynamic_headers = {}
dynamic_wandb_api_key = standard_callback_dynamic_params.get("wandb_api_key")
dynamic_weave_project_id = standard_callback_dynamic_params.get("weave_project_id")
if dynamic_wandb_api_key:
auth_header = _get_weave_authorization_header(
api_key=dynamic_wandb_api_key,
)
dynamic_headers["Authorization"] = auth_header
if dynamic_weave_project_id:
dynamic_headers["project_id"] = dynamic_weave_project_id
return dynamic_headers if dynamic_headers else None

1
litellm/litellm_core_utils/README.md
View file

@ -9,4 +9,5 @@ Core files:
- `default_encoding.py`: code for loading the default encoding (tiktoken)
- `get_llm_provider_logic.py`: code for inferring the LLM provider from a given model name.
- `duration_parser.py`: code for parsing durations - e.g. "1d", "1mo", "10s"
- `api_route_to_call_types.py`: mapping of API routes to their corresponding CallTypes (e.g., `/chat/completions` -> [acompletion, completion])

38
litellm/litellm_core_utils/api_route_to_call_types.py Normal file
View file

@ -0,0 +1,38 @@
"""
Dictionary mapping API routes to their corresponding CallTypes in LiteLLM.
This dictionary maps each API endpoint to the CallTypes that can be used for that route.
Each route can have both async (prefixed with 'a') and sync call types.
"""
from litellm.types.utils import API_ROUTE_TO_CALL_TYPES, CallTypes
def get_call_types_for_route(route: str) -> list:
"""
Get the list of CallTypes for a given API route.
Args:
route: API route path (e.g., "/chat/completions")
Returns:
List of CallTypes for that route, or empty list if route not found
"""
return API_ROUTE_TO_CALL_TYPES.get(route, [])
def get_routes_for_call_type(call_type: CallTypes) -> list:
"""
Get all routes that use a specific CallType.
Args:
call_type: The CallType to search for
Returns:
List of routes that use this CallType
"""
routes = []
for route, types in API_ROUTE_TO_CALL_TYPES.items():
if call_type in types:
routes.append(route)
return routes

1
litellm/litellm_core_utils/custom_logger_registry.py
View file

@ -75,6 +75,7 @@ class CustomLoggerRegistry:
"langfuse_otel": OpenTelemetry,
"arize_phoenix": OpenTelemetry,
"langtrace": OpenTelemetry,
"weave_otel": OpenTelemetry,
"mlflow": MlflowLogger,
"langfuse": LangfusePromptManagement,
"otel": OpenTelemetry,

10
litellm/litellm_core_utils/get_llm_provider_logic.py
View file

@ -840,6 +840,16 @@ def _get_openai_compatible_provider_info( # noqa: PLR0915
) = litellm.ClarifaiConfig()._get_openai_compatible_provider_info(
api_base, api_key
)
elif custom_llm_provider == "ragflow":
full_model = f"ragflow/{model}"
(
api_base,
dynamic_api_key,
_,
) = litellm.RAGFlowConfig()._get_openai_compatible_provider_info(
full_model, api_base, api_key, "ragflow"
)
model = full_model
if api_base is not None and not isinstance(api_base, str):
raise Exception("api base needs to be a string. api_base={}".format(api_base))

35
litellm/litellm_core_utils/litellm_logging.py
View file

@ -71,6 +71,7 @@ from litellm.litellm_core_utils.redact_messages import (
from litellm.llms.base_llm.ocr.transformation import OCRResponse
from litellm.llms.base_llm.search.transformation import SearchResponse
from litellm.responses.utils import ResponseAPILoggingUtils
from litellm.types.agents import LiteLLMSendMessageResponse
from litellm.types.containers.main import ContainerObject
from litellm.types.llms.openai import (
AllMessageValues,
@ -1738,6 +1739,7 @@ class Logging(LiteLLMLoggingBaseClass):
and logging_result.get("object") == "search" # Search API (dict format)
or isinstance(logging_result, VideoObject)
or isinstance(logging_result, ContainerObject)
or isinstance(logging_result, LiteLLMSendMessageResponse) # A2A
or (self.call_type == CallTypes.call_mcp_tool.value)
):
return True
@ -3617,15 +3619,15 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
for callback in _in_memory_loggers:
if (
isinstance(callback, OpenTelemetry)
isinstance(callback, ArizePhoenixLogger)
and callback.callback_name == "arize_phoenix"
):
return callback # type: ignore
_otel_logger = OpenTelemetry(
_arize_phoenix_otel_logger = ArizePhoenixLogger(
config=otel_config, callback_name="arize_phoenix"
)
_in_memory_loggers.append(_otel_logger)
return _otel_logger # type: ignore
_in_memory_loggers.append(_arize_phoenix_otel_logger)
return _arize_phoenix_otel_logger # type: ignore
elif logging_integration == "otel":
from litellm.integrations.opentelemetry import OpenTelemetry
@ -3800,6 +3802,31 @@ def _init_custom_logger_compatible_class( # noqa: PLR0915
)
_in_memory_loggers.append(_otel_logger)
return _otel_logger # type: ignore
elif logging_integration == "weave_otel":
from litellm.integrations.opentelemetry import (
OpenTelemetryConfig,
)
from litellm.integrations.weave.weave_otel import WeaveOtelLogger, get_weave_otel_config
weave_otel_config = get_weave_otel_config()
otel_config = OpenTelemetryConfig(
exporter=weave_otel_config.protocol,
endpoint=weave_otel_config.endpoint,
headers=weave_otel_config.otlp_auth_headers,
)
for callback in _in_memory_loggers:
if (
isinstance(callback, WeaveOtelLogger)
and callback.callback_name == "weave_otel"
):
return callback # type: ignore
_otel_logger = WeaveOtelLogger(
config=otel_config, callback_name="weave_otel"
)
_in_memory_loggers.append(_otel_logger)
return _otel_logger # type: ignore
elif logging_integration == "pagerduty":
for callback in _in_memory_loggers:
if isinstance(callback, PagerDutyAlerting):

54
litellm/litellm_core_utils/prompt_templates/factory.py
View file

@ -3446,8 +3446,25 @@ class BedrockConverseMessagesProcessor:
@staticmethod
def _initial_message_setup(
messages: List,
model: str,
llm_provider: str,
user_continue_message: Optional[ChatCompletionUserMessage] = None,
) -> List:
# gracefully handle base case of no messages at all
if len(messages) == 0:
if user_continue_message is not None:
messages.append(user_continue_message)
elif litellm.modify_params:
messages.append(DEFAULT_USER_CONTINUE_MESSAGE)
else:
raise litellm.BadRequestError(
message=BAD_MESSAGE_ERROR_STR
+ "bedrock requires at least one non-system message",
model=model,
llm_provider=llm_provider,
)
# if initial message is assistant message
if messages[0].get("role") is not None and messages[0]["role"] == "assistant":
if user_continue_message is not None:
messages.insert(0, user_continue_message)
@ -3475,18 +3492,8 @@ class BedrockConverseMessagesProcessor:
contents: List[BedrockMessageBlock] = []
msg_i = 0
## BASE CASE ##
if len(messages) == 0:
raise litellm.BadRequestError(
message=BAD_MESSAGE_ERROR_STR
+ "bedrock requires at least one non-system message",
model=model,
llm_provider=llm_provider,
)
# if initial message is assistant message
messages = BedrockConverseMessagesProcessor._initial_message_setup(
messages, user_continue_message
messages, model, llm_provider, user_continue_message
)
while msg_i < len(messages):
@ -3847,28 +3854,9 @@ def _bedrock_converse_messages_pt( # noqa: PLR0915
contents: List[BedrockMessageBlock] = []
msg_i = 0
## BASE CASE ##
if len(messages) == 0:
raise litellm.BadRequestError(
message=BAD_MESSAGE_ERROR_STR
+ "bedrock requires at least one non-system message",
model=model,
llm_provider=llm_provider,
)
# if initial message is assistant message
if messages[0].get("role") is not None and messages[0]["role"] == "assistant":
if user_continue_message is not None:
messages.insert(0, user_continue_message)
elif litellm.modify_params:
messages.insert(0, DEFAULT_USER_CONTINUE_MESSAGE)
# if final message is assistant message
if messages[-1].get("role") is not None and messages[-1]["role"] == "assistant":
if user_continue_message is not None:
messages.append(user_continue_message)
elif litellm.modify_params:
messages.append(DEFAULT_USER_CONTINUE_MESSAGE)
messages = BedrockConverseMessagesProcessor._initial_message_setup(
messages, model, llm_provider, user_continue_message
)
while msg_i < len(messages):
user_content: List[BedrockContentBlock] = []

53
litellm/litellm_core_utils/streaming_handler.py
View file

@ -96,9 +96,9 @@ class CustomStreamWrapper:
self.system_fingerprint: Optional[str] = None
self.received_finish_reason: Optional[str] = None
self.intermittent_finish_reason: Optional[str] = (
None # finish reasons that show up mid-stream
)
self.intermittent_finish_reason: Optional[
str
] = None # finish reasons that show up mid-stream
self.special_tokens = [
"<|assistant|>",
"<|system|>",
@ -735,7 +735,7 @@ class CustomStreamWrapper:
and completion_obj["function_call"] is not None
)
or (
"tool_calls" in model_response.choices[0].delta
"tool_calls" in model_response.choices[0].delta
and model_response.choices[0].delta["tool_calls"] is not None
)
or (
@ -889,7 +889,6 @@ class CustomStreamWrapper:
## check if openai/azure chunk
original_chunk = response_obj.get("original_chunk", None)
if original_chunk:
if len(original_chunk.choices) > 0:
choices = []
for choice in original_chunk.choices:
@ -906,7 +905,6 @@ class CustomStreamWrapper:
print_verbose(f"choices in streaming: {choices}")
setattr(model_response, "choices", choices)
else:
return
model_response.system_fingerprint = (
original_chunk.system_fingerprint
@ -1435,9 +1433,9 @@ class CustomStreamWrapper:
_json_delta = delta.model_dump()
print_verbose(f"_json_delta: {_json_delta}")
if "role" not in _json_delta or _json_delta["role"] is None:
_json_delta["role"] = (
"assistant" # mistral's api returns role as None
)
_json_delta[
"role"
] = "assistant" # mistral's api returns role as None
if "tool_calls" in _json_delta and isinstance(
_json_delta["tool_calls"], list
):
@ -1533,7 +1531,7 @@ class CustomStreamWrapper:
async def _call_post_streaming_deployment_hook(self, chunk):
"""
Call the post-call streaming deployment hook for callbacks.
This allows callbacks to modify streaming chunks before they're returned.
"""
try:
@ -1544,15 +1542,17 @@ class CustomStreamWrapper:
# Get request kwargs from logging object
request_data = self.logging_obj.model_call_details
call_type_str = self.logging_obj.call_type
try:
typed_call_type = CallTypes(call_type_str)
except ValueError:
typed_call_type = None
# Call hooks for all callbacks
for callback in litellm.callbacks:
if isinstance(callback, CustomLogger) and hasattr(callback, "async_post_call_streaming_deployment_hook"):
if isinstance(callback, CustomLogger) and hasattr(
callback, "async_post_call_streaming_deployment_hook"
):
result = await callback.async_post_call_streaming_deployment_hook(
request_data=request_data,
response_chunk=chunk,
@ -1560,11 +1560,14 @@ class CustomStreamWrapper:
)
if result is not None:
chunk = result
return chunk
except Exception as e:
from litellm._logging import verbose_logger
verbose_logger.exception(f"Error in post-call streaming deployment hook: {str(e)}")
verbose_logger.exception(
f"Error in post-call streaming deployment hook: {str(e)}"
)
return chunk
def cache_streaming_response(self, processed_chunk, cache_hit: bool):
@ -1687,7 +1690,7 @@ class CustomStreamWrapper:
response, "usage"
): # remove usage from chunk, only send on final chunk
# Convert the object to a dictionary
obj_dict = response.dict()
obj_dict = response.model_dump()
# Remove an attribute (e.g., 'attr2')
if "usage" in obj_dict:
@ -1852,7 +1855,7 @@ class CustomStreamWrapper:
processed_chunk, "usage"
): # remove usage from chunk, only send on final chunk
# Convert the object to a dictionary
obj_dict = processed_chunk.dict()
obj_dict = processed_chunk.model_dump()
# Remove an attribute (e.g., 'attr2')
if "usage" in obj_dict:
@ -1872,11 +1875,15 @@ class CustomStreamWrapper:
if self.sent_last_chunk is True and self.stream_options is None:
usage = calculate_total_usage(chunks=self.chunks)
processed_chunk._hidden_params["usage"] = usage
# Call post-call streaming deployment hook for final chunk
if self.sent_last_chunk is True:
processed_chunk = await self._call_post_streaming_deployment_hook(processed_chunk)
processed_chunk = (
await self._call_post_streaming_deployment_hook(
processed_chunk
)
)
return processed_chunk
raise StopAsyncIteration
else: # temporary patch for non-aiohttp async calls
@ -1890,9 +1897,9 @@ class CustomStreamWrapper:
chunk = next(self.completion_stream)
if chunk is not None and chunk != b"":
print_verbose(f"PROCESSED CHUNK PRE CHUNK CREATOR: {chunk}")
processed_chunk: Optional[ModelResponseStream] = (
self.chunk_creator(chunk=chunk)
)
processed_chunk: Optional[
ModelResponseStream
] = self.chunk_creator(chunk=chunk)
print_verbose(
f"PROCESSED CHUNK POST CHUNK CREATOR: {processed_chunk}"
)

187
litellm/llms/anthropic/chat/guardrail_translation/handler.py
View file

@ -12,10 +12,17 @@ Pattern Overview:
4. Apply guardrail responses back to the original structure
"""
import json
from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, cast
from litellm._logging import verbose_proxy_logger
from litellm.llms.anthropic.experimental_pass_through.adapters.transformation import (
LiteLLMAnthropicMessagesAdapter,
)
from litellm.llms.base_llm.guardrail_translation.base_translation import BaseTranslation
from litellm.types.guardrails import GenericGuardrailAPIInputs
from litellm.types.llms.anthropic import AllAnthropicToolsValues
from litellm.types.llms.openai import ChatCompletionToolParam
if TYPE_CHECKING:
from litellm.integrations.custom_guardrail import CustomGuardrail
@ -36,6 +43,10 @@ class AnthropicMessagesHandler(BaseTranslation):
Methods can be overridden to customize behavior for different message formats.
"""
def __init__(self):
super().__init__()
self.adapter = LiteLLMAnthropicMessagesAdapter()
async def process_input_messages(
self,
data: dict,
@ -46,11 +57,13 @@ class AnthropicMessagesHandler(BaseTranslation):
Process input messages by applying guardrails to text content.
"""
messages = data.get("messages")
tools = data.get("tools", None)
if messages is None:
return data
texts_to_check: List[str] = []
images_to_check: List[str] = []
tools_to_check: List[ChatCompletionToolParam] = []
task_mappings: List[Tuple[int, Optional[int]]] = []
# Track (message_index, content_index) for each text
# content_index is None for string content, int for list content
@ -65,18 +78,28 @@ class AnthropicMessagesHandler(BaseTranslation):
task_mappings=task_mappings,
)
if tools is not None:
self._extract_input_tools(
tools=tools,
tools_to_check=tools_to_check,
)
# Step 2: Apply guardrail to all texts in batch
if texts_to_check:
guardrailed_texts, guardrailed_images = (
await guardrail_to_apply.apply_guardrail(
texts=texts_to_check,
request_data=data,
input_type="request",
images=images_to_check if images_to_check else None,
logging_obj=litellm_logging_obj,
)
inputs = GenericGuardrailAPIInputs(texts=texts_to_check)
if images_to_check:
inputs["images"] = images_to_check
if tools_to_check:
inputs["tools"] = tools_to_check
guardrailed_inputs = await guardrail_to_apply.apply_guardrail(
inputs=inputs,
request_data=data,
input_type="request",
logging_obj=litellm_logging_obj,
)
guardrailed_texts = guardrailed_inputs.get("texts", [])
# Step 3: Map guardrail responses back to original message structure
await self._apply_guardrail_responses_to_input(
messages=messages,
@ -104,15 +127,17 @@ class AnthropicMessagesHandler(BaseTranslation):
Override this method to customize text/image extraction logic.
"""
content = message.get("content", None)
if content is None:
tools = message.get("tools", None)
if content is None and tools is None:
return
if isinstance(content, str):
## CHECK FOR TEXT + IMAGES
if content is not None and isinstance(content, str):
# Simple string content
texts_to_check.append(content)
task_mappings.append((msg_idx, None))
elif isinstance(content, list):
elif content is not None and isinstance(content, list):
# List content (e.g., multimodal with text and images)
for content_idx, content_item in enumerate(content):
# Extract text
@ -130,6 +155,22 @@ class AnthropicMessagesHandler(BaseTranslation):
if data:
images_to_check.append(data)
def _extract_input_tools(
self,
tools: List[Dict[str, Any]],
tools_to_check: List[ChatCompletionToolParam],
) -> None:
"""
Extract tools from a message.
"""
## CHECK FOR TOOLS
if tools is not None and isinstance(tools, list):
# TRANSFORM ANTHROPIC TOOLS TO OPENAI TOOLS
openai_tools = self.adapter.translate_anthropic_tools_to_openai(
tools=cast(List[AllAnthropicToolsValues], tools)
)
tools_to_check.extend(openai_tools)
async def _apply_guardrail_responses_to_input(
self,
messages: List[Dict[str, Any]],
@ -223,16 +264,18 @@ class AnthropicMessagesHandler(BaseTranslation):
if user_metadata:
request_data["litellm_metadata"] = user_metadata
guardrailed_texts, guardrailed_images = (
await guardrail_to_apply.apply_guardrail(
texts=texts_to_check,
request_data=request_data,
input_type="response",
images=images_to_check if images_to_check else None,
logging_obj=litellm_logging_obj,
)
inputs = GenericGuardrailAPIInputs(texts=texts_to_check)
if images_to_check:
inputs["images"] = images_to_check
guardrailed_inputs = await guardrail_to_apply.apply_guardrail(
inputs=inputs,
request_data=request_data,
input_type="response",
logging_obj=litellm_logging_obj,
)
guardrailed_texts = guardrailed_inputs.get("texts", [])
# Step 3: Map guardrail responses back to original response structure
await self._apply_guardrail_responses_to_output(
response=response,
@ -246,6 +289,112 @@ class AnthropicMessagesHandler(BaseTranslation):
return response
async def process_output_streaming_response(
self,
responses_so_far: List[Any],
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Optional[Any] = None,
user_api_key_dict: Optional[Any] = None,
) -> List[Any]:
"""
Process output streaming response by applying guardrails to text content.
Get the string so far, check the apply guardrail to the string so far, and return the list of responses so far.
"""
string_so_far = self.get_streaming_string_so_far(responses_so_far)
guardrailed_inputs = await guardrail_to_apply.apply_guardrail( # allow rejecting the response, if invalid
inputs={"texts": [string_so_far]},
request_data={},
input_type="response",
logging_obj=litellm_logging_obj,
)
return responses_so_far
def get_streaming_string_so_far(self, responses_so_far: List[Any]) -> str:
"""
Parse streaming responses and extract accumulated text content.
Handles two formats:
1. Raw bytes in SSE (Server-Sent Events) format from Anthropic API
2. Parsed dict objects (for backwards compatibility)
SSE format example:
b'event: content_block_delta\\ndata: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" curious"}}\\n\\n'
Dict format example:
{
"type": "content_block_delta",
"index": 0,
"delta": {
"type": "text_delta",
"text": " curious"
}
}
"""
text_so_far = ""
for response in responses_so_far:
# Handle raw bytes in SSE format
if isinstance(response, bytes):
text_so_far += self._extract_text_from_sse(response)
# Handle already-parsed dict format
elif isinstance(response, dict):
delta = response.get("delta") if response.get("delta") else None
if delta and delta.get("type") == "text_delta":
text = delta.get("text", "")
if text:
text_so_far += text
return text_so_far
def _extract_text_from_sse(self, sse_bytes: bytes) -> str:
"""
Extract text content from Server-Sent Events (SSE) format.
Args:
sse_bytes: Raw bytes in SSE format
Returns:
Accumulated text from all content_block_delta events
"""
text = ""
try:
# Decode bytes to string
sse_string = sse_bytes.decode("utf-8")
# Split by double newline to get individual events
events = sse_string.split("\n\n")
for event in events:
if not event.strip():
continue
# Parse event lines
lines = event.strip().split("\n")
event_type = None
data_line = None
for line in lines:
if line.startswith("event:"):
event_type = line[6:].strip()
elif line.startswith("data:"):
data_line = line[5:].strip()
# Only process content_block_delta events
if event_type == "content_block_delta" and data_line:
try:
data = json.loads(data_line)
delta = data.get("delta", {})
if delta.get("type") == "text_delta":
text += delta.get("text", "")
except json.JSONDecodeError:
verbose_proxy_logger.warning(
f"Failed to parse JSON from SSE data: {data_line}"
)
except Exception as e:
verbose_proxy_logger.error(f"Error extracting text from SSE: {e}")
return text
def _has_text_content(self, response: "AnthropicMessagesResponse") -> bool:
"""
Check if response has any text content to process.

8
litellm/llms/anthropic/chat/handler.py
View file

@ -436,9 +436,7 @@ class AnthropicChatCompletion(BaseLLM):
else:
if client is None or not isinstance(client, HTTPHandler):
client = _get_httpx_client(
params={"timeout": timeout}
)
client = _get_httpx_client(params={"timeout": timeout})
else:
client = client
@ -528,9 +526,7 @@ class ModelResponseIterator:
usage_object=cast(dict, anthropic_usage_chunk), reasoning_content=None
)
def _content_block_delta_helper(
self, chunk: dict
) -> Tuple[
def _content_block_delta_helper(self, chunk: dict) -> Tuple[
str,
Optional[ChatCompletionToolCallChunk],
List[Union[ChatCompletionThinkingBlock, ChatCompletionRedactedThinkingBlock]],

9
litellm/llms/anthropic/chat/transformation.py
View file

@ -786,6 +786,9 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
valid_content: bool = False
system_message_block = ChatCompletionSystemMessage(**message)
if isinstance(system_message_block["content"], str):
# Skip empty text blocks - Anthropic API raises errors for empty text
if not system_message_block["content"]:
continue
anthropic_system_message_content = AnthropicSystemMessageContent(
type="text",
text=system_message_block["content"],
@ -800,10 +803,14 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
valid_content = True
elif isinstance(message["content"], list):
for _content in message["content"]:
# Skip empty text blocks - Anthropic API raises errors for empty text
text_value = _content.get("text")
if _content.get("type") == "text" and not text_value:
continue
anthropic_system_message_content = (
AnthropicSystemMessageContent(
type=_content.get("type"),
text=_content.get("text"),
text=text_value,
)
)
if "cache_control" in _content:

2
litellm/llms/azure/azure.py
View file

@ -1020,7 +1020,7 @@ class AzureChatCompletion(BaseAzureLLM, BaseLLM):
headers: dict,
client=None,
timeout=None,
) -> litellm.ImageResponse:
) -> ImageResponse:
response: Optional[dict] = None
try:

11
litellm/llms/azure/chat/gpt_5_transformation.py
View file

@ -40,7 +40,11 @@ class AzureOpenAIGPT5Config(AzureOpenAIConfig, OpenAIGPT5Config):
or optional_params.get("reasoning_effort")
)
if reasoning_effort_value == "none":
# gpt-5.1 supports reasoning_effort='none', but other gpt-5 models don't
# See: https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/reasoning
is_gpt_5_1 = self.is_model_gpt_5_1_model(model)
if reasoning_effort_value == "none" and not is_gpt_5_1:
if litellm.drop_params is True or (
drop_params is not None and drop_params is True
):
@ -54,7 +58,7 @@ class AzureOpenAIGPT5Config(AzureOpenAIConfig, OpenAIGPT5Config):
raise UnsupportedParamsError(
status_code=400,
message=(
"Azure OpenAI does not support reasoning_effort='none'. "
"Azure OpenAI does not support reasoning_effort='none' for this model. "
"Supported values are: 'low', 'medium', and 'high'. "
"To drop this parameter, set `litellm.drop_params=True` or for proxy:\n\n"
"`litellm_settings:\n drop_params: true`\n"
@ -70,7 +74,8 @@ class AzureOpenAIGPT5Config(AzureOpenAIConfig, OpenAIGPT5Config):
drop_params=drop_params,
)
if result.get("reasoning_effort") == "none":
# Only drop reasoning_effort='none' for non-gpt-5.1 models
if result.get("reasoning_effort") == "none" and not is_gpt_5_1:
result.pop("reasoning_effort")
return result

4
litellm/llms/azure_ai/embed/handler.py
View file

@ -58,7 +58,7 @@ class AzureAIEmbedding(OpenAIChatCompletion):
data: ImageEmbeddingRequest,
timeout: float,
logging_obj,
model_response: litellm.EmbeddingResponse,
model_response: EmbeddingResponse,
optional_params: dict,
api_key: Optional[str],
api_base: Optional[str],
@ -138,7 +138,7 @@ class AzureAIEmbedding(OpenAIChatCompletion):
input: List,
timeout: float,
logging_obj,
model_response: litellm.EmbeddingResponse,
model_response: EmbeddingResponse,
optional_params: dict,
api_key: Optional[str] = None,
api_base: Optional[str] = None,

16
litellm/llms/base_llm/guardrail_translation/base_translation.py
View file

@ -1,5 +1,5 @@
from abc import ABC, abstractmethod
from typing import TYPE_CHECKING, Any, Dict, Optional
from typing import TYPE_CHECKING, Any, Dict, List, Optional
if TYPE_CHECKING:
from litellm.integrations.custom_guardrail import CustomGuardrail
@ -84,3 +84,17 @@ class BaseTranslation(ABC):
user_api_key_dict: User API key metadata (passed separately since response doesn't contain it)
"""
pass
async def process_output_streaming_response(
self,
responses_so_far: List[Any],
guardrail_to_apply: "CustomGuardrail",
litellm_logging_obj: Optional["LiteLLMLoggingObj"] = None,
user_api_key_dict: Optional["UserAPIKeyAuth"] = None,
) -> Any:
"""
Process output streaming response with guardrails.
Optional to override in subclasses.
"""
return responses_so_far

4
litellm/llms/bedrock/base_aws_llm.py
View file

@ -353,6 +353,10 @@ class BaseAWSLLM:
model_id = BaseAWSLLM._get_model_id_from_model_with_spec(
model_id, spec="deepseek_r1"
)
elif provider == "openai" and "openai/" in model_id:
model_id = BaseAWSLLM._get_model_id_from_model_with_spec(
model_id, spec="openai"
)
return model_id
@staticmethod

96
litellm/llms/bedrock/batches/handler.py Normal file
View file

@ -0,0 +1,96 @@
from openai.types.batch import BatchRequestCounts
from openai.types.batch import Metadata as OpenAIBatchMetadata
from litellm.types.utils import LiteLLMBatch
class BedrockBatchesHandler:
"""
Handler for Bedrock Batches.
Specific providers/models needed some special handling.
E.g. Twelve Labs Embedding Async Invoke
"""
@staticmethod
def _handle_async_invoke_status(
batch_id: str, aws_region_name: str, logging_obj=None, **kwargs
) -> "LiteLLMBatch":
"""
Handle async invoke status check for AWS Bedrock.
This is for Twelve Labs Embedding Async Invoke.
Args:
batch_id: The async invoke ARN
aws_region_name: AWS region name
**kwargs: Additional parameters
Returns:
dict: Status information including status, output_file_id (S3 URL), etc.
"""
import asyncio
from litellm.llms.bedrock.embed.embedding import BedrockEmbedding
async def _async_get_status():
# Create embedding handler instance
embedding_handler = BedrockEmbedding()
# Get the status of the async invoke job
status_response = await embedding_handler._get_async_invoke_status(
invocation_arn=batch_id,
aws_region_name=aws_region_name,
logging_obj=logging_obj,
**kwargs,
)
# Transform response to a LiteLLMBatch object
from litellm.types.utils import LiteLLMBatch
openai_batch_metadata: OpenAIBatchMetadata = {
"output_file_id": status_response["outputDataConfig"][
"s3OutputDataConfig"
]["s3Uri"],
"failure_message": status_response.get("failureMessage") or "",
"model_arn": status_response["modelArn"],
}
result = LiteLLMBatch(
id=status_response["invocationArn"],
object="batch",
status=status_response["status"],
created_at=status_response["submitTime"],
in_progress_at=status_response["lastModifiedTime"],
completed_at=status_response.get("endTime"),
failed_at=status_response.get("endTime")
if status_response["status"] == "failed"
else None,
request_counts=BatchRequestCounts(
total=1,
completed=1 if status_response["status"] == "completed" else 0,
failed=1 if status_response["status"] == "failed" else 0,
),
metadata=openai_batch_metadata,
completion_window="24h",
endpoint="/v1/embeddings",
input_file_id="",
)
return result
# Since this function is called from within an async context via run_in_executor,
# we need to create a new event loop in a thread to avoid conflicts
import concurrent.futures
def run_in_thread():
new_loop = asyncio.new_event_loop()
asyncio.set_event_loop(new_loop)
try:
return new_loop.run_until_complete(_async_get_status())
finally:
new_loop.close()
with concurrent.futures.ThreadPoolExecutor() as executor:
future = executor.submit(run_in_thread)
return future.result()

10
litellm/llms/bedrock/chat/converse_handler.py
View file

@ -29,6 +29,7 @@ def make_sync_call(
logging_obj: LiteLLMLoggingObject,
json_mode: Optional[bool] = False,
fake_stream: bool = False,
stream_chunk_size: int = 1024,
):
if client is None:
client = _get_httpx_client() # Create a new client if none provided
@ -66,7 +67,7 @@ def make_sync_call(
)
else:
decoder = AWSEventStreamDecoder(model=model)
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=1024))
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=stream_chunk_size))
# LOGGING
logging_obj.post_call(
@ -102,6 +103,7 @@ class BedrockConverseLLM(BaseAWSLLM):
fake_stream: bool = False,
json_mode: Optional[bool] = False,
api_key: Optional[str] = None,
stream_chunk_size: int = 1024,
) -> CustomStreamWrapper:
request_data = await litellm.AmazonConverseConfig()._async_transform_request(
model=model,
@ -143,6 +145,7 @@ class BedrockConverseLLM(BaseAWSLLM):
logging_obj=logging_obj,
fake_stream=fake_stream,
json_mode=json_mode,
stream_chunk_size=stream_chunk_size,
)
streaming_response = CustomStreamWrapper(
completion_stream=completion_stream,
@ -260,6 +263,7 @@ class BedrockConverseLLM(BaseAWSLLM):
):
## SETUP ##
stream = optional_params.pop("stream", None)
stream_chunk_size = optional_params.pop("stream_chunk_size", 1024)
unencoded_model_id = optional_params.pop("model_id", None)
fake_stream = optional_params.pop("fake_stream", False)
json_mode = optional_params.get("json_mode", False)
@ -356,7 +360,8 @@ class BedrockConverseLLM(BaseAWSLLM):
json_mode=json_mode,
fake_stream=fake_stream,
credentials=credentials,
api_key=api_key
api_key=api_key,
stream_chunk_size=stream_chunk_size,
) # type: ignore
### ASYNC COMPLETION
return self.async_completion(
@ -433,6 +438,7 @@ class BedrockConverseLLM(BaseAWSLLM):
logging_obj=logging_obj,
json_mode=json_mode,
fake_stream=fake_stream,
stream_chunk_size=stream_chunk_size,
)
streaming_response = CustomStreamWrapper(
completion_stream=completion_stream,

102
litellm/llms/bedrock/chat/converse_transformation.py
View file

@ -246,6 +246,93 @@ class AmazonConverseConfig(BaseConfig):
llm_provider="bedrock",
)
def _is_nova_lite_2_model(self, model: str) -> bool:
"""
Check if the model is a Nova Lite 2 model that supports reasoningConfig.
Nova Lite 2 models use a different reasoning configuration structure compared to
Anthropic's thinking parameter and GPT-OSS's reasoning_effort parameter.
Supported models:
- amazon.nova-2-lite-v1:0
- us.amazon.nova-2-lite-v1:0
- eu.amazon.nova-2-lite-v1:0
- apac.amazon.nova-2-lite-v1:0
Args:
model: The model identifier
Returns:
True if the model is a Nova Lite 2 model, False otherwise
Examples:
>>> config = AmazonConverseConfig()
>>> config._is_nova_lite_2_model("amazon.nova-2-lite-v1:0")
True
>>> config._is_nova_lite_2_model("us.amazon.nova-2-lite-v1:0")
True
>>> config._is_nova_lite_2_model("amazon.nova-pro-1-5-v1:0")
False
>>> config._is_nova_lite_2_model("amazon.nova-pro-v1:0")
False
"""
# Remove regional prefix if present (us., eu., apac.)
model_without_region = model
for prefix in ["us.", "eu.", "apac."]:
if model.startswith(prefix):
model_without_region = model[len(prefix) :]
break
# Check if the model is specifically Nova Lite 2
return "nova-2-lite" in model_without_region
def _transform_reasoning_effort_to_reasoning_config(
self, reasoning_effort: str
) -> dict:
"""
Transform reasoning_effort parameter to Nova 2 reasoningConfig structure.
Nova 2 models use a reasoningConfig structure in additionalModelRequestFields
that differs from both Anthropic's thinking parameter and GPT-OSS's reasoning_effort.
Args:
reasoning_effort: The reasoning effort level, must be "low" or "high"
Returns:
dict: A dictionary containing the reasoningConfig structure:
{
"reasoningConfig": {
"type": "enabled",
"maxReasoningEffort": "low" | "medium" |"high"
}
}
Raises:
BadRequestError: If reasoning_effort is not "low", "medium" or "high"
Examples:
>>> config = AmazonConverseConfig()
>>> config._transform_reasoning_effort_to_reasoning_config("high")
{'reasoningConfig': {'type': 'enabled', 'maxReasoningEffort': 'high'}}
>>> config._transform_reasoning_effort_to_reasoning_config("low")
{'reasoningConfig': {'type': 'enabled', 'maxReasoningEffort': 'low'}}
"""
valid_values = ["low", "medium", "high"]
if reasoning_effort not in valid_values:
raise litellm.exceptions.BadRequestError(
message=f"Invalid reasoning_effort value '{reasoning_effort}' for Nova 2 models. "
f"Supported values: {valid_values}",
model="amazon.nova-2-lite-v1:0",
llm_provider="bedrock_converse",
)
return {
"reasoningConfig": {
"type": "enabled",
"maxReasoningEffort": reasoning_effort,
}
}
def get_supported_openai_params(self, model: str) -> List[str]:
from litellm.utils import supports_function_calling
@ -299,6 +386,10 @@ class AmazonConverseConfig(BaseConfig):
if "gpt-oss" in model:
supported_params.append("reasoning_effort")
elif self._is_nova_lite_2_model(model):
# Nova Lite 2 models support reasoning_effort (transformed to reasoningConfig)
# These models use a different reasoning structure than Anthropic's thinking parameter
supported_params.append("reasoning_effort")
elif (
"claude-3-7" in model
or "claude-sonnet-4" in model
@ -564,6 +655,12 @@ class AmazonConverseConfig(BaseConfig):
# GPT-OSS models: keep reasoning_effort as-is
# It will be passed through to additionalModelRequestFields
optional_params["reasoning_effort"] = value
elif self._is_nova_lite_2_model(model):
# Nova Lite 2 models: transform to reasoningConfig
reasoning_config = (
self._transform_reasoning_effort_to_reasoning_config(value)
)
optional_params.update(reasoning_config)
else:
# Anthropic and other models: convert to thinking parameter
optional_params["thinking"] = AnthropicConfig._map_reasoning_effort(
@ -574,8 +671,9 @@ class AmazonConverseConfig(BaseConfig):
self._validate_request_metadata(value) # type: ignore
optional_params["requestMetadata"] = value
# Only update thinking tokens for non-GPT-OSS models
if "gpt-oss" not in model:
# Only update thinking tokens for non-GPT-OSS models and non-Nova-Lite-2 models
# Nova Lite 2 handles token budgeting differently through reasoningConfig
if "gpt-oss" not in model and not self._is_nova_lite_2_model(model):
self.update_optional_params_with_thinking_tokens(
non_default_params=non_default_params, optional_params=optional_params
)

65
litellm/llms/bedrock/chat/invoke_handler.py
View file

@ -73,6 +73,9 @@ bedrock_tool_name_mappings: InMemoryCache = InMemoryCache(
max_size_in_memory=50, default_ttl=600
)
from litellm.llms.bedrock.chat.converse_transformation import AmazonConverseConfig
from litellm.llms.bedrock.chat.invoke_transformations.amazon_openai_transformation import (
AmazonBedrockOpenAIConfig,
)
converse_config = AmazonConverseConfig()
@ -189,6 +192,7 @@ async def make_call(
fake_stream: bool = False,
json_mode: Optional[bool] = False,
bedrock_invoke_provider: Optional[litellm.BEDROCK_INVOKE_PROVIDERS_LITERAL] = None,
stream_chunk_size: int = 1024,
):
try:
if client is None:
@ -232,7 +236,7 @@ async def make_call(
json_mode=json_mode,
)
completion_stream = decoder.aiter_bytes(
response.aiter_bytes(chunk_size=1024)
response.aiter_bytes(chunk_size=stream_chunk_size)
)
elif bedrock_invoke_provider == "deepseek_r1":
decoder = AmazonDeepSeekR1StreamDecoder(
@ -240,12 +244,12 @@ async def make_call(
sync_stream=False,
)
completion_stream = decoder.aiter_bytes(
response.aiter_bytes(chunk_size=1024)
response.aiter_bytes(chunk_size=stream_chunk_size)
)
else:
decoder = AWSEventStreamDecoder(model=model)
completion_stream = decoder.aiter_bytes(
response.aiter_bytes(chunk_size=1024)
response.aiter_bytes(chunk_size=stream_chunk_size)
)
# LOGGING
@ -278,6 +282,7 @@ def make_sync_call(
fake_stream: bool = False,
json_mode: Optional[bool] = False,
bedrock_invoke_provider: Optional[litellm.BEDROCK_INVOKE_PROVIDERS_LITERAL] = None,
stream_chunk_size: int = 1024,
):
try:
if client is None:
@ -318,16 +323,16 @@ def make_sync_call(
sync_stream=True,
json_mode=json_mode,
)
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=1024))
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=stream_chunk_size))
elif bedrock_invoke_provider == "deepseek_r1":
decoder = AmazonDeepSeekR1StreamDecoder(
model=model,
sync_stream=True,
)
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=1024))
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=stream_chunk_size))
else:
decoder = AWSEventStreamDecoder(model=model)
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=1024))
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=stream_chunk_size))
# LOGGING
logging_obj.post_call(
@ -401,6 +406,10 @@ class BedrockLLM(BaseAWSLLM):
prompt = prompt_factory(
model=model, messages=messages, custom_llm_provider="bedrock"
)
elif provider == "openai":
# OpenAI uses messages directly, no prompt conversion needed
# Return empty prompt as it won't be used
prompt = ""
elif provider == "cohere":
prompt, chat_history = cohere_message_pt(messages=messages)
else:
@ -578,6 +587,30 @@ class BedrockLLM(BaseAWSLLM):
)
elif provider == "meta" or provider == "llama":
outputText = completion_response["generation"]
elif provider == "openai":
# OpenAI imported models use OpenAI Chat Completions format
if "choices" in completion_response and len(completion_response["choices"]) > 0:
choice = completion_response["choices"][0]
if "message" in choice:
outputText = choice["message"].get("content")
elif "text" in choice: # fallback for completion format
outputText = choice["text"]
# Set finish reason
if "finish_reason" in choice:
model_response.choices[0].finish_reason = map_finish_reason(
choice["finish_reason"]
)
# Set usage if available
if "usage" in completion_response:
usage = completion_response["usage"]
_usage = litellm.Usage(
prompt_tokens=usage.get("prompt_tokens", 0),
completion_tokens=usage.get("completion_tokens", 0),
total_tokens=usage.get("total_tokens", 0),
)
setattr(model_response, "usage", _usage)
elif provider == "mistral":
outputText = completion_response["outputs"][0]["text"]
model_response.choices[0].finish_reason = completion_response[
@ -698,6 +731,7 @@ class BedrockLLM(BaseAWSLLM):
## SETUP ##
stream = optional_params.pop("stream", None)
stream_chunk_size = optional_params.pop("stream_chunk_size", 1024)
provider = self.get_bedrock_invoke_provider(model)
modelId = self.get_bedrock_model_id(
@ -895,6 +929,20 @@ class BedrockLLM(BaseAWSLLM):
): # completion(top_k=3) > anthropic_config(top_k=3) <- allows for dynamic variables to be passed in
inference_params[k] = v
data = json.dumps({"prompt": prompt, **inference_params})
elif provider == "openai":
## OpenAI imported models use OpenAI Chat Completions format (messages-based)
# Use AmazonBedrockOpenAIConfig for proper OpenAI transformation
openai_config = AmazonBedrockOpenAIConfig()
supported_params = openai_config.get_supported_openai_params(model=model)
# Filter to only supported OpenAI params
filtered_params = {
k: v for k, v in inference_params.items()
if k in supported_params
}
# OpenAI uses messages format, not prompt
data = json.dumps({"messages": messages, **filtered_params})
else:
## LOGGING
logging_obj.pre_call(
@ -958,6 +1006,7 @@ class BedrockLLM(BaseAWSLLM):
headers=prepped.headers,
timeout=timeout,
client=client,
stream_chunk_size=stream_chunk_size,
) # type: ignore
### ASYNC COMPLETION
return self.async_completion(
@ -1003,7 +1052,7 @@ class BedrockLLM(BaseAWSLLM):
decoder = AWSEventStreamDecoder(model=model)
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=1024))
completion_stream = decoder.iter_bytes(response.iter_bytes(chunk_size=stream_chunk_size))
streaming_response = CustomStreamWrapper(
completion_stream=completion_stream,
model=model,
@ -1123,6 +1172,7 @@ class BedrockLLM(BaseAWSLLM):
logger_fn=None,
headers={},
client: Optional[AsyncHTTPHandler] = None,
stream_chunk_size: int = 1024,
) -> CustomStreamWrapper:
# The call is not made here; instead, we prepare the necessary objects for the stream.
@ -1138,6 +1188,7 @@ class BedrockLLM(BaseAWSLLM):
messages=messages,
logging_obj=logging_obj,
fake_stream=True if "ai21" in api_base else False,
stream_chunk_size=stream_chunk_size,
),
model=model,
custom_llm_provider="bedrock",

3
litellm/llms/bedrock/chat/invoke_transformations/amazon_nova_transformation.py
View file

@ -10,7 +10,6 @@ from typing import Any, List, Optional
import httpx
import litellm
from litellm.litellm_core_utils.litellm_logging import Logging
from litellm.types.llms.bedrock import BedrockInvokeNovaRequest
from litellm.types.llms.openai import AllMessageValues
@ -80,7 +79,7 @@ class AmazonInvokeNovaConfig(AmazonInvokeConfig, AmazonConverseConfig):
encoding: Any,
api_key: Optional[str] = None,
json_mode: Optional[bool] = None,
) -> litellm.ModelResponse:
) -> ModelResponse:
return AmazonConverseConfig.transform_response(
self,
model,

98
litellm/llms/bedrock/chat/invoke_transformations/amazon_qwen2_transformation.py Normal file
View file

@ -0,0 +1,98 @@
"""
Handles transforming requests for `bedrock/invoke/{qwen2} models`
Inherits from `AmazonQwen3Config` since Qwen2 and Qwen3 architectures are mostly similar.
The main difference is in the response format: Qwen2 uses "text" field while Qwen3 uses "generation" field.
Qwen2 + Invoke API Tutorial: https://docs.aws.amazon.com/bedrock/latest/userguide/invoke-imported-model.html
"""
from typing import Any, List, Optional
import httpx
from litellm.llms.bedrock.chat.invoke_transformations.amazon_qwen3_transformation import (
AmazonQwen3Config,
)
from litellm.llms.bedrock.chat.invoke_transformations.base_invoke_transformation import (
LiteLLMLoggingObj,
)
from litellm.types.llms.openai import AllMessageValues
from litellm.types.utils import ModelResponse
class AmazonQwen2Config(AmazonQwen3Config):
"""
Config for sending `qwen2` requests to `/bedrock/invoke/`
Inherits from AmazonQwen3Config since Qwen2 and Qwen3 architectures are mostly similar.
The main difference is in the response format: Qwen2 uses "text" field while Qwen3 uses "generation" field.
Reference: https://docs.aws.amazon.com/bedrock/latest/userguide/invoke-imported-model.html
"""
def transform_response(
self,
model: str,
raw_response: httpx.Response,
model_response: ModelResponse,
logging_obj: LiteLLMLoggingObj,
request_data: dict,
messages: List[AllMessageValues],
optional_params: dict,
litellm_params: dict,
encoding: Any,
api_key: Optional[str] = None,
json_mode: Optional[bool] = None,
) -> ModelResponse:
"""
Transform Qwen2 Bedrock response to OpenAI format
Qwen2 uses "text" field, but we also support "generation" field for compatibility.
"""
try:
if hasattr(raw_response, 'json'):
response_data = raw_response.json()
else:
response_data = raw_response
# Extract the generated text - Qwen2 uses "text" field, but also support "generation" for compatibility
generated_text = response_data.get("generation", "") or response_data.get("text", "")
# Clean up the response (remove assistant start token if present)
if generated_text.startswith("<|im_start|>assistant\n"):
generated_text = generated_text[len("<|im_start|>assistant\n"):]
if generated_text.endswith("<|im_end|>"):
generated_text = generated_text[:-len("<|im_end|>")]
# Set the content in the existing model_response structure
if hasattr(model_response, 'choices') and len(model_response.choices) > 0:
choice = model_response.choices[0]
if hasattr(choice, 'message'):
choice.message.content = generated_text
choice.finish_reason = "stop"
else:
# Handle streaming choices
choice.delta.content = generated_text
choice.finish_reason = "stop"
# Set usage information if available in response
if "usage" in response_data:
usage_data = response_data["usage"]
if hasattr(model_response, 'usage'):
model_response.usage.prompt_tokens = usage_data.get("prompt_tokens", 0)
model_response.usage.completion_tokens = usage_data.get("completion_tokens", 0)
model_response.usage.total_tokens = usage_data.get("total_tokens", 0)
return model_response
except Exception as e:
if logging_obj:
logging_obj.post_call(
input=messages,
api_key=api_key,
original_response=raw_response,
additional_args={"error": str(e)},
)
raise e

9
litellm/llms/bedrock/chat/invoke_transformations/base_invoke_transformation.py
View file

@ -258,6 +258,15 @@ class AmazonInvokeConfig(BaseConfig, BaseAWSLLM):
litellm_params=litellm_params,
headers=headers,
)
elif provider == "openai":
# OpenAI imported models use OpenAI Chat Completions format
return litellm.AmazonBedrockOpenAIConfig().transform_request(
model=model,
messages=messages,
optional_params=optional_params,
litellm_params=litellm_params,
headers=headers,
)
else:
raise BedrockError(
status_code=404,

21
litellm/llms/bedrock/common_utils.py
View file

@ -27,6 +27,25 @@ class BedrockError(BaseLLMException):
pass
# Lazy import cache to avoid circular imports and performance impact
_get_model_info = None
def get_cached_model_info():
"""
Lazy import and cache get_model_info to avoid circular imports.
This function is used by bedrock transformation classes that need get_model_info
but cannot import it at module level due to circular import issues.
The function is cached after first use to avoid performance impact.
"""
global _get_model_info
if _get_model_info is None:
from litellm import get_model_info
_get_model_info = get_model_info
return _get_model_info
class AmazonBedrockGlobalConfig:
def __init__(self):
pass
@ -616,6 +635,8 @@ def get_bedrock_chat_config(model: str):
return litellm.AmazonInvokeNovaConfig()
elif bedrock_invoke_provider == "qwen3":
return litellm.AmazonQwen3Config()
elif bedrock_invoke_provider == "qwen2":
return litellm.AmazonQwen2Config()
elif bedrock_invoke_provider == "twelvelabs":
return litellm.AmazonTwelveLabsPegasusConfig()
else:

206
litellm/llms/bedrock/files/handler.py Normal file
View file

@ -0,0 +1,206 @@
import asyncio
import base64
from typing import Any, Coroutine, Optional, Tuple, Union
import httpx
from litellm import LlmProviders
from litellm.llms.custom_httpx.http_handler import get_async_httpx_client
from litellm.types.llms.openai import (
FileContentRequest,
HttpxBinaryResponseContent,
)
from litellm.types.utils import SpecialEnums
from ..base_aws_llm import BaseAWSLLM
class BedrockFilesHandler(BaseAWSLLM):
"""
Handles downloading files from S3 for Bedrock batch processing.
This implementation downloads files from S3 buckets where Bedrock
stores batch output files.
"""
def __init__(self):
super().__init__()
self.async_httpx_client = get_async_httpx_client(
llm_provider=LlmProviders.BEDROCK,
)
def _extract_s3_uri_from_file_id(self, file_id: str) -> str:
"""
Extract S3 URI from encoded file ID.
The file ID can be in two formats:
1. Base64-encoded unified file ID containing: llm_output_file_id,s3://bucket/path
2. Direct S3 URI: s3://bucket/path
Args:
file_id: Encoded file ID or direct S3 URI
Returns:
S3 URI (e.g., "s3://bucket-name/path/to/file")
"""
# First, try to decode if it's a base64-encoded unified file ID
try:
# Add padding if needed
padded = file_id + "=" * (-len(file_id) % 4)
decoded = base64.urlsafe_b64decode(padded).decode()
# Check if it's a unified file ID format
if decoded.startswith(SpecialEnums.LITELM_MANAGED_FILE_ID_PREFIX.value):
# Extract llm_output_file_id from the decoded string
if "llm_output_file_id," in decoded:
s3_uri = decoded.split("llm_output_file_id,")[1].split(";")[0]
return s3_uri
except Exception:
pass
# If not base64 encoded or doesn't contain llm_output_file_id, assume it's already an S3 URI
if file_id.startswith("s3://"):
return file_id
# If it doesn't start with s3://, assume it's a direct S3 URI and add the prefix
return f"s3://{file_id}"
def _parse_s3_uri(self, s3_uri: str) -> Tuple[str, str]:
"""
Parse S3 URI to extract bucket name and object key.
Args:
s3_uri: S3 URI (e.g., "s3://bucket-name/path/to/file")
Returns:
Tuple of (bucket_name, object_key)
"""
if not s3_uri.startswith("s3://"):
raise ValueError(f"Invalid S3 URI format: {s3_uri}. Expected format: s3://bucket-name/path/to/file")
# Remove 's3://' prefix
path = s3_uri[5:]
if "/" in path:
bucket_name, object_key = path.split("/", 1)
else:
bucket_name = path
object_key = ""
return bucket_name, object_key
async def afile_content(
self,
file_content_request: FileContentRequest,
optional_params: dict,
timeout: Union[float, httpx.Timeout],
max_retries: Optional[int],
) -> HttpxBinaryResponseContent:
"""
Download file content from S3 bucket for Bedrock files.
Args:
file_content_request: Contains file_id (encoded or S3 URI)
optional_params: Optional parameters containing AWS credentials
timeout: Request timeout
max_retries: Max retry attempts
Returns:
HttpxBinaryResponseContent: Binary content wrapped in compatible response format
"""
import boto3
from botocore.credentials import Credentials
file_id = file_content_request.get("file_id")
if not file_id:
raise ValueError("file_id is required in file_content_request")
# Extract S3 URI from file ID
s3_uri = self._extract_s3_uri_from_file_id(file_id)
bucket_name, object_key = self._parse_s3_uri(s3_uri)
# Get AWS credentials
aws_region_name = self._get_aws_region_name(
optional_params=optional_params, model=""
)
credentials: Credentials = self.get_credentials(
aws_access_key_id=optional_params.get("aws_access_key_id"),
aws_secret_access_key=optional_params.get("aws_secret_access_key"),
aws_session_token=optional_params.get("aws_session_token"),
aws_region_name=aws_region_name,
aws_session_name=optional_params.get("aws_session_name"),
aws_profile_name=optional_params.get("aws_profile_name"),
aws_role_name=optional_params.get("aws_role_name"),
aws_web_identity_token=optional_params.get("aws_web_identity_token"),
aws_sts_endpoint=optional_params.get("aws_sts_endpoint"),
)
# Create S3 client
s3_client = boto3.client(
"s3",
aws_access_key_id=credentials.access_key,
aws_secret_access_key=credentials.secret_key,
aws_session_token=credentials.token,
region_name=aws_region_name,
)
# Download file from S3
try:
response = s3_client.get_object(Bucket=bucket_name, Key=object_key)
file_content = response["Body"].read()
except Exception as e:
raise ValueError(f"Failed to download file from S3: {s3_uri}. Error: {str(e)}")
# Create mock HTTP response
mock_response = httpx.Response(
status_code=200,
content=file_content,
headers={"content-type": "application/octet-stream"},
request=httpx.Request(method="GET", url=s3_uri),
)
return HttpxBinaryResponseContent(response=mock_response)
def file_content(
self,
_is_async: bool,
file_content_request: FileContentRequest,
api_base: Optional[str],
optional_params: dict,
timeout: Union[float, httpx.Timeout],
max_retries: Optional[int],
) -> Union[
HttpxBinaryResponseContent, Coroutine[Any, Any, HttpxBinaryResponseContent]
]:
"""
Download file content from S3 bucket for Bedrock files.
Supports both sync and async operations.
Args:
_is_async: Whether to run asynchronously
file_content_request: Contains file_id (encoded or S3 URI)
api_base: API base (unused for S3 operations)
optional_params: Optional parameters containing AWS credentials
timeout: Request timeout
max_retries: Max retry attempts
Returns:
HttpxBinaryResponseContent or Coroutine: Binary content wrapped in compatible response format
"""
if _is_async:
return self.afile_content(
file_content_request=file_content_request,
optional_params=optional_params,
timeout=timeout,
max_retries=max_retries,
)
else:
return asyncio.run(
self.afile_content(
file_content_request=file_content_request,
optional_params=optional_params,
timeout=timeout,
max_retries=max_retries,
)
)

Some files were not shown because too many files have changed in this diff Show more