scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-12 08:42:18 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 207

Merge branch 'BerriAI:main' into main

Browse source
This commit is contained in:
Valteru Gowtham 2026-10-10 18:54:30 +05:30 • committed by GitHub
commit afa895c195
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1519 changed files with 64963 additions and 136347 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

124
.circleci/config.yml
View file

@ -3378,38 +3378,14 @@ workflows:
- not:
equal: ["", << pipeline.parameters.routing_parity_base >>]
jobs:
- using_litellm_on_windows:
filters:
branches:
only: main
- windows_release_wheel:
filters:
branches:
only: main
- provider_replay_harness:
filters:
branches:
only: main
- base_sdk_install:
filters:
branches:
only: main
- local_testing_part1:
filters:
branches:
only: main
- local_testing_part2:
filters:
branches:
only: main
- langfuse_logging_unit_tests:
filters:
branches:
only: main
- litellm_router_testing:
filters:
branches:
only: main
- using_litellm_on_windows
- windows_release_wheel
- provider_replay_harness
- base_sdk_install
- local_testing_part1
- local_testing_part2
- langfuse_logging_unit_tests
- litellm_router_testing
- auth_ui_unit_tests
- build_docker_database_image
- e2e_ui_testing
@ -3429,65 +3405,26 @@ workflows:
- proxy_store_model_in_db_tests:
requires:
- build_docker_database_image
- proxy_build_from_pip_tests:
filters:
branches:
only: main
- proxy_build_from_pip_tests
- proxy_pass_through_endpoint_tests:
requires:
- build_docker_database_image
- proxy_e2e_anthropic_messages_tests:
requires:
- build_docker_database_image
- llm_translation_testing:
filters:
branches:
only: main
- realtime_translation_testing:
filters:
branches:
only: main
- llm_translation_testing
- realtime_translation_testing
- guardrails_testing
- google_generate_content_endpoint_testing:
filters:
branches:
only: main
- ocr_testing:
filters:
branches:
only: main
- search_testing:
filters:
branches:
only: main
- batches_testing:
filters:
branches:
only: main
- litellm_utils_testing:
filters:
branches:
only: main
- pass_through_unit_testing:
filters:
branches:
only: main
- image_gen_testing:
filters:
branches:
only: main
- logging_testing:
filters:
branches:
only: main
- audio_testing:
filters:
branches:
only: main
- google_generate_content_endpoint_testing
- ocr_testing
- search_testing
- batches_testing
- litellm_utils_testing
- pass_through_unit_testing
- image_gen_testing
- logging_testing
- audio_testing
- upload-coverage:
filters:
branches:
only: main
requires:
- realtime_translation_testing
- google_generate_content_endpoint_testing
@ -3505,27 +3442,12 @@ workflows:
- db_migration_disable_update_check:
requires:
- build_docker_database_image
- installing_litellm_on_python:
filters:
branches:
only: main
- installing_litellm_on_python_3_13:
filters:
branches:
only: main
- installing_litellm_on_python_v2_migration_resolver:
filters:
branches:
only: main
- installing_litellm_on_python
- installing_litellm_on_python_3_13
- installing_litellm_on_python_v2_migration_resolver
- helm_chart_testing:
filters:
branches:
only: main
requires:
- build_docker_database_image
- test_bad_database_url:
filters:
branches:
only: main
requires:
- build_docker_database_image

8
.github/ci-coverage-allowlist.yml vendored
View file

@ -34,6 +34,14 @@ test_paths:
- tests/local_testing/test_caching.py
- tests/local_testing/test_disk_cache_unit_tests.py
- tests/local_testing/test_gcs_cache_unit_tests.py
- reason: >-
Mixed file that no job has ever run in full. Every CircleCI job that globs
tests/local_testing deselects it by name ("caching") or keeps only another keyword, so only
its router test ran, and that test now lives in tests/unit/router_utils/pre_call_checks.
Its two offline tests now live in tests/unit/llms/anthropic/chat, and the five left all need
ANTHROPIC_API_KEY or Vertex credentials
paths:
- tests/local_testing/test_anthropic_prompt_caching.py
- reason: >-
The end-to-end suite runs against a deployed proxy from its own in-cluster rig rather than
from a pull request; it needs a live gateway and provider credentials no PR job holds

60
.github/workflows/helm_unit_test.yml vendored
View file

@ -1,60 +0,0 @@
name: Helm unit test
on:
pull_request:
push:
branches:
- main
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
unit-test:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
- name: Check Lens Compose configuration
run: |
python3 -m unittest discover -s deploy/lens -p 'test_*.py'
python3 deploy/lens/configure.py --version 1.2.3 --env-file "$RUNNER_TEMP/lens.env"
docker compose --env-file "$RUNNER_TEMP/lens.env" -f deploy/lens/stack.yaml config --quiet
- name: Set up Helm 3.11.1
uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4 # v4.3.1
with:
version: "3.11.1"
- name: Download and verify Helm Unit Test Plugin
run: |
curl -fsSLo "$RUNNER_TEMP/helm-unittest.tgz" https://github.com/helm-unittest/helm-unittest/releases/download/v0.8.2/helm-unittest-linux-amd64-0.8.2.tgz
echo "56ab3091e6fa52a7c92ee951def9bed957f295d9ce98483aed404e748d7b3a94 $RUNNER_TEMP/helm-unittest.tgz" | sha256sum -c -
- name: Install Helm Unit Test Plugin
run: |
PLUGIN_DIR="$(helm env HELM_PLUGINS)/helm-unittest"
mkdir -p "$PLUGIN_DIR"
tar -xzf "$RUNNER_TEMP/helm-unittest.tgz" -C "$PLUGIN_DIR"
helm plugin list
- name: Run unit tests
run: |
for chart in helm/litellm-helm helm/litellm; do
declared="$(grep -h '^suite:' "$chart"/tests/*.yaml | wc -l | tr -d '[:space:]')"
output="$(mktemp)"
helm unittest -f 'tests/*.yaml' "$chart" | tee "$output"
executed="$(sed -n 's/^Test Suites:.*[[:space:]]\([0-9][0-9]*\) total$/\1/p' "$output")"
if [ "$declared" != "$executed" ]; then
echo "::error::$chart declares $declared test suites but helm-unittest ran $executed. Suites are being skipped silently, so their assertions never execute."
exit 1
fi
echo "$chart: all $declared declared test suites ran"
done

58
.github/workflows/image-scan.yml vendored
View file

@ -17,10 +17,8 @@ on:
- gateway/routes/allowlist.py
- backend/Dockerfile
- backend/main.py
- deploy/lens/**
- litellm-rust/**
- litellm/proxy/lens/**
- tests/e2e/migrations/lens_compose_smoke.sh
- docker/component_entrypoint.sh
- docker/entrypoint.sh
- litellm/proxy/prisma_migration.py
@ -47,57 +45,6 @@ concurrency:
cancel-in-progress: true
jobs:
lens-worker-image:
name: lens-worker-image (${{ matrix.arch }})
runs-on: ${{ matrix.runner }}
if: >-
github.event_name != 'pull_request' ||
github.event.pull_request.head.repo.full_name == github.repository
timeout-minutes: 45
permissions:
contents: read
strategy:
fail-fast: false
matrix:
include:
- arch: amd64
runner: ubuntu-latest
grype_sha256: edda0968d8827daab01d32b3cd7de192ae0915005e7bbfcfef9e68e79bc43343
- arch: arm64
runner: ubuntu-24.04-arm
grype_sha256: 553e4c36d9d61349830ba6034d43b8700a7f10576d3e2f4981c0fd2b96086465
steps:
- uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
- name: Build the release worker
env:
RELEASE_TAG: sha-${{ github.sha }}
run: docker build --build-arg LITELLM_RELEASE_TAG="${RELEASE_TAG}" -f deploy/lens/Dockerfile -t lens-worker-scan .
- name: Verify the standalone worker on a read-only filesystem
env:
RELEASE_TAG: sha-${{ github.sha }}
run: |
docker run --rm --network none --read-only --cap-drop ALL \
--tmpfs /tmp:rw,noexec,nosuid,size=1g --security-opt no-new-privileges \
lens-worker-scan --version | grep -F "litellm-lens $RELEASE_TAG protocol="
- name: Download Grype v0.114.0
env:
ARCH: ${{ matrix.arch }}
GRYPE_SHA256: ${{ matrix.grype_sha256 }}
run: |
curl -fsSL --retry 3 -o "$RUNNER_TEMP/grype.tar.gz" \
"https://github.com/anchore/grype/releases/download/v0.114.0/grype_0.114.0_linux_${ARCH}.tar.gz"
echo "${GRYPE_SHA256} $RUNNER_TEMP/grype.tar.gz" | sha256sum -c -
tar xzf "$RUNNER_TEMP/grype.tar.gz" -C "$RUNNER_TEMP" grype
chmod +x "$RUNNER_TEMP/grype"
- name: Scan the worker for fixable HIGH/CRITICAL CVEs
env:
GRYPE_MATCH_PYTHON_USING_CPES: "true"
run: |
"$RUNNER_TEMP/grype" lens-worker-scan \
--config .grype.yaml --only-fixed --fail-on high --output table
image-scan:
name: image-scan
runs-on: ubuntu-latest
@ -200,11 +147,6 @@ jobs:
python -m pip install "pytest==9.0.3"
python -m pytest tests/proxy_migration_tests/test_offline_image_migration.py tests/proxy_migration_tests/test_image_bedrock_realtime_extra.py tests/proxy_migration_tests/test_image_admin_mcp.py -v
- name: Verify the bundled Lens Compose installation and restart
if: matrix.dockerfile == 'Dockerfile'
env:
LITELLM_IMAGE: litellm-runtime-scan:${{ github.sha }}
run: bash tests/e2e/migrations/lens_compose_smoke.sh
migrations-image:
name: migrations-image

78
.github/workflows/lens-install-smoke.yml vendored
View file

@ -2,6 +2,19 @@ name: Lens installation smoke
on:
workflow_dispatch:
inputs:
lens_image:
description: Verified baseline Lens image at ghcr.io/berriai/lens@sha256
required: true
type: string
lens_upgrade_image:
description: Compatible Lens upgrade image at ghcr.io/berriai/lens@sha256
required: true
type: string
gateway_baseline_ref:
description: Previously qualified gateway source commit, full SHA
required: true
type: string
permissions:
contents: read
@ -28,27 +41,46 @@ jobs:
dockerfile: migrations/Dockerfile
- component: monolith
dockerfile: Dockerfile
- component: worker
dockerfile: deploy/lens/Dockerfile
- component: gateway
dockerfile: gateway/Dockerfile
baseline: true
- component: backend
dockerfile: backend/Dockerfile
baseline: true
- component: monolith
dockerfile: Dockerfile
baseline: true
env:
COMPONENT: ${{ matrix.component }}
DOCKERFILE: ${{ matrix.dockerfile }}
SOURCE_REF: ${{ matrix.baseline && inputs.gateway_baseline_ref || github.sha }}
IMAGE_TAG: ${{ matrix.baseline && 'v0.0.0-lens-ci-baseline' || 'v0.0.0-lens-ci' }}
CURRENT_REF: ${{ github.sha }}
steps:
- uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
ref: ${{ env.SOURCE_REF }}
- name: Verify the selected source
shell: bash -euo pipefail {0}
run: |
[[ "$SOURCE_REF" =~ ^[0-9a-f]{40}$ ]]
test "$(git rev-parse HEAD)" = "$SOURCE_REF"
if [[ "$IMAGE_TAG" == v0.0.0-lens-ci-baseline ]]; then
test "$SOURCE_REF" != "$CURRENT_REF"
fi
- name: Build the matching release image
run: |
docker build --build-arg LITELLM_RELEASE_TAG=v0.0.0-lens-ci \
-f "$DOCKERFILE" -t "lens-ci-$COMPONENT:v0.0.0-lens-ci" .
docker build --build-arg "LITELLM_RELEASE_TAG=$IMAGE_TAG" \
-f "$DOCKERFILE" -t "lens-ci-$COMPONENT:$IMAGE_TAG" .
- name: Save the matching release image
run: |
docker save "lens-ci-$COMPONENT:v0.0.0-lens-ci" \
| gzip -1 > "$RUNNER_TEMP/lens-install-$COMPONENT.tar.gz"
docker save "lens-ci-$COMPONENT:$IMAGE_TAG" \
| gzip -1 > "$RUNNER_TEMP/lens-install-$COMPONENT-$IMAGE_TAG.tar.gz"
- uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
with:
name: lens-install-${{ matrix.component }}-${{ github.sha }}
path: ${{ runner.temp }}/lens-install-${{ matrix.component }}.tar.gz
name: lens-install-${{ matrix.component }}-${{ env.IMAGE_TAG }}-${{ github.sha }}
path: ${{ runner.temp }}/lens-install-${{ matrix.component }}-${{ env.IMAGE_TAG }}.tar.gz
compression-level: 0
retention-days: 3
if-no-files-found: error
@ -69,11 +101,37 @@ jobs:
path: ${{ runner.temp }}/lens-install-images
- name: Load the matching release images
run: |
for component in gateway backend ui migrations monolith worker; do
archive="$RUNNER_TEMP/lens-install-images/lens-install-$component.tar.gz"
for archive in "$RUNNER_TEMP"/lens-install-images/lens-install-*.tar.gz; do
gzip -dc "$archive" | docker load
rm "$archive"
done
- uses: sigstore/cosign-installer@3454372f43399081ed03b604cb2d021dabca52bb # v3.8.2
with:
cosign-release: v2.4.3
- uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
registry: ghcr.io
username: ${{ vars.LENS_READ_USER || github.actor }}
password: ${{ secrets.LENS_READ_TOKEN }}
- name: Verify independent Lens images before loading them
env:
LENS_IMAGE: ${{ inputs.lens_image }}
LENS_UPGRADE_IMAGE: ${{ inputs.lens_upgrade_image }}
LENS_PUBLIC_KEY: ${{ vars.LENS_COSIGN_PUBLIC_KEY }}
LENS_PUBLIC_KEY_SHA256: ${{ vars.LENS_COSIGN_PUBLIC_KEY_SHA256 }}
shell: bash -euo pipefail {0}
run: |
[[ "$LENS_PUBLIC_KEY_SHA256" =~ ^[0-9a-f]{64}$ ]]
printf '%s' "$LENS_PUBLIC_KEY" > "$RUNNER_TEMP/lens.pub"
echo "$LENS_PUBLIC_KEY_SHA256 $RUNNER_TEMP/lens.pub" | sha256sum --check
for image in "$LENS_IMAGE" "$LENS_UPGRADE_IMAGE"; do
[[ "$image" =~ ^ghcr.io/berriai/lens@sha256:[0-9a-f]{64}$ ]]
cosign verify --key "$RUNNER_TEMP/lens.pub" "$image" > /dev/null
cosign verify-attestation --key "$RUNNER_TEMP/lens.pub" --type spdxjson "$image" > /dev/null
docker pull "$image"
done
docker tag "$LENS_IMAGE" lens-ci-worker:baseline
docker tag "$LENS_UPGRADE_IMAGE" lens-ci-worker:upgrade
- name: Install pinned Kubernetes test tools
run: |
curl --fail --location --output "$RUNNER_TEMP/kind" \

111
.github/workflows/lens-worker.yml vendored
View file

@ -1,111 +0,0 @@
name: Lens Worker Image
on:
pull_request:
branches: [main, litellm_oss_branch, "litellm_**"]
paths:
- deploy/lens/**
- litellm-rust/**
- litellm/proxy/lens/**
- tests/proxy_behavior/lens/**
- .github/workflows/lens-worker.yml
push:
branches: [main]
paths:
- deploy/lens/**
- litellm-rust/**
- litellm/proxy/lens/**
- tests/proxy_behavior/lens/**
- .github/workflows/lens-worker.yml
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
lens-worker-image:
permissions:
contents: read
packages: write
runs-on: ${{ matrix.runner }}
timeout-minutes: 45
strategy:
fail-fast: false
matrix:
include:
- arch: amd64
runner: ubuntu-latest
- arch: arm64
runner: ubuntu-24.04-arm
steps:
- uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
- name: Build native Lens service
env:
RELEASE_TAG: sha-${{ github.sha }}
run: docker build --build-arg LITELLM_RELEASE_TAG="$RELEASE_TAG" -f deploy/lens/Dockerfile -t lens-worker .
- name: Verify version and unprivileged runtime
env:
RELEASE_TAG: sha-${{ github.sha }}
run: bash deploy/lens/smoke.sh lens-worker "$RELEASE_TAG"
- name: Verify confined Python on the native architecture
env:
RELEASE_TAG: sha-${{ github.sha }}
run: |
docker build --target smoke --build-arg LITELLM_RELEASE_TAG="$RELEASE_TAG" -f deploy/lens/Dockerfile -t lens-smoke .
docker run --rm --network none --read-only --cap-drop ALL \
--tmpfs /tmp:rw,noexec,nosuid,size=1g --security-opt no-new-privileges lens-smoke
- name: Reject custom builds without a matching release tag
run: |
if docker build --progress plain -f deploy/lens/Dockerfile -t lens-worker:unversioned . > missing-tag.log 2>&1; then
echo "::error::An unversioned worker build unexpectedly succeeded"
exit 1
fi
grep -F 'LITELLM_RELEASE_TAG: Pass --build-arg LITELLM_RELEASE_TAG matching the gateway' missing-tag.log
- name: Publish development architecture
if: github.event_name != 'pull_request' && github.repository == 'BerriAI/litellm' && github.ref == 'refs/heads/main'
env:
REGISTRY_TOKEN: ${{ secrets.GITHUB_TOKEN }}
REGISTRY_USER: ${{ github.actor }}
IMAGE: ghcr.io/berriai/litellm-lens-worker-dev:sha-${{ github.sha }}-${{ matrix.arch }}
ARCH: ${{ matrix.arch }}
run: |
printf '%s' "$REGISTRY_TOKEN" | docker login ghcr.io -u "$REGISTRY_USER" --password-stdin
docker tag lens-worker "$IMAGE"
docker push "$IMAGE"
mkdir -p digests
docker inspect --format='{{index .RepoDigests 0}}' "$IMAGE" > "digests/$ARCH"
- uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
if: github.event_name != 'pull_request' && github.repository == 'BerriAI/litellm' && github.ref == 'refs/heads/main'
with:
name: lens-digest-${{ matrix.arch }}
path: digests/
retention-days: 1
publish:
name: Publish Lens development index
needs: lens-worker-image
if: github.event_name != 'pull_request' && github.repository == 'BerriAI/litellm' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
permissions:
packages: write
steps:
- uses: actions/download-artifact@95815c38cf2ff2164869cbab79da8d1f422bc89e # v4.2.1
with:
pattern: lens-digest-*
merge-multiple: true
path: digests
- name: Publish both tested architectures
env:
REGISTRY_TOKEN: ${{ secrets.GITHUB_TOKEN }}
REGISTRY_USER: ${{ github.actor }}
IMAGE: ghcr.io/berriai/litellm-lens-worker-dev:sha-${{ github.sha }}
run: |
printf '%s' "$REGISTRY_TOKEN" | docker login ghcr.io -u "$REGISTRY_USER" --password-stdin
docker buildx imagetools create --tag "$IMAGE" "$(cat digests/amd64)" "$(cat digests/arm64)"
printf 'Lens worker image: `%s`\n' "$IMAGE" >> "$GITHUB_STEP_SUMMARY"

397
.github/workflows/required-checks-legacy.yml vendored
View file

@ -1,397 +0,0 @@
name: Required checks (legacy)
on:
pull_request:
branches:
- main
- "litellm_**"
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
cancel-in-progress: true
jobs:
lint:
name: lint
permissions:
contents: read
pull-requests: read
actions: read
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
ref: ${{ github.event.pull_request.head.sha }}
fetch-depth: 1
clean: true
persist-credentials: false
- name: Detect relevant changes
id: changes
uses: ./.github/actions/detect-changes
- name: Fetch gate base (merge-base with target branch)
if: steps.changes.outputs.decision != 'skip'
env:
GH_TOKEN: ${{ github.token }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: |
retry() { "$@" || { sleep 15; "$@"; } || { sleep 30; "$@"; }; }
MERGE_BASE=$(retry gh api "repos/${{ github.repository }}/compare/${BASE_SHA}...${HEAD_SHA}?per_page=1" --jq '.merge_base_commit.sha')
test -n "$MERGE_BASE"
retry git fetch --no-tags --depth=1 origin "$MERGE_BASE"
echo "GATE_BASE_SHA=$MERGE_BASE" >> "$GITHUB_ENV"
- name: Set up Python
if: steps.changes.outputs.decision != 'skip'
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"
- name: Set up uv
if: steps.changes.outputs.decision != 'skip'
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"
- name: Cache uv dependencies
if: steps.changes.outputs.decision != 'skip'
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: |
~/.cache/uv
.venv
key: ${{ runner.os }}-uv-lint-${{ hashFiles('uv.lock') }}
restore-keys: |
${{ runner.os }}-uv-lint-
- name: Clean Python cache
if: steps.changes.outputs.decision != 'skip'
run: |
find . -type d -name "__pycache__" -exec rm -rf {} + || true
find . -name "*.pyc" -delete || true
- name: Check uv.lock is up to date
if: steps.changes.outputs.decision != 'skip'
run: |
uv lock --check || (echo "❌ uv.lock is out of sync with pyproject.toml. Run 'uv lock' locally and commit the result." && exit 1)
- name: Cache the Rust build
if: steps.changes.outputs.decision != 'skip'
uses: ./.github/actions/cache-cargo-build
- name: Install dependencies
if: steps.changes.outputs.decision != 'skip'
run: |
uv sync --frozen --group proxy-dev --group e2e-dev
- name: Cache Prisma binaries
if: steps.changes.outputs.decision != 'skip'
uses: ./.github/actions/cache-prisma-binaries
- name: Generate Prisma client
if: steps.changes.outputs.decision != 'skip'
run: |
uv run --no-sync prisma generate --schema litellm/proxy/schema.prisma
- name: Check ruff format
if: steps.changes.outputs.decision != 'skip'
run: |
git diff --name-only --diff-filter=ACMR "$GATE_BASE_SHA" HEAD -- ':(glob)litellm/**/*.py' | grep -v '^litellm/enterprise/' > "$RUNNER_TEMP/ruff_format_files.txt" || true
if [ ! -s "$RUNNER_TEMP/ruff_format_files.txt" ]; then
echo "No changed litellm Python files to check with ruff format."
exit 0
fi
xargs uv run --no-sync ruff format --check --exclude '/enterprise/' < "$RUNNER_TEMP/ruff_format_files.txt"
- name: Debug - Check file state
if: steps.changes.outputs.decision != 'skip'
run: |
echo "Current branch:"
git branch --show-current
echo "Last 3 commits:"
git log --oneline -3
echo "File content around line 43:"
head -50 litellm/litellm_core_utils/custom_logger_registry.py | tail -10
- name: Check MCP operation boundary
if: steps.changes.outputs.decision != 'skip'
run: uv run --no-sync python scripts/check_mcp_operation_boundary.py
- name: Run Ruff linting
if: steps.changes.outputs.decision != 'skip'
run: |
cd litellm
uv run --no-sync ruff check .
cd ..
- name: Run Ruff linting (test tree)
if: steps.changes.outputs.decision != 'skip'
run: |
uv run --no-sync ruff check --config ruff-tests.toml tests
- name: Check strict ruff rules (delta vs merge-base counts)
if: steps.changes.outputs.decision != 'skip'
env:
GH_TOKEN: ${{ github.token }}
run: |
uv run --no-sync python scripts/ruff_strict_gate.py --base "$GATE_BASE_SHA"
- name: Check type discipline (mutable collections / casts / type guards / kwargs / unexplained suppressions, delta vs merge-base counts)
if: steps.changes.outputs.decision != 'skip'
env:
GH_TOKEN: ${{ github.token }}
run: |
uv run --no-sync python scripts/type_discipline_gate.py --base "$GATE_BASE_SHA"
- name: Check test quality (zero-assert / mock-echo tests, sys.path.insert, raw env writes, litellm global mutation, credential-gated skips, conftest snapshot inventory, delta vs merge-base counts)
if: steps.changes.outputs.decision != 'skip'
env:
GH_TOKEN: ${{ github.token }}
run: |
uv run --no-sync python scripts/test_quality_gate.py --base "$GATE_BASE_SHA"
- name: Print OpenAI version
if: steps.changes.outputs.decision != 'skip'
run: |
uv run --no-sync python -c "import openai; print(f'OpenAI version: {openai.__version__}')"
- name: Check basedpyright (delta vs merge-base counts)
if: steps.changes.outputs.decision != 'skip'
env:
GH_TOKEN: ${{ github.token }}
run: |
uv run --no-sync python scripts/type_check_gate.py --base "$GATE_BASE_SHA"
- name: Check tests/e2e basedpyright (zero errors)
if: steps.changes.outputs.decision != 'skip'
run: |
if git diff --name-only --diff-filter=ACMRD "$GATE_BASE_SHA" HEAD -- ':(glob)tests/e2e/**/*.py' ':(glob)tests/e2e_harness/**/*.py' pyrightconfig.json | grep -q .; then
uv run --no-sync basedpyright tests/e2e tests/e2e_harness
else
echo "No changed tests/e2e Python files; skipping."
fi
- name: Run the e2e harness tests
if: steps.changes.outputs.decision != 'skip'
env:
LITELLM_MASTER_KEY: sk-e2e-harness-tests-reach-no-proxy
run: |
if ! git diff --name-only --diff-filter=ACMRD "$GATE_BASE_SHA" HEAD -- tests/e2e tests/e2e_harness ':(exclude)tests/e2e/ui' pyproject.toml uv.lock .github/workflows/test-linting.yml | grep -q .; then
echo "No changed e2e harness files; skipping."
exit 0
fi
retry() { "$@" || { sleep 15; "$@"; } || { sleep 30; "$@"; }; }
CLAUDE_VERSION="$(retry uv run --no-sync python tests/e2e/claude_code/pr_gate_version_resolver.py)"
tests/e2e/claude_code/cron_vm/install_claude_code.sh "$CLAUDE_VERSION" "$RUNNER_TEMP/claude-cli"
PATH="$RUNNER_TEMP/claude-cli:$PATH" uv run --no-sync pytest -q tests/e2e_harness
- name: Check for circular imports
if: steps.changes.outputs.decision != 'skip'
run: |
cd litellm
uv run --no-sync python ../tests/documentation_tests/test_circular_imports.py
cd ..
- name: Check import safety
if: steps.changes.outputs.decision != 'skip'
run: |
uv run --no-sync python -c "from litellm import *" || (echo '🚨 import failed, this means you introduced unprotected imports! 🚨'; exit 1)
frontend-lint:
name: frontend-lint
permissions:
contents: read
runs-on: ubuntu-latest
timeout-minutes: 8
defaults:
run:
working-directory: ui/litellm-dashboard
steps:
- name: Checkout repository
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
fetch-depth: 1
persist-credentials: false
- name: Collect changed files
id: changed
env:
GH_TOKEN: ${{ github.token }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: |
merge_base=$(gh api "repos/${{ github.repository }}/compare/${BASE_SHA}...${HEAD_SHA}?per_page=1" --jq '.merge_base_commit.sha')
test -n "$merge_base"
git fetch --no-tags --depth=1 origin "$merge_base" "$HEAD_SHA"
: > "$RUNNER_TEMP/prettier_files.txt"
: > "$RUNNER_TEMP/eslint_files.txt"
while IFS= read -r f; do
[ -f "$f" ] || continue
case "$f" in
*.js | *.jsx | *.ts | *.tsx | *.mjs | *.cjs)
printf '%s\n' "$f" >> "$RUNNER_TEMP/prettier_files.txt"
printf '%s\n' "$f" >> "$RUNNER_TEMP/eslint_files.txt" ;;
*.json | *.css | *.scss | *.md | *.mdx | *.yml | *.yaml | *.html)
printf '%s\n' "$f" >> "$RUNNER_TEMP/prettier_files.txt" ;;
esac
done < <(git diff --name-only --diff-filter=ACMR --relative "$merge_base" "$HEAD_SHA" -- .)
if [ -s "$RUNNER_TEMP/prettier_files.txt" ] || [ -s "$RUNNER_TEMP/eslint_files.txt" ]; then
echo "has_files=true" >> "$GITHUB_OUTPUT"
else
echo "has_files=false" >> "$GITHUB_OUTPUT"
echo "No lintable UI files changed in this PR; nothing to check."
fi
- name: Setup Node.js
if: steps.changed.outputs.has_files == 'true'
uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5.0.0
with:
node-version-file: ui/litellm-dashboard/.nvmrc
cache: "npm"
cache-dependency-path: ui/litellm-dashboard/package-lock.json
- name: Install dependencies
if: steps.changed.outputs.has_files == 'true'
run: npm ci
- name: Lint changed files (prettier + eslint)
if: steps.changed.outputs.has_files == 'true'
run: |
prettier_files=()
eslint_files=()
while IFS= read -r f; do prettier_files+=("$f"); done < "$RUNNER_TEMP/prettier_files.txt"
while IFS= read -r f; do eslint_files+=("$f"); done < "$RUNNER_TEMP/eslint_files.txt"
status=0
if [ ${#prettier_files[@]} -gt 0 ]; then
echo "::group::Prettier (${#prettier_files[@]} files)"
npx prettier --check "${prettier_files[@]}" || { status=1; echo "::error::Unformatted files. Fix with: npm run format"; }
echo "::endgroup::"
fi
if [ ${#eslint_files[@]} -gt 0 ]; then
echo "::group::ESLint (${#eslint_files[@]} files)"
npx eslint --no-warn-ignored --pass-on-unpruned-suppressions "${eslint_files[@]}" || status=1
echo "::endgroup::"
fi
exit $status
- name: Check lint budgets
if: ${{ !cancelled() && steps.changed.outputs.has_files == 'true' }}
run: |
npx eslint . -f json -o "$RUNNER_TEMP/lint-report.json" || true
node scripts/check-lint-budgets.mjs "$RUNNER_TEMP/lint-report.json" eslint-budgets.json
- name: Check for dead code (knip)
if: ${{ !cancelled() && steps.changed.outputs.has_files == 'true' }}
run: npm run knip:ci
assert-ci-coverage:
name: assert-ci-coverage
permissions:
contents: read
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"
- name: Assert every test file and Dockerfile is invoked by a job
run: |
python -m pip install "pyyaml==6.0.3"
python .github/scripts/assert_ci_coverage.py
- name: Assert no -k expression deselects a file from every job that globs it
run: python .github/scripts/assert_ci_coverage.py --slices
- name: Assert .github/workflows/ holds only workflows, correctly named
run: python .github/scripts/assert_workflow_dir_hygiene.py
dashboard-build:
name: Dashboard build
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: Checkout
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
- name: Build the dashboard stage
run: docker build --target ui-builder -f Dockerfile .
core-checks:
name: Core checks (Python ${{ matrix.python-version }})
runs-on: ubuntu-24.04
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
env:
LITELLM_LOCAL_MODEL_COST_MAP: "True"
steps:
- name: Checkout
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: ${{ matrix.python-version }}
- name: Set up uv
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"
- name: Install dependencies
run: .github/scripts/uv_sync_with_retries.sh --frozen --extra proxy --extra cli --group dev --group proxy-dev --python ${{ matrix.python-version }}
- name: Create the loopback-only network namespace
run: |
sudo ip netns add smoke
sudo ip netns exec smoke ip link set lo up
cat > "${RUNNER_TEMP}/in-netns" <<'WRAP'
#!/usr/bin/env bash
set -euo pipefail
exec sudo --preserve-env=LITELLM_LOCAL_MODEL_COST_MAP ip netns exec smoke setpriv --reuid "$(id -u)" --regid "$(id -g)" --init-groups -- env HOME="${HOME}" PATH="${PATH}" "$@"
WRAP
chmod +x "${RUNNER_TEMP}/in-netns"
echo "IN_NETNS=${RUNNER_TEMP}/in-netns" >> "${GITHUB_ENV}"
- name: Verify namespace isolation
run: $IN_NETNS .venv/bin/python .github/scripts/run_merge_smoke.py verify-isolation
- name: Verify interpreter version
run: $IN_NETNS .venv/bin/python .github/scripts/run_merge_smoke.py interpreter --expect ${{ matrix.python-version }}
- name: Import and CLI checks
run: $IN_NETNS .venv/bin/python .github/scripts/run_merge_smoke.py cli
- name: Proxy startup check
run: $IN_NETNS .venv/bin/python .github/scripts/run_merge_smoke.py proxy-startup --diagnostics-dir "${RUNNER_TEMP}/smoke-diagnostics"
- name: Upload smoke diagnostics
if: always()
uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
with:
name: merge-smoke-diagnostics-py${{ matrix.python-version }}
path: ${{ runner.temp }}/smoke-diagnostics
if-no-files-found: ignore
- name: Remove the network namespace
if: always()
run: sudo ip netns delete smoke

31
.github/workflows/test-linting.yml vendored
View file

@ -593,6 +593,12 @@ jobs:
- name: check_migrations_no_data_rewrites
run: uv run --no-sync python ./tests/code_coverage_tests/check_migrations_no_data_rewrites.py
- name: check_greptile_files_json
run: uv run --no-sync python ./tests/code_coverage_tests/check_greptile_files_json.py
- name: test_check_greptile_files_json
run: uv run --no-sync pytest -q --noconftest -p no:cacheprovider -c /dev/null tests/code_coverage_tests/test_check_greptile_files_json.py
- name: check_no_publicly_known_master_key
run: uv run --no-sync python ./tests/code_coverage_tests/check_no_publicly_known_master_key.py
@ -639,9 +645,32 @@ jobs:
- name: Assert .github/workflows/ holds only workflows, correctly named
run: python .github/scripts/assert_workflow_dir_hygiene.py
semgrep:
name: semgrep
permissions:
contents: read
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"
- name: Set up uv
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"
- name: Run Semgrep (custom rules)
run: uv tool run --from 'semgrep==1.157.0' semgrep scan --config .semgrep/rules . --error
lint-passed:
name: lint passed
needs: [python, secret-scan, ui, ui-api-types, code-quality, ci-coverage]
needs: [python, secret-scan, ui, ui-api-types, code-quality, ci-coverage, semgrep]
if: always()
runs-on: ubuntu-latest
timeout-minutes: 2

40
.github/workflows/test-litellm-ui-build.yml vendored
View file

@ -1,40 +0,0 @@
name: UI Build Check
permissions:
contents: read
pull-requests: read
on:
pull_request:
branches:
- main
- "litellm_**"
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
build-ui:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout repository
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
- name: Detect relevant changes
id: changes
uses: ./.github/actions/detect-changes
with:
category: ui
# Built through the image stage rather than the checkout, because the
# stage copies ui/litellm-dashboard/ alone: an import reaching above the
# dashboard root resolves in a checkout and fails in every image we ship.
# Dockerfile, docker/Dockerfile.non_root and ui/Dockerfile share this
# stage verbatim, so building one covers all three.
- name: Build the dashboard as the shipped images build it
if: steps.changes.outputs.decision != 'skip'
run: docker build --target ui-builder -f Dockerfile .

16
.github/workflows/test-rust.yml vendored
View file

@ -6,9 +6,9 @@ on:
- "litellm-rust/**"
- "litellm/rust_bridge/**"
- "scripts/generate_trace_types.py"
- "scripts/generate_lens_contract.py"
- "litellm/proxy/lens/**"
- "scripts/trace_codegen/**"
- "scripts/lens_assets/**"
- "litellm/tracing/**"
- "tests/test_litellm_rust/**"
- "litellm/integrations/custom_logger.py"
- "litellm/litellm_core_utils/litellm_logging.py"
@ -37,9 +37,9 @@ on:
- "litellm-rust/**"
- "litellm/rust_bridge/**"
- "scripts/generate_trace_types.py"
- "scripts/generate_lens_contract.py"
- "litellm/proxy/lens/**"
- "scripts/trace_codegen/**"
- "scripts/lens_assets/**"
- "litellm/tracing/**"
- "tests/test_litellm_rust/**"
- "litellm/integrations/custom_logger.py"
- "litellm/litellm_core_utils/litellm_logging.py"
@ -93,7 +93,7 @@ jobs:
cache-on-failure: true
save-if: ${{ github.ref == 'refs/heads/main' }}
- run: cargo clippy --workspace --all-targets --locked --features litellm-traces/schema,litellm-traces-clickhouse/schema -- -D warnings
- run: cargo clippy --workspace --all-targets --locked -- -D warnings
rust-test:
runs-on: ubuntu-latest
@ -139,11 +139,7 @@ jobs:
working-directory: .
run: uv run scripts/generate_trace_types.py --check
- name: Check generated Lens contracts
working-directory: .
run: uv run scripts/generate_lens_contract.py --check
- run: cargo nextest run --workspace --locked --features litellm-traces/schema,litellm-traces-clickhouse/schema
- run: cargo nextest run --workspace --locked
- run: cargo test --workspace --doc --locked

37
.github/workflows/test-semgrep.yml vendored
View file

@ -1,37 +0,0 @@
name: Semgrep
on:
pull_request:
branches:
- main
- "litellm_**"
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
semgrep:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"
- name: Set up uv
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"
- name: Run Semgrep (custom rules)
run: uv tool run --from 'semgrep==1.157.0' semgrep scan --config .semgrep/rules . --error

132
.github/workflows/test-unit.yml vendored
View file

@ -137,6 +137,7 @@ jobs:
include:
- shard: core-utils
test-path: |-
tests/unit/batch_completion
tests/unit/decisions
tests/unit/litellm_core_utils
python-version: "3.12"
@ -213,7 +214,7 @@ jobs:
job-timeout-minutes: 60
dist: loadscope
- shard: llms-a-to-g
- shard: llms-providers
test-path: |-
tests/unit/llms/test_cache_control_and_reasoning.py
tests/unit/llms/test_custom_llm.py
@ -271,16 +272,6 @@ jobs:
tests/unit/llms/github_copilot
tests/unit/llms/gradient_ai
tests/unit/llms/groq
--ignore=tests/unit/llms/base_llm/batches/base_batches_config_test.py
python-version: "3.12"
workers: 4
reruns: 0
timeout-minutes: 20
job-timeout-minutes: 60
dist: loadscope
- shard: llms-h-to-z
test-path: |-
tests/unit/llms/heroku
tests/unit/llms/hosted_vllm
tests/unit/llms/huggingface
@ -299,6 +290,7 @@ jobs:
tests/unit/llms/lm_studio
tests/unit/llms/manus
tests/unit/llms/meta_llama
tests/unit/llms/microsoft_365_copilot
tests/unit/llms/minimax
tests/unit/llms/mistral
tests/unit/llms/modelscope
@ -344,6 +336,7 @@ jobs:
tests/unit/llms/snowflake
tests/unit/llms/soniox
tests/unit/llms/stability
tests/unit/llms/strands_decider
tests/unit/llms/tavily
tests/unit/llms/tencent
tests/unit/llms/tinyfish
@ -361,6 +354,7 @@ jobs:
tests/unit/llms/xinference
tests/unit/llms/you_com
tests/unit/llms/zai
--ignore=tests/unit/llms/base_llm/batches/base_batches_config_test.py
python-version: "3.12"
workers: 4
reruns: 0
@ -454,7 +448,6 @@ jobs:
- shard: enterprise-repositories-secrets
test-path: >-
tests/proxy_behavior/lens/test_connection.py
tests/unit/enterprise/enterprise_callbacks/test_callback_controls.py
tests/unit/enterprise/enterprise_callbacks/test_llm_guard.py
tests/unit/enterprise/enterprise_callbacks/test_secret_detection.py
@ -923,7 +916,7 @@ jobs:
- shard: lens-python-310
test-path: >-
tests/unit/proxy/lens/test_inference.py
tests/unit/proxy/lens
python-version: "3.10"
workers: 0
reruns: 0
@ -1235,6 +1228,117 @@ jobs:
uv run --no-sync python ./tests/documentation_tests/test_router_settings.py
uv run --no-sync python ./tests/documentation_tests/test_api_docs.py
uv run --no-sync python ./tests/documentation_tests/test_circular_imports.py
helm:
name: helm
permissions:
contents: read
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
- name: Set up Helm 3.11.1
uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4 # v4.3.1
with:
version: "3.11.1"
- name: Download and verify Helm Unit Test Plugin
run: |
curl -fsSLo "$RUNNER_TEMP/helm-unittest.tgz" https://github.com/helm-unittest/helm-unittest/releases/download/v0.8.2/helm-unittest-linux-amd64-0.8.2.tgz
echo "56ab3091e6fa52a7c92ee951def9bed957f295d9ce98483aed404e748d7b3a94 $RUNNER_TEMP/helm-unittest.tgz" | sha256sum -c -
- name: Install Helm Unit Test Plugin
run: |
PLUGIN_DIR="$(helm env HELM_PLUGINS)/helm-unittest"
mkdir -p "$PLUGIN_DIR"
tar -xzf "$RUNNER_TEMP/helm-unittest.tgz" -C "$PLUGIN_DIR"
helm plugin list
- name: Run unit tests
run: |
for chart in helm/litellm-helm helm/litellm; do
declared="$(grep -h '^suite:' "$chart"/tests/*.yaml | wc -l | tr -d '[:space:]')"
output="$(mktemp)"
helm unittest -f 'tests/*.yaml' "$chart" | tee "$output"
executed="$(sed -n 's/^Test Suites:.*[[:space:]]\([0-9][0-9]*\) total$/\1/p' "$output")"
if [ "$declared" != "$executed" ]; then
echo "::error::$chart declares $declared test suites but helm-unittest ran $executed. Suites are being skipped silently, so their assertions never execute."
exit 1
fi
echo "$chart: all $declared declared test suites ran"
done
openai-sdk-compat:
name: OpenAI ${{ matrix.openai-version }} compatibility
permissions:
contents: read
pull-requests: read
runs-on: ubuntu-latest
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
openai-version: ["2.20.0", "3.0.0", "3.25.0"]
env:
LITELLM_LOCAL_MODEL_COST_MAP: "True"
PYTHONPATH: ${{ github.workspace }}
UV_PYTHON: "3.10"
OPENAI_VERSION: ${{ matrix.openai-version }}
steps:
- uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
- name: Detect relevant changes
id: changes
timeout-minutes: 2
uses: ./.github/actions/detect-changes
- name: Set up Python
if: steps.changes.outputs.decision != 'skip'
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: ${{ env.UV_PYTHON }}
- name: Set up uv
if: steps.changes.outputs.decision != 'skip'
uses: ./.github/actions/setup-uv-with-retries
with:
version: "0.10.9"
- name: Install dependencies
if: steps.changes.outputs.decision != 'skip'
run: |
.github/scripts/uv_sync_with_retries.sh --frozen --group dev --group proxy-dev --extra proxy --no-install-workspace
uv pip install "openai==${OPENAI_VERSION:?}" --exclude-newer-package openai=2026-10-07T00:00:00Z
uv run --no-sync python -c 'import os, openai; assert openai.__version__ == os.environ["OPENAI_VERSION"], openai.__version__'
- name: Run OpenAI SDK compatibility tests
if: steps.changes.outputs.decision != 'skip'
run: |
uv run --no-sync pytest \
tests/unit/llms/openai/test_openai.py \
tests/unit/llms/openai/test_openai_common_utils.py \
tests/unit/llms/openai/test_openai_file_content_streaming.py \
tests/unit/test_openai_embedding_encoding_format_default.py \
tests/unit/llms/inception/test_inception_completion_transformation.py::test_inception_fim_targets_fim_endpoint \
tests/unit/llms/inception/test_inception_completion_transformation.py::test_inception_fim_async \
tests/unit/llms/azure/test_azure_exception_mapping.py \
tests/unit/test_completion_timeout_resolution.py \
tests/unit/types/test_router.py \
tests/unit/test_exception_header_preservation.py \
tests/unit/integrations/otel/test_otel_v2_sources_of_truth.py::test_speech_binary_response_still_streaming_reports_the_bytes_downloaded_so_far \
--tb=short -q
- name: Run workload identity tests with SDK 3
if: steps.changes.outputs.decision != 'skip' && startsWith(matrix.openai-version, '3.')
run: uv run --no-sync pytest tests/unit/llms/openai/test_openai_workload_identity.py --tb=short -q
- name: Check consumer types with the installed SDK
if: steps.changes.outputs.decision != 'skip'
run: uv run --no-sync basedpyright --pythonpath .venv/bin/python --project tests/typing/pyrightconfig.json
coverage:
name: coverage
needs: unit
@ -1283,7 +1387,7 @@ jobs:
unit-passed:
name: unit passed
permissions: {}
needs: [rust-bridge, assert-shard-coverage, unit, ui-unit, docs]
needs: [rust-bridge, assert-shard-coverage, unit, ui-unit, docs, helm, openai-sdk-compat]
if: always()
runs-on: ubuntu-latest
timeout-minutes: 2

1
.gitignore vendored
View file

@ -154,3 +154,4 @@ coverage-rust.xml
# make lens-dev worker token, generated config and logs
.lens-dev/
.mypy_cache/

473
.greptile/files.json Normal file
View file

@ -0,0 +1,473 @@
{
"files": [
{
"path": "litellm/_v2/AGENTS.md",
"description": "Conventions for code under litellm/_v2/",
"scope": [
"litellm/_v2/**"
]
},
{
"path": "litellm/_v2/cache/AGENTS.md",
"description": "Conventions for code under litellm/_v2/cache/",
"scope": [
"litellm/_v2/cache/**"
]
},
{
"path": "litellm/proxy/_experimental/mcp_server/AGENTS.md",
"description": "Conventions for code under litellm/proxy/_experimental/mcp_server/",
"scope": [
"litellm/proxy/_experimental/mcp_server/**"
]
},
{
"path": "litellm/rust_bridge/AGENTS.md",
"description": "Conventions for code under litellm/rust_bridge/",
"scope": [
"litellm/rust_bridge/**"
]
},
{
"path": "litellm/tracing/AGENTS.md",
"description": "Conventions for code under litellm/tracing/",
"scope": [
"litellm/tracing/**"
]
},
{
"path": "litellm-rust/AGENTS.md",
"description": "Conventions for code under litellm-rust/",
"scope": [
"litellm-rust/**"
]
},
{
"path": "litellm-rust/crates/cache-response/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/cache-response/",
"scope": [
"litellm-rust/crates/cache-response/**"
]
},
{
"path": "litellm-rust/crates/callbacks-legacy-python/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/callbacks-legacy-python/",
"scope": [
"litellm-rust/crates/callbacks-legacy-python/**"
]
},
{
"path": "litellm-rust/crates/coroutine/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/coroutine/",
"scope": [
"litellm-rust/crates/coroutine/**"
]
},
{
"path": "litellm-rust/crates/gateway/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/gateway/",
"scope": [
"litellm-rust/crates/gateway/**"
]
},
{
"path": "litellm-rust/crates/gateway-auth/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/gateway-auth/",
"scope": [
"litellm-rust/crates/gateway-auth/**"
]
},
{
"path": "litellm-rust/crates/gateway-inference/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/gateway-inference/",
"scope": [
"litellm-rust/crates/gateway-inference/**"
]
},
{
"path": "litellm-rust/crates/gateway-mcp/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/gateway-mcp/",
"scope": [
"litellm-rust/crates/gateway-mcp/**"
]
},
{
"path": "litellm-rust/crates/host/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/host/",
"scope": [
"litellm-rust/crates/host/**"
]
},
{
"path": "litellm-rust/crates/host/src/machine/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/host/src/machine/",
"scope": [
"litellm-rust/crates/host/src/machine/**"
]
},
{
"path": "litellm-rust/crates/host-http/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/host-http/",
"scope": [
"litellm-rust/crates/host-http/**"
]
},
{
"path": "litellm-rust/crates/host-native/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/host-native/",
"scope": [
"litellm-rust/crates/host-native/**"
]
},
{
"path": "litellm-rust/crates/host-python/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/host-python/",
"scope": [
"litellm-rust/crates/host-python/**"
]
},
{
"path": "litellm-rust/crates/http/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/http/",
"scope": [
"litellm-rust/crates/http/**"
]
},
{
"path": "litellm-rust/crates/inference/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/inference/",
"scope": [
"litellm-rust/crates/inference/**"
]
},
{
"path": "litellm-rust/crates/inference-chat/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/inference-chat/",
"scope": [
"litellm-rust/crates/inference-chat/**"
]
},
{
"path": "litellm-rust/crates/inference-messages/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/inference-messages/",
"scope": [
"litellm-rust/crates/inference-messages/**"
]
},
{
"path": "litellm-rust/crates/inference-ocr/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/inference-ocr/",
"scope": [
"litellm-rust/crates/inference-ocr/**"
]
},
{
"path": "litellm-rust/crates/inference-responses/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/inference-responses/",
"scope": [
"litellm-rust/crates/inference-responses/**"
]
},
{
"path": "litellm-rust/crates/inference-transcription/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/inference-transcription/",
"scope": [
"litellm-rust/crates/inference-transcription/**"
]
},
{
"path": "litellm-rust/crates/llms/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/llms/",
"scope": [
"litellm-rust/crates/llms/**"
]
},
{
"path": "litellm-rust/crates/llms/src/anthropic/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/llms/src/anthropic/",
"scope": [
"litellm-rust/crates/llms/src/anthropic/**"
]
},
{
"path": "litellm-rust/crates/llms/src/anthropic/batches/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/llms/src/anthropic/batches/",
"scope": [
"litellm-rust/crates/llms/src/anthropic/batches/**"
]
},
{
"path": "litellm-rust/crates/llms/src/anthropic/count_tokens/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/llms/src/anthropic/count_tokens/",
"scope": [
"litellm-rust/crates/llms/src/anthropic/count_tokens/**"
]
},
{
"path": "litellm-rust/crates/llms/src/anthropic/messages/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/llms/src/anthropic/messages/",
"scope": [
"litellm-rust/crates/llms/src/anthropic/messages/**"
]
},
{
"path": "litellm-rust/crates/llms/src/aws_textract/ocr/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/llms/src/aws_textract/ocr/",
"scope": [
"litellm-rust/crates/llms/src/aws_textract/ocr/**"
]
},
{
"path": "litellm-rust/crates/llms/src/azure_ai/messages/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/llms/src/azure_ai/messages/",
"scope": [
"litellm-rust/crates/llms/src/azure_ai/messages/**"
]
},
{
"path": "litellm-rust/crates/llms/src/base_llm/messages/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/llms/src/base_llm/messages/",
"scope": [
"litellm-rust/crates/llms/src/base_llm/messages/**"
]
},
{
"path": "litellm-rust/crates/llms/src/bedrock/messages/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/llms/src/bedrock/messages/",
"scope": [
"litellm-rust/crates/llms/src/bedrock/messages/**"
]
},
{
"path": "litellm-rust/crates/llms-types/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/llms-types/",
"scope": [
"litellm-rust/crates/llms-types/**"
]
},
{
"path": "litellm-rust/crates/llms-types/src/formats/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/llms-types/src/formats/",
"scope": [
"litellm-rust/crates/llms-types/src/formats/**"
]
},
{
"path": "litellm-rust/crates/llms-types/src/formats/messages/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/llms-types/src/formats/messages/",
"scope": [
"litellm-rust/crates/llms-types/src/formats/messages/**"
]
},
{
"path": "litellm-rust/crates/llms-types/src/providers/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/llms-types/src/providers/",
"scope": [
"litellm-rust/crates/llms-types/src/providers/**"
]
},
{
"path": "litellm-rust/crates/model-catalog/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/model-catalog/",
"scope": [
"litellm-rust/crates/model-catalog/**"
]
},
{
"path": "litellm-rust/crates/python-bridge/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/python-bridge/",
"scope": [
"litellm-rust/crates/python-bridge/**"
]
},
{
"path": "litellm-rust/crates/python-bridge/src/cache/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/python-bridge/src/cache/",
"scope": [
"litellm-rust/crates/python-bridge/src/cache/**"
]
},
{
"path": "litellm-rust/crates/python-bridge/src/cache/native/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/python-bridge/src/cache/native/",
"scope": [
"litellm-rust/crates/python-bridge/src/cache/native/**"
]
},
{
"path": "litellm-rust/crates/python-bridge/src/cache/python/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/python-bridge/src/cache/python/",
"scope": [
"litellm-rust/crates/python-bridge/src/cache/python/**"
]
},
{
"path": "litellm-rust/crates/python-bridge/src/routes/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/python-bridge/src/routes/",
"scope": [
"litellm-rust/crates/python-bridge/src/routes/**"
]
},
{
"path": "litellm-rust/crates/python-compat/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/python-compat/",
"scope": [
"litellm-rust/crates/python-compat/**"
]
},
{
"path": "litellm-rust/crates/secrets-aws/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/secrets-aws/",
"scope": [
"litellm-rust/crates/secrets-aws/**"
]
},
{
"path": "litellm-rust/crates/secrets-azure/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/secrets-azure/",
"scope": [
"litellm-rust/crates/secrets-azure/**"
]
},
{
"path": "litellm-rust/crates/secrets-cyberark/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/secrets-cyberark/",
"scope": [
"litellm-rust/crates/secrets-cyberark/**"
]
},
{
"path": "litellm-rust/crates/secrets-google/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/secrets-google/",
"scope": [
"litellm-rust/crates/secrets-google/**"
]
},
{
"path": "litellm-rust/crates/secrets-hashicorp/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/secrets-hashicorp/",
"scope": [
"litellm-rust/crates/secrets-hashicorp/**"
]
},
{
"path": "litellm-rust/crates/storage-clickhouse/AGENTS.md",
"description": "Conventions for code under litellm-rust/crates/storage-clickhouse/",
"scope": [
"litellm-rust/crates/storage-clickhouse/**"
]
},
{
"path": "litellm-rust/docs/adrs/AGENTS.md",
"description": "Conventions for code under litellm-rust/docs/adrs/",
"scope": [
"litellm-rust/docs/adrs/**"
]
},
{
"path": "tests/AGENTS.md",
"description": "Conventions for code under tests/",
"scope": [
"tests/**"
]
},
{
"path": "tests/e2e/AGENTS.md",
"description": "Conventions for code under tests/e2e/",
"scope": [
"tests/e2e/**"
]
},
{
"path": "tests/e2e_harness/AGENTS.md",
"description": "Conventions for code under tests/e2e_harness/",
"scope": [
"tests/e2e_harness/**"
]
},
{
"path": "tests/integration/AGENTS.md",
"description": "Conventions for code under tests/integration/",
"scope": [
"tests/integration/**"
]
},
{
"path": "tests/integration/translation/AGENTS.md",
"description": "Conventions for code under tests/integration/translation/",
"scope": [
"tests/integration/translation/**"
]
},
{
"path": "tests/rust-python-harness/AGENTS.md",
"description": "Conventions for code under tests/rust-python-harness/",
"scope": [
"tests/rust-python-harness/**"
]
},
{
"path": "tests/rust-python-harness/strategies/e2e_parity/AGENTS.md",
"description": "Conventions for code under tests/rust-python-harness/strategies/e2e_parity/",
"scope": [
"tests/rust-python-harness/strategies/e2e_parity/**"
]
},
{
"path": "tests/rust-python-harness/strategies/trace_parity/AGENTS.md",
"description": "Conventions for code under tests/rust-python-harness/strategies/trace_parity/",
"scope": [
"tests/rust-python-harness/strategies/trace_parity/**"
]
},
{
"path": "tests/rust-python-harness/strategies/unit_tests_parity/AGENTS.md",
"description": "Conventions for code under tests/rust-python-harness/strategies/unit_tests_parity/",
"scope": [
"tests/rust-python-harness/strategies/unit_tests_parity/**"
]
},
{
"path": "tests/rust-python-harness/strategies/unit_tests_rust/AGENTS.md",
"description": "Conventions for code under tests/rust-python-harness/strategies/unit_tests_rust/",
"scope": [
"tests/rust-python-harness/strategies/unit_tests_rust/**"
]
},
{
"path": "tests/test_litellm_rust/AGENTS.md",
"description": "Conventions for code under tests/test_litellm_rust/",
"scope": [
"tests/test_litellm_rust/**"
]
},
{
"path": "tests/unit/AGENTS.md",
"description": "Conventions for code under tests/unit/",
"scope": [
"tests/unit/**"
]
},
{
"path": "tests/unit/rust_bridge/AGENTS.md",
"description": "Conventions for code under tests/unit/rust_bridge/",
"scope": [
"tests/unit/rust_bridge/**"
]
},
{
"path": "ui/litellm-dashboard/AGENTS.md",
"description": "Conventions for code under ui/litellm-dashboard/",
"scope": [
"ui/litellm-dashboard/**"
]
},
{
"path": "ui/litellm-dashboard/src/components/chat/AGENTS.md",
"description": "Conventions for code under ui/litellm-dashboard/src/components/chat/",
"scope": [
"ui/litellm-dashboard/src/components/chat/**"
]
}
]
}

34
AGENTS.md
View file

@ -25,7 +25,7 @@ Same thing for bug fixes. The tests should make it so that this specific bug can
Never test structure of code only function of it
A test must only fail when litellm code changes. Never pin facts we don't own (a vendor's price, a third party's field, an upstream default, today's date) as literals or as "X must be absent"; assert the invariant our code guarantees instead, e.g. two rows agree, a value is within range, a field is derived from another. If an outside fact is truly load-bearing, cite its source and date next to the assertion so a reader can tell stale from broken
A test must only fail when litellm code changes. Never pin facts we don't own (a vendor's price, a third party's field, an upstream default, today's date) as literals or as "X must be absent". Assert the invariant our code guarantees instead, e.g. two rows agree, a value is within range, a field is derived from another. If an outside fact is truly load-bearing, cite its source and date next to the assertion so a reader can tell stale from broken
`tests/unit/` mirrors `litellm/` in a parallel path (see `tests/unit/AGENTS.md`). Name tests `test_<filename>.py`, but always match the existing test file in the directory you touch — many provider dirs use longer descriptive names (e.g. `test_anthropic_chat_transformation.py`) to avoid ambiguity across sibling folders. For bug fixes, extend the existing mapped test file rather than creating a new one. Only create a new test file for a new feature (provider, endpoint, or transformation module) that has no mapped test yet, following that directory's naming convention (or `test_<filename>.py` if you're the first test there). One focused regression test beats many shallow ones
@ -33,20 +33,20 @@ End-to-end tests belong in `tests/e2e/` and must follow the harness conventions
When creating PRs, target the repository's current default branch for both internal and external / OSS contributions. Check it with `python3 scripts/default_branch.py --branch` instead of assuming a branch name or relying on cached `origin/HEAD`
When writing a PR body, treat the comments and imperative instructions inside .github/pull_request_template.md as rules to follow, not just layout. Agent harnesses may strip HTML comments from copies of that file injected into context, so read .github/pull_request_template.md from disk before writing a PR body to make sure you see every comment rule. A section you have nothing to put in (Relevant issues, Affected release, Linear ticket, Caveats, QA runbook, and so on) is removed entirely, heading included, never left as an empty title
When writing a PR body, treat the comments and imperative instructions inside .github/PULL_REQUEST_TEMPLATE/general.md as rules to follow, not just layout. Agent harnesses may strip HTML comments from copies of that file injected into context, so read .github/PULL_REQUEST_TEMPLATE/general.md from disk before writing a PR body to make sure you see every comment rule. A section you have nothing to put in (Relevant issues, Affected release, Linear ticket, Caveats, QA runbook, and so on) is removed entirely, heading included, never left as an empty title
Same applies for filing bug reports and feature requests, with .github/ISSUE_TEMPLATE/bug_report.yml and .github/ISSUE_TEMPLATE/feature_request.yml, respectively
If you're resolving a linear ticket, in the "## Linear ticket" section of the PR, say "Resolves LIT-1234", replacing "LIT-1234" with the actual ticket id that you're resolving. If you don't have the ticket id, don't make one up or search for it. Just drop the section
Never use `pytest` commands or the like as "Screenshots / Proof of Fix". We prefer curl'ing a live proxy instance running on localhost:4000 (I like to run it with `python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --detailed_debug --reload --use_v2_migration_resolver 2>&1 | tee litellm.log`; the Admin UI dev server is `npm run dev` in `ui/litellm-dashboard`, served on port 3000) and showing both the command run and the output. Also, it should hit real LLM provider APIs, not mocks, and cost real $$$ because that is the most realistic test. The proof of fix should be exactly what the end user / customer would see / do. The run logs in PR #27703 is a prime example of how to do it (not a huge fan of using a python test script that future me and the team will have no visibility into; I prefer just curl commands or a short list of bash commands (e.g., using `for`)). If it's a UI thing, or the main use case runs through a headful agentic coding tool like Claude Code or Codex, drive that surface yourself and embed your own before and after screenshots of it in the PR (the Admin UI page, or what the coding tool shows), next to an ordered list of the URLs to go to (e.g., http://localhost:4000/ui/?page=logs), where to click, and what fields to fill out so a reviewer can reproduce it
Never use `pytest` commands or the like as "Screenshots / Proof of Fix". We prefer curl'ing a live proxy instance running on localhost:4000 (I like to run it with `python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --detailed_debug --reload --use_v2_migration_resolver 2>&1 | tee litellm.log`. The Admin UI dev server is `npm run dev` in `ui/litellm-dashboard`, served on port 3000) and showing both the command run and the output. Also, it should hit real LLM provider APIs, not mocks, and cost real $$$ because that is the most realistic test. The proof of fix should be exactly what the end user / customer would see / do. The run logs in PR #27703 is a prime example of how to do it (not a huge fan of using a python test script that future me and the team will have no visibility into. I prefer just curl commands or a short list of bash commands (e.g., using `for`)). If it's a UI thing, or the main use case runs through a headful agentic coding tool like Claude Code or Codex, drive that surface yourself and embed your own before and after screenshots of it in the PR (the Admin UI page, or what the coding tool shows), next to an ordered list of the URLs to go to (e.g., http://localhost:4000/ui/?page=logs), where to click, and what fields to fill out so a reviewer can reproduce it
If you ever write any human-facing text (pull requests, issues, commit messages, discussion posts, github comments, release notes, docs, etc.), always follow these guidelines to sound less AI-y:
- don't use emojis
- don't use "—". Instead, reach for ",", ".", conjunction words, ":", ";", etc. in descending order of preference: vary among them, weighted toward the front of the list, and skip "," where it would cause a comma splice or the sentence is getting long. Overusing any one of them, ";" especially, also feels AI-y. A word cap does not penalize you for adding more sentences: when writing under tight word budgets, prefer a period split or a conjunction over ";", and keep to at most one ";" per message
- don't use "—". Don't use ";" or ":" unless it would be nonsensical not to. Instead, reach for ",", ".", conjunction words, etc. in descending order of preference: vary among them, weighted toward the front of the list, and skip "," where it would cause a comma splice or the sentence is getting long. Overusing any one of them also feels AI-y. A word cap does not penalize you for adding more sentences
- don't use the pattern "It's not X, it's Y", "You're not X, you're Y", etc.
- unless explicitly asked, don't use bulleted or numbered lists unless it would be nonsensical not to. Instead, prefer prose
- don't add a trailing "." at the end of paragraphs (just like this file). That means every paragraph, not just the last one (of the markdown file, PR description, GitHub comment, etc.). Rule of thumb: if you're adding new line(s) before the next sentence, don't add a "."
- don't add a trailing "." at the end of paragraphs (just like this file). That means every paragraph, not just the last one (of the markdown file, PR description, GitHub comment, etc.). Rule of thumb: if you're adding new line(s) before the next sentence, don't add a ".". Exception is that docs (i.e., litellm-docs) should have trailing "."
- don't use →. Instead, prefer not to use arrows, and if need be, use -> instead
- use plain, simple, everyday engineering language: the common phrase engineers actually say over rare compact phrasing, in grammatically complete sentences. When explicitly asked to use bullets or ordered lists and structure legitimately helps the reader, prefer nested bullets (any depth is fine) over dense lines in a flat structure
@ -58,17 +58,17 @@ The four lint gates (`scripts/ruff_strict_gate.py`, `scripts/type_discipline_gat
`make check` (f.k.a. `make pre-commit`, which still works identically as an alias) saves its complete output to a log file in .git (overwriting previous logs) and prints that path as its first and last output lines. To inspect a run, read or grep that log instead of re-running the multi-minute checks just to see a different slice
`make check`, `make lint`, `scripts/pre_commit_lint.sh`, and the standalone lint gates (`scripts/ruff_strict_gate.py`, `scripts/type_discipline_gate.py`, `scripts/type_check_gate.py`) each hold one of 2 machine-wide slots, so when other sessions or worktrees on the same box are already running heavy work, yours prints "all N machine-wide slots are busy; queueing" and then stays quiet until a slot frees. Give the command a long timeout and let it wait rather than killing it, retrying it, or assuming it hung. Don't change the # of machine-wide slots or make it unlimited by setting `LITELLM_GATE_SLOTS=0`
`make check`, `make lint`, `scripts/pre_commit_lint.sh`, and the standalone lint gates (`scripts/ruff_strict_gate.py`, `scripts/type_discipline_gate.py`, `scripts/type_check_gate.py`) each hold one of 2 machine-wide slots, so when other sessions or worktrees on the same box are already running heavy work, yours prints "all N machine-wide slots are busy" followed by "queueing" and then stays quiet until a slot frees. Give the command a long timeout and let it wait rather than killing it, retrying it, or assuming it hung. Don't change the # of machine-wide slots or make it unlimited by setting `LITELLM_GATE_SLOTS=0`
If you're trying to create a new function that relies on untyped stuff, instead of adding more Any's and pushing `reportAny` / `reportExplicitAny` closer to their cap in `ANY_CAPS`, just validate it in the caller with Pydantic (a model or `TypeAdapter` that returns the typed thing or raises will do) and then pass the now typed variable in
If you get an LIT001 fail, refactor the code to follow functional programming best practices rather than introducing mutable data structures. For example, build values in one shot with comprehensions or generators wrapped in `tuple()` / `MappingProxyType()` / `frozenset()` instead of seeding an empty `list`/`dict`/`set` and mutating it over time. Ideally, `# mutable-ok` is never used; reach for it only as a genuine last resort when an immutable rewrite is truly impossible, and always pair it with a real reason
If you get an LIT001 fail, refactor the code to follow functional programming best practices rather than introducing mutable data structures. For example, build values in one shot with comprehensions or generators wrapped in `tuple()` / `MappingProxyType()` / `frozenset()` instead of seeding an empty `list`/`dict`/`set` and mutating it over time. Ideally, `# mutable-ok` is never used. Reach for it only as a genuine last resort when an immutable rewrite is truly impossible, and always pair it with a real reason
Every lint or type suppression must name the exact rule inside brackets and carry a reason comment, e.g. `# pyright: ignore[reportArgumentType] # stubs lack async overload` or `# noqa: TID251 # <reason>`. `# type: ignore` is banned (LIT009): pyrightconfig.json sets `enableTypeIgnoreComments` to false, so it silently does nothing
Commit and push your work when you're done without asking
When referencing or running models (coding, QA'ing, writing docs, writing tests, etc.), use the latest model in that model family unless otherwise specified; treat your training knowledge, memories, configs, and tests as stale, and determine the family's latest with model_prices_and_context_window.json or the web
When referencing or running models (coding, QA'ing, writing docs, writing tests, etc.), use the latest model in that model family unless otherwise specified. Treat your training knowledge, memories, configs, and tests as stale, and determine the family's latest with model_prices_and_context_window.json or the web
Always pull before starting any work. The checkout or worktree may be sitting on a stale branch
@ -84,7 +84,7 @@ Monkeypatching attributes of a class to do testing is an anti-pattern. Prefer de
Do not put names of customers or customer company names in code, PR descriptions, issue bodies, etc. This means never mention literally any company name. Especially if you're about to say a sentence mentioning that the reason the PR exists was a feature/model/bug fix/etc. requested by a company. That's the indication that you should replace that company name with "the customer". e.g. not "Model request from Acme (Pylon #1234)" but "Model request from a customer (Pylon #1234)". This is because the codebase is public. The only exception is for publicly known providers or vendors such as OpenAI, Anthropic, AWS Bedrock, etc. only IF we're adding support for that provider/vendor in general and NOT if that PR or whatnot was a request by one of them, and they're actually one of our customers.
CI supply-chain safety: Never pipe a remote script into a shell (`curl ... | bash`, `wget ... | sh`); download the artifact to a file, verify its SHA-256 checksum, then install. Pin every external tool to a specific version with a full URL (not `latest` or `stable`). Verify checksums for all downloaded binaries, using the provider's official `.sha256` / `.sha256sum` sidecar when available. These rules apply to every download in CI
CI supply-chain safety: Never pipe a remote script into a shell (`curl ... | bash`, `wget ... | sh`). Download the artifact to a file, verify its SHA-256 checksum, then install. Pin every external tool to a specific version with a full URL (not `latest` or `stable`). Verify checksums for all downloaded binaries, using the provider's official `.sha256` / `.sha256sum` sidecar when available. These rules apply to every download in CI
Prisma migrations apply synchronously at proxy boot, before it serves traffic, so a migration must only change schema, never rewrite rows. No `UPDATE`, `DELETE` or `MERGE`, and no `INSERT ... SELECT`: on a spend-log-sized table any of those is minutes of downtime plus a doubled heap that plain autovacuum won't give back. `tests/code_coverage_tests/check_migrations_no_data_rewrites.py` enforces this. When a rewrite is genuinely bounded and has to ship inside the migration, mark the statement `-- data-migration-ok: <what bounds it>`
@ -92,19 +92,19 @@ Follow these coding conventions for new/updated code (a three-line fix in a lega
- Composition over inheritance
- Never-nester: early returns over deep nesting
- Don't throw; model failures as values (One function (e.g., raise_public) maps error union to existing public exception contracts via exhaustive match + assert_never)
- No mutation; don't reassign variables, global or local. Instead of mutable lists and dicts, prefer tuples, frozen dataclasses (with slots=True), `MappingProxyType`, etc.
- Annotate every variable with `: Final` (LIT010). Unpacking and walrus targets cannot carry the annotation, so they are implicitly final. Don't rebind them. Never rebind or mutate function parameters (LIT011); `self`/`cls` attribute stores are the exception. If rebinding or in-place mutation is truly unavoidable, suppress with `# rebind-ok: <reason>`
- Don't throw. Model failures as values (One function (e.g., raise_public) maps error union to existing public exception contracts via exhaustive match + assert_never)
- No mutation. Don't reassign variables, global or local. Instead of mutable lists and dicts, prefer tuples, frozen dataclasses (with slots=True), `MappingProxyType`, etc.
- Annotate every variable with `: Final` (LIT010). Unpacking and walrus targets cannot carry the annotation, so they are implicitly final. Don't rebind them. Never rebind or mutate function parameters (LIT011). `self`/`cls` attribute stores are the exception. If rebinding or in-place mutation is truly unavoidable, suppress with `# rebind-ok: <reason>`
- Qualify every TypedDict field with `ReadOnly[...]` (LIT012), which nests freely with `Required` / `NotRequired` / `Annotated` in any order. If making the key writable is truly unavoidable, suppress with `# writable-ok: <reason>`
- Comprehensions take at most one `for` clause and one `if` clause (LIT014); split stacked clauses into a helper generator, a named intermediate, or a plain loop. Suppress with `# comprehension-ok: <reason>` only when unavoidable
- Comprehensions take at most one `for` clause and one `if` clause (LIT014). Split stacked clauses into a helper generator, a named intermediate, or a plain loop. Suppress with `# comprehension-ok: <reason>` only when unavoidable
- Every pydantic model must be frozen (LIT015), set via `model_config = ConfigDict(frozen=True)`, a dict-literal `model_config`, an inner `class Config`, or the class keywords. Subclasses inherit it unless they override it. Replace in-place field writes with `model_copy(update=...)`. If making the model mutable is truly unavoidable, suppress with `# frozen-ok: <reason>` on the `class` line
- Use dependency injection
- Fully typed; no `Any` or coarse types like `dict[str, Any]` or just `dict`. Every function parameter must be strongly typed
- Fully typed. No `Any` or coarse types like `dict[str, Any]` or just `dict`. Every function parameter must be strongly typed
- Use tagged unions + match
- No monster files or god objects
- No file sprawl: deliberate file and folder structure
- Standard over hand-rolled: use the official SDK or a library where one exists; where none does, follow industry standards instead of inventing local conventions
- API-fragmentation-aware: when logic must branch on which API surface produced or consumes data (e.g. chat completions vs Anthropic Messages vs Responses API shapes), proactively look for an existing shared helper (e.g. `litellm_core_utils/prompt_templates/factory.py`) before writing per-surface parsing in the new module; if none exists, add one there instead of duplicating the same format-detection logic in every new guardrail/integration
- Standard over hand-rolled: use the official SDK or a library where one exists. Where none does, follow industry standards instead of inventing local conventions
- API-fragmentation-aware: when logic must branch on which API surface produced or consumes data (e.g. chat completions vs Anthropic Messages vs Responses API shapes), proactively look for an existing shared helper (e.g. `litellm_core_utils/prompt_templates/factory.py`) before writing per-surface parsing in the new module. If none exists, add one there instead of duplicating the same format-detection logic in every new guardrail/integration
Follow conventional commits for commit names and PR titles
@ -130,4 +130,4 @@ Before implementing:
Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify
Before requesting maintainer review, verify the current PR tip passes required CI and code coverage, meets Greptile confidence of at least 4/5, and has acceptable Veria and Bugbot reviews. Inspect warnings and findings, fix actionable issues, and rerun the affected checks and reviewers after changes. Record evidence for any false positive or unavailable review; never treat a pending or missing bot result as a pass. Do not lower coverage thresholds or raise `ANY_CAPS` to satisfy a check
Before requesting maintainer review, verify the current PR tip passes required CI and code coverage, meets Greptile confidence of at least 4/5, and has acceptable Veria and Bugbot reviews. Inspect warnings and findings, fix actionable issues, and rerun the affected checks and reviewers after changes. Record evidence for any false positive or unavailable review. Never treat a pending or missing bot result as a pass. Do not lower coverage thresholds or raise `ANY_CAPS` to satisfy a check

1
Dockerfile
View file

@ -39,6 +39,7 @@ ENV NEXT_TELEMETRY_DISABLED=1 \
WORKDIR /ui
COPY ui/litellm-dashboard/package.json ui/litellm-dashboard/package-lock.json ./
COPY ui/litellm-dashboard/vendor/ ./vendor/
RUN --mount=type=cache,target=/root/.npm npm ci --prefer-offline
COPY ui/litellm-dashboard/ ./

6
Makefile
View file

@ -4,7 +4,7 @@
.PHONY: help test test-unit test-unit-llms test-unit-proxy-guardrails test-unit-proxy-core test-unit-proxy-misc test-unit-proxy-root \
test-unit-integrations test-unit-core-utils test-unit-other test-unit-root \
test-proxy-unit-a test-proxy-unit-b test-integration test-unit-helm \
test-rust-extension rust-sqlx-prepare lens-dev \
test-rust-extension rust-sqlx-prepare \
info lint lint-inner lint-dev lint-checks format \
lint-basedpyright lint-e2e-basedpyright lint-type-discipline \
lint-ruff-strict lint-gate lint-test-quality \
@ -54,7 +54,6 @@ help:
@echo " make test-unit-helm - Run helm unit tests"
@echo " make test-rust-extension - Build the Rust extension and run its public Python tests"
@echo " make rust-sqlx-prepare - Refresh litellm-rust/crates/db/.sqlx against a migrated Postgres container"
@echo " make lens-dev - Run proxy + Lens worker + hot-reload dashboard (ARGS=\"--seed large --seed-logs\", LENS_DEV_PROXY_PORT, LENS_DEV_UI_PORT)"
@echo ""
@echo "Heavy targets (check, lint) queue for LITELLM_GATE_SLOTS machine-wide"
@echo "slots (default 2; 0 disables) so parallel sessions don't thrash one machine."
@ -290,9 +289,6 @@ test-rust-extension:
rust-sqlx-prepare:
cd litellm-rust && cargo run -p litellm-db-testing --bin sqlx-prepare
lens-dev:
./scripts/lens_dev.sh $(ARGS)
test: install-test-deps
$(UV_RUN) pytest tests/

3
backend/routes/allowlist.py
View file

@ -91,9 +91,10 @@ BACKEND_PATH_PREFIXES: tuple[str, ...] = (
"/user_agent",
"/usage/",
"/daily/",
# Deployment-wide gateway request counts. Scoped to the analytics read rather
# Deployment-wide gateway request and error counts. Scoped to the analytics reads rather
# than all of /gateway/, which stays free for data-plane routes.
"/gateway/daily/",
"/gateway/errors/",
# CloudZero cost-export admin (init / settings / export / dry-run / delete)
"/cloudzero/",
# Caching admin

56
ci_cd/cost_map_guard.py
View file

@ -14,8 +14,9 @@ import argparse
import json
import subprocess
import sys
from collections.abc import Sequence
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
from types import MappingProxyType
from typing import Final
from generate_model_prices_schema import SPECIAL_ROOT_KEYS, build_schema, render, validation_errors
@ -25,6 +26,7 @@ BACKUP_PATH: Final = "litellm/model_prices_and_context_window_backup.json"
SCHEMA_PATH: Final = "model_prices_and_context_window.schema.json"
GUARDED_PATHS: Final = (COST_MAP_PATH, BACKUP_PATH, SCHEMA_PATH)
BOT_BRANCH_PREFIX: Final = "litellm_cost_map_sync_"
EVALUATION_OUTPUT_PRICE_SOURCES: Final[Mapping[str, str]] = MappingProxyType({})
CostMap = dict[str, object]
@ -51,7 +53,31 @@ def _rendered_schema(cost_map: CostMap) -> str:
return str(error)
def _file_failures(head: Snapshot, head_map: CostMap) -> tuple[str, ...]:
def _evaluation_output_price_failure(
key: str, entry: object, evaluation_output_price_sources: Mapping[str, str]
) -> str | None:
if not isinstance(entry, dict) or entry.get("mode") != "evaluation":
return None
output_cost: Final = entry.get("output_cost_per_token")
if (
not isinstance(output_cost, (int, float))
or isinstance(output_cost, bool)
or output_cost <= 0
or key in evaluation_output_price_sources
):
return None
return (
f"{key}: mode evaluation rows bill input only; output_cost_per_token must be 0 unless the vendor "
"prices output tokens for this model (add the source to EVALUATION_OUTPUT_PRICE_SOURCES)"
)
def _file_failures(
head: Snapshot,
head_map: CostMap,
*,
evaluation_output_price_sources: Mapping[str, str] = EVALUATION_OUTPUT_PRICE_SOURCES,
) -> tuple[str, ...]:
schema_text: Final = _rendered_schema(head_map)
if not schema_text.startswith("{"):
return (schema_text,)
@ -75,6 +101,16 @@ def _file_failures(head: Snapshot, head_map: CostMap) -> tuple[str, ...]:
f"{COST_MAP_PATH} does not validate against its schema: {error}"
for error in validation_errors(head_map, json.loads(schema_text))[:20]
),
*(
failure
for key, entry in head_map.items()
if (
failure := _evaluation_output_price_failure(
key, entry, evaluation_output_price_sources
)
)
is not None
),
)
@ -121,13 +157,25 @@ def contract_for(bot: bool, changed_files: Sequence[str]) -> str:
return "human PR, file checks only" if touches_cost_map(changed_files) else "human PR, cost map untouched"
def guard_failures(base: Snapshot, head: Snapshot, changed_files: Sequence[str], bot: bool) -> tuple[str, ...]:
def guard_failures(
base: Snapshot,
head: Snapshot,
changed_files: Sequence[str],
bot: bool,
*,
evaluation_output_price_sources: Mapping[str, str] = EVALUATION_OUTPUT_PRICE_SOURCES,
) -> tuple[str, ...]:
if not bot and not touches_cost_map(changed_files):
return ()
head_map: Final = _parse_object(head.cost_map, COST_MAP_PATH)
if isinstance(head_map, str):
return (head_map,)
return (*_file_failures(head, head_map), *(_bot_failures(base, head_map, changed_files) if bot else ()))
return (
*_file_failures(
head, head_map, evaluation_output_price_sources=evaluation_output_price_sources
),
*(_bot_failures(base, head_map, changed_files) if bot else ()),
)
def _git(*args: str) -> str | None:

41
deploy/lens/Dockerfile
View file

@ -1,41 +0,0 @@
ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:1d95114038f76513a9ace6fca107d5582b08c65981f81f61cb56bf7fd2ef216d
ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:1d95114038f76513a9ace6fca107d5582b08c65981f81f61cb56bf7fd2ef216d
FROM $LITELLM_BUILD_IMAGE AS builder
RUN apk add --no-cache rust build-base cmake perl pkgconf openssl-dev libseccomp-dev python-3.13
WORKDIR /src
COPY .cargo/ .cargo/
COPY litellm-rust/ litellm-rust/
COPY litellm/proxy/lens/prompts/ litellm/proxy/lens/prompts/
WORKDIR /src/litellm-rust
ENV CARGO_PROFILE_RELEASE_DEBUG=0 CARGO_PROFILE_RELEASE_STRIP=symbols
RUN cargo build --locked --release -p litellm-lens
COPY deploy/lens/python_policy.c /tmp/python_policy.c
RUN cc -std=c11 -D_GNU_SOURCE -O2 -Wall -Wextra -Werror /tmp/python_policy.c -lseccomp -o /tmp/python-policy && \
/tmp/python-policy /tmp/python.seccomp
FROM builder AS test-builder
RUN cargo test --locked --release -p litellm-lens --test sandbox --no-run --message-format=json > /tmp/test-artifacts.json && \
python3.13 -c 'import json, pathlib, shutil; rows = [json.loads(line) for line in pathlib.Path("/tmp/test-artifacts.json").read_text().splitlines()]; artifact, = [r["executable"] for r in rows if r.get("executable") and r["target"]["name"] == "sandbox"]; shutil.copyfile(artifact, "/tmp/lens-sandbox-tests")' && \
chmod 755 /tmp/lens-sandbox-tests
FROM $LITELLM_RUNTIME_IMAGE AS service
ARG LITELLM_RELEASE_TAG=""
RUN : "${LITELLM_RELEASE_TAG:?Pass --build-arg LITELLM_RELEASE_TAG matching the gateway}"
RUN apk add --no-cache python-3.13 setpriv libgcc libstdc++ openssl ca-certificates
ENV LITELLM_RELEASE_TAG=${LITELLM_RELEASE_TAG} PYTHONDONTWRITEBYTECODE=1
WORKDIR /app
COPY --from=builder /src/litellm-rust/target/release/litellm-lens /usr/local/bin/litellm-lens
COPY --from=builder /tmp/python.seccomp /app/lens/python.seccomp
COPY deploy/lens/python_runtime.py /tmp/python_runtime.py
RUN python3.13 -S /tmp/python_runtime.py /app/lens/python-runtime.json && rm /tmp/python_runtime.py
USER 65532:65532
EXPOSE 4318
ENTRYPOINT ["/usr/local/bin/litellm-lens"]
FROM service AS smoke
COPY --from=test-builder /tmp/lens-sandbox-tests /usr/local/bin/lens-sandbox-tests
ENTRYPOINT ["/usr/local/bin/lens-sandbox-tests"]
CMD ["--ignored", "--nocapture", "--test-threads=1"]
FROM service AS runtime

15
deploy/lens/Dockerfile.dockerignore
View file

@ -1,15 +0,0 @@
**
!deploy/
!deploy/lens/
!deploy/lens/python_policy.c
!deploy/lens/python_runtime.py
!litellm/
!litellm/proxy/
!litellm/proxy/lens/
!litellm/proxy/lens/prompts/
!litellm/proxy/lens/prompts/**
!.cargo/
!.cargo/**
!litellm-rust/
!litellm-rust/**
litellm-rust/target/

282
deploy/lens/README.md
View file

@ -1,282 +0,0 @@
# Lens service
Lens records agent activity and investigates it in a separate Rust service. LiteLLM serves model requests, the dashboard, and investigation settings. Lens owns trace ingestion and ClickHouse access; PostgreSQL stays with LiteLLM
Agent exporters send traces directly to Lens. LiteLLM sends its optional request logs through a bounded background queue. If Lens or ClickHouse is unavailable, model requests continue; traces can be delayed or dropped according to the exporter's retry policy. The gateway never waits for ClickHouse during startup or inference
## New local installation
Install Docker with Compose, Python 3.10 or later, and Git. Clone LiteLLM, select a published release that includes Lens, and start the existing Compose stack:
```bash
git clone https://github.com/BerriAI/litellm.git
cd litellm
python3 deploy/lens/configure.py --version <release-version>
docker compose --env-file deploy/lens/.env -f deploy/lens/stack.yaml up -d --wait
```
The configuration command generates your keys and database passwords once, saves them in `deploy/lens/.env` with owner-only permissions, and preserves them on subsequent runs. Back up this file alongside your database volumes. Both images use the selected release; there is no local image build
Open `http://localhost:4000/ui/` and sign in as `admin` using `LITELLM_MASTER_KEY` from the saved file. Open **Lens**, select your framework, generate a tracing key, and copy the displayed configuration. The trace endpoint is already filled in. Keep your agent's existing model credentials; the tracing key only authorizes trace uploads
PostgreSQL and ClickHouse use persistent Docker volumes and have no host ports. The dashboard and trace listener bind to localhost. Use your normal TLS and ingress for a hosted deployment. Stop the stack with `docker compose --env-file deploy/lens/.env -f deploy/lens/stack.yaml down`; omit `-v` to retain data
Under **Lens > Investigations > Connect worker**, choose an analysis model and monthly budget. The deployed service connects automatically after you save these settings. There is no worker command or second token to copy
## Existing LiteLLM installation
Keep your gateway, PostgreSQL database, deployment tool, and existing encryption keys. Deploy the matching Lens image, give it access to ClickHouse, and configure the service connection on LiteLLM
| Variable | LiteLLM | Lens service |
| --- | --- | --- |
| `LITELLM_LENS_SERVICE_TOKEN` | Same private random secret, at least 32 characters | Same secret |
| `LITELLM_LENS_URL` | Internal Lens URL, such as `http://lens-worker:4318` | Not needed |
| `LITELLM_LENS_PUBLIC_URL` | Ingestion base URL reachable by your agents | Not needed |
| `LITELLM_URL` | Not needed | LiteLLM URL reachable from Lens |
| `CLICKHOUSE_URL` | Remove it from Lens tracing configuration | ClickHouse HTTP URL with credentials |
| `CLICKHOUSE_DATABASE` | Not needed for Lens | Existing database name, defaults to `litellm` |
| `AGENT_TRACING_RETENTION_DAYS` | Not needed for Lens | Retention for traces and Lens request logs, defaults to `14` |
Remove the old `general_settings.tracing.store` configuration used for Lens from LiteLLM. Keep unrelated logging integrations and their configuration. Only Lens should reach its ClickHouse database. The shared service secret is an infrastructure credential: keep it out of browser code, agent exporters, screenshots, and public ingress headers
Expose the Lens HTTP listener on port 4318 through TLS. Route `/lens-ingest` on your existing hostname directly to Lens at the load balancer, then set `LITELLM_LENS_PUBLIC_URL=https://<your-host>/lens-ingest`. The gateway must not proxy these uploads. Alternatively use a separate hostname and forward `/v1/` to Lens. Keep `/internal/` private; it requires the service secret
### Standalone Docker or a container host
Build from the same source commit and `LITELLM_RELEASE_TAG` as your running gateway:
```bash
export LITELLM_RELEASE_TAG='<gateway-release-identity>'
export LENS_WORKER_IMAGE='<your-registry>/litellm-lens-worker:<image-tag>'
docker build --build-arg LITELLM_RELEASE_TAG="$LITELLM_RELEASE_TAG" \
-f deploy/lens/Dockerfile -t "$LENS_WORKER_IMAGE" .
```
Publish that image to a registry your host can pull from. Prefer a digest reference for hosted deployments. Public development images use `ghcr.io/berriai/litellm-lens-worker-dev:sha-<full-commit>`; check that the exact image exists before selecting it. An arbitrary commit may not have a published image
The image supports native amd64 and arm64. For worker-only Compose, use `deploy/lens/compose.yaml` with a private environment file containing `LENS_WORKER_IMAGE`, `LITELLM_URL`, `LITELLM_LENS_SERVICE_TOKEN`, and `CLICKHOUSE_URL`:
```bash
docker compose --env-file /path/to/private/lens.env \
-f deploy/lens/compose.yaml up -d
```
The Compose listener binds to localhost. Your reverse proxy must reach it. On Render, run Lens as a web service with the same environment and listener port 4318, not an outbound-only background worker. Use `/health/live` for process health and `/health/ready` to check storage and tracing credentials
Lens does not need provider credentials, PostgreSQL credentials, a GPU, or the LiteLLM Python package. The image includes a small CPython runtime only for the investigator's confined calculation tool. Keep the shipped security settings, temporary filesystem, and resource limits
### Kubernetes with Helm
Both `helm/litellm` and `helm/litellm-helm` support Lens. Keep your existing chart, release name, namespace, and values. Add:
```yaml
lensWorker:
enabled: true
```
Then run your usual Helm deployment command using the matching published chart. The chart supplies the matching Lens image, generates the shared service secret, starts a single ClickHouse instance with a persistent volume, and connects the services. Your cluster needs a default storage class, or set `lensWorker.clickhouse.storageClassName`. Bundled storage defaults to 20 GiB; set `lensWorker.clickhouse.storage` before installation to choose another size
When your chart manages an ingress with one hostname, the chart fills in the public tracing address and routes `/lens-ingest` directly to Lens. TLS is detected from `ingress.tls` or an ALB certificate annotation. With custom ingress, multiple hostnames, or TLS terminated elsewhere, set the address explicitly:
```yaml
lensWorker:
enabled: true
publicUrl: https://<your-litellm-host>/lens-ingest
```
For a dedicated trace hostname, configure `lensWorker.ingress.enabled`, `host`, `className`, and `tls`. Its hostname supplies the public address unless you override `publicUrl`. Internal Lens routes stay private
To use an existing ClickHouse database and secrets managed by your platform, keep these overrides:
```yaml
lensWorker:
enabled: true
serviceTokenSecret:
name: litellm-lens-service
key: service-token
clickhouseSecret:
name: litellm-lens-clickhouse
key: url
clickhouseDatabase: litellm
retentionDays: 14
```
Supplying `clickhouseSecret.name` uses that database and disables bundled storage. Keep your database name and retention policy. For GitOps tools that render Helm without cluster access, supply both existing secrets so rendering cannot regenerate credentials
Normal Helm upgrades reuse the generated credentials. Secrets are retained on uninstall, and the ClickHouse volume is retained by Kubernetes. Back them up together. Treat changing the database, storage class, or secret reference as an infrastructure change, not a routine version update
After deployment, open **Lens**. If it was already open, click **Check setup**. The setup section moves to your framework and tracing key when Lens is reachable and storage is ready. Investigation setup asks for the analysis model and budget; the installed service connects automatically
## Upgrade
Upgrade LiteLLM and Lens from the same source commit and release identity. For a coordinated published release, use its matching worker version; `deploy/lens/stack.yaml` starts LiteLLM, Lens, PostgreSQL, and ClickHouse for new installations. Standalone images remain available. Publishing an image does not update running containers
Keep the same databases, encryption keys, shared service secret, and public ingestion URL. Pause scheduled investigations and finish or cancel active runs, update both images through your usual deployment process, then check ingestion and run an investigation before resuming schedules. Do not run `docker compose down -v`
## Development
`make lens-dev` starts LiteLLM, Lens, and the hot-reload dashboard. Set `LENS_DEV_PROXY_PORT` and `LENS_DEV_UI_PORT` to change the local ports. For containers, pass the same release identity to both builds. Unversioned or incompatible workers are refused before claiming work
## Configure a lens
Choose agent runs, individual LLM requests, or both. The matching-activity preview updates as you choose an application (the recorded OpenTelemetry service.name) or, for request activity, a LiteLLM model group and add metadata conditions. It shows run names, timestamps, and trace IDs; open a run to inspect its original steps before starting analysis. Suggestions come from up to 100 recent executions and may not include every recorded attribute. You can enter other exact keys and values. Leave service and filters blank for all activity your account can access. Filters are exact key/value matches, combined with AND. Trace filters match span or resource attributes on the same span. Request filters match logged metadata, including caller metadata stored under `requester_metadata`; `tag=value` matches request tags. `swarm=research` works only if your instrumentation records that attribute
Describe how the agent should behave and optionally add specific checks. Select the lookback window, team and metadata, then choose the percentage to review and an optional maximum. **100% with no maximum selects every matching run**. The preview pages through all matching activity and lets you select particular runs. Percentage sampling uses a stable hash order, rounds up, and applies the optional maximum after the percentage
Choose your analysis model, parallelism and monthly budget. Parallelism controls simultaneous model calls, not the number of runs selected. New lenses run once by default. Turn on monitoring to repeat the same setup at a custom interval. **Run now** uses the same saved settings immediately, including the same lookback window and sampling. Each scan recalculates the window and reuses completed reviews when the selected trace content, expected behavior, enabled checks and analysis model are unchanged. Budget, name and schedule edits preserve reuse. Duplicate a lens when you want a separate investigation without changing an existing monitor
Pausing stops future scheduled scans; cancel the active scan separately if needed. The worker polls every 2 to 15 seconds, backing off while idle; creating a lens or clicking Run now queues a scan, and due schedules are queued when the worker polls. Scans for the same lens never overlap, and its next interval starts after completion. Closing the browser does not stop the worker. A running scan retains its analysis settings and selected execution IDs across retries. Budget edits apply to subsequent model calls, including those in an active scan
## Read the results
Needs attention shows issues, highest priority first. Patterns contains useful trends and successful behavior that may not need a fix. Each finding starts with a short explanation and a next step when useful. Expand the limitations for uncertainty and counterexamples. Evidence is grouped by run and collapsed until you need it; each quote opens the original step
Use the **Investigation run** selector or **History** to reopen previous results. Each run keeps its new or updated findings, settings, selected traces, coverage and cost. An unchanged rerun adds no findings; choose **All accumulated findings** to see saved findings across runs. The **Agent traces** tab lists the selected sample, including traces without an observed issue and traces with insufficient evidence. The tab is called **LLM requests** or **Traces and requests** for those activity types. Findings distinguish distinct affected traces from contributing investigation runs. Counterexamples remain visible as evidence without increasing the affected count. Merged finding links continue to resolve to the retained finding
Enter an explanation under **What should Lens remember?** and choose **This is expected** to dismiss expected behavior, **Mark resolved** after fixing an issue, or **Reopen** to reopen a resolved issue. These actions save the feedback together with the status; typing feedback alone neither saves it nor resolves the finding. Feedback informs later analysis and reconciliation without invalidating completed reviews. It stays with the finding when evidence recurs, does not alter historical evidence, and does not exempt different problems
## What a scan does
The proxy selects executions received or updated within the configured lookback window, with a two-minute settling period. Older rows without receipt timestamps use execution end time. Overlapping scans do not increment a finding's occurrence count for the same execution ID
A trace is spans sharing a trace ID within one team, not an automatically reconstructed conversation session. Requests are individual LLM calls. When both sources are enabled, requests correlated to a recorded span by response ID are excluded to reduce double counting
The worker reads complete selected trace content to compute fingerprints before making paid model calls. Completed review checkpoints are reused only within the same lens when both the complete content and investigation criteria match. Criteria are the expected behavior, enabled check IDs and instructions, and analysis model. Lens fingerprints these values rather than using the time of an unrelated settings edit. Scope and sampling changes preserve matching reviews for traces selected again; duplicating a lens starts an independent set of reviews. Initial reviews are confined to their assigned trace and retain observations, cited excerpts and metadata. Reviewers use catalog, read, search and optional Python tools; Python receives selected evidence as streamed input. Pending observation batches are grouped in parallel and investigated against the original evidence, then reconciled with saved findings. A recurring cause extends its existing finding, preserving feedback, earlier evidence and contributing run history
If every selected review is already incorporated into findings, the run completes with a reuse count, zero model calls and zero analysis cost, including when the monthly budget is exhausted. The run remains in history. A partially failed run preserves completed review checkpoints; pending grouping or investigation can still need paid model calls even when all selected trace reviews are reused. A trace without a complete matching checkpoint needs a review. Live progress distinguishes reviews eligible for reuse from reviews actually recorded; failed or cancelled runs report only the reuse they completed
There is no fixed total run, span, candidate or investigation-turn cutoff. Agents can replace their active conversation with working notes. If a request exceeds the configured model's context window, the worker compacts the conversation automatically and resumes with references to its archived tool history. Original evidence remains accessible through the gateway while it is available and retained. Tool results and working notes remain accessible during the investigation; character ranges make even a single oversized result readable in pieces. A review reports an error if the task or its replacement notes cannot fit. Context windows, the configured budget, worker resources and recorded evidence still bound practical work. The investigator has no browsing, code-editing or production-action tools
The live review drawer shows loading, trace review, parallel grouping, reconciliation and candidate investigation. It reports current model and tool operations, including context compaction, and retains tool-call counts on completed trace reviews. These counts describe attempted calls, not successful executions. This progress channel contains operation metadata, not Python code or tool output. Preliminary observations remain separate from final findings and their validated evidence
Each model response must match its JSON schema. A malformed response gets one repair attempt through the same budget controls. A session review that remains invalid or cannot fit marks that execution unassessable while other reviews continue. Broken evidence pagination or missing content pages return tool errors so the agent can inspect narrower spans or other evidence. Unreadable citations receive repair feedback. Verified excerpts remain available without fetching their source again. The affected source counts as partial, including failures discovered during later investigations, while the reviewer owns its assessment. Findings are published only after comparison with each other and saved findings finishes. If analysis stops before that comparison completes, completed trace reviews and their evidence remain saved for reuse, and the run retains its assessments and error. A later run can retry grouping and investigation. Runs with useful completed assessments or reconciled findings show partial results; total failures are marked failed. Transport errors, cancellation and budget exhaustion stop further analysis. Both the worker and proxy validate quoted evidence against original content. Per-run issue assessments follow supporting citations, including evidence found by another run's reviewer; counterexamples do not mark a run affected. Findings retain exact quotes and open the source trace or request. Resolve a finding after a fix, or dismiss it with a reason. A resolved finding reopens when new execution IDs support the same pattern; dismissed findings remain dismissed
Coverage distinguishes eligible, sampled, reviewed, partial, and unassessable executions. Findings describe observations in the sample, not population-wide success rates or proven causes. A root span does not prove that a trace contains every expected span. Long, missing, redacted, or expired content limits the conclusions
## Operations and limits
PostgreSQL stores configurations, findings and all scan history, returned in pages of 50 jobs. Workers claim jobs with optimistic concurrency and a five-minute lease, renewed every 30 seconds. A disconnected job can be reclaimed up to three times. Cancellation stops subsequent work; a model call already in flight may finish and incur cost
Lens checks which selected traces have reusable reviews before requesting model budget. Reuse needs no model call or reservation. New trace reviews and unfinished grouping or investigation can incur cost
Before each model call, Lens reserves a conservative allowance based on the input and permitted output. The summary separates settled monthly spend, unexpired reservations and available budget. With a $100 limit, $45 spent and $10 reserved, $45 is available for additional calls. Successful calls settle to recorded cost and release unused capacity. Failed or timed-out requests release their hold; abandoned holds expire after the proxy request timeout plus a grace period
Calls wait when concurrent reservations temporarily hold the remaining capacity. Waiting and model execution share the proxy's request timeout. If one request's allowance exceeds the unspent monthly budget, the error reports what the request needs and what remains. Reduce the deployment's output allowance or increase the limit. Paid analysis stops when the monthly limit is spent; completed reviews can still be reused. The monthly budget renews on the UTC calendar month
The assigned virtual key has independent budgets, model permissions and rate limits. Several investigations can share that key, so its limit can stop analysis even when one lens has budget left. Every worker needs a billing key assigned through worker setup or **Settings**. Analysis spend appears under that key in **Virtual Keys** and normal request logs, with Lens, run and worker IDs in request metadata. Analysis prompts and responses are redacted from spend logs; source traces and findings remain available through the administrator-only Lens API. Terminal budget, authentication or transport failures stop the scan after applicable retries and preserve completed checkpoints
Lens requires ClickHouse for both sources. It does not reconstruct sessions from unrelated trace IDs, guarantee exhaustive reviews, or automatically fix agent code. Trace contents can change as late spans arrive, even though a job's selected IDs are fixed. Changed content requires a matching review before reuse. Findings should be reviewed by a person before acting on them
## API access
The UI and API use the same scan lifecycle. Authenticate with a proxy administrator credential for writes, or a proxy-admin viewer credential for reads. Worker credentials are only for worker operations
```bash
curl "$LITELLM_URL/lens" -H "Authorization: Bearer $LITELLM_API_KEY" \
-H 'Content-Type: application/json' -d '{
"name": "Research quality", "model": "your-model-alias",
"context": "Answer the requested question using cited, retrieved evidence.",
"source": "traces", "lookback_hours": 24,
"sample_percent": 100, "sample_size": null, "concurrency": 8,
"enabled": true, "interval_minutes": 1440, "monthly_budget": 50
}'
curl "$LITELLM_URL/lens/$LENS_ID/runs" -X POST \
-H "Authorization: Bearer $LITELLM_API_KEY" -H 'Content-Type: application/json' -d '{}'
curl "$LITELLM_URL/lens/$LENS_ID/runs?offset=0" -H "Authorization: Bearer $LITELLM_API_KEY"
curl "$LITELLM_URL/lens/$LENS_ID/runs/$BATCH_ID" -H "Authorization: Bearer $LITELLM_API_KEY"
```
Creation queues the first batch. Posting to `/lens/{id}/runs` queues another, or returns the existing active batch. The run response contains its ID under `jobs[0].id`. Poll the batch URL for status, findings and assessments. List responses omit large result payloads; request a batch to retrieve them. Supply an optional complete `settings` object on the runs POST for a one-off override; the saved lens stays unchanged. Selection accepts `team_id`, exact `filters`, and opaque `execution_ids` returned by `/lens/preview/sample`. Preview accepts `offset` and `as_of` to keep the time window fixed while paging. Feedback uses `PATCH /lens/{id}/findings/{finding_id}` with `status` and `reason`
## Local development
`make lens-dev ARGS=--seed` starts the full dev stack. The live dashboard is at `http://localhost:3000/ui/lens/`, with login at `http://localhost:3000/ui/login/`. Next.js forwards API requests to the proxy on port 4000, so login and navigation stay in the live UI and edits hot-reload
The default is Next.js dev with no production build (`LENS_DEV_BUILD_UI=0`). Set `LENS_DEV_BUILD_UI=1` when you also want a fresh static dashboard at `http://localhost:4000/ui/`. Build output goes to `.lens-dev/logs/ui-build.log`; a failed build stops startup. Both modes keep the live dashboard on port 3000. Startup checks the live login route before seeding and fails with the UI log path if Next.js exits. `LENS_DEV_STARTUP_TIMEOUT_SECONDS` controls startup readiness retries (default 300; `LENS_DEV_READINESS_REQUEST_TIMEOUT_SECONDS` caps each HTTP probe, default 5)
For local fixture data, run `make lens-dev ARGS=--seed`. Use `make lens-dev ARGS="--seed large"` for 2,000 fixture copies spread over the last 24 hours, about 860,000 spans with linked request logs, plus three long sessions of roughly 1,150, 9,200 and 92,000 spans in a single trace for drawer paging and the oversized read path. Their trace IDs are printed at the end. To seed a running stack without restarting it, use `make lens-dev ARGS="--seed-only --seed large --copies 100"`. Every profile replays one copy of every checked-in capture through authenticated `/v1/traces`, including failures, retries, streaming and multiple agent frameworks, and verifies linked spend totals through the proxy. Large seeds then copy that first copy inside ClickHouse and PostgreSQL with `INSERT ... SELECT`, rewriting trace, span and call IDs so each copy keeps its own spend, and verify the last copy through the proxy
Seeds append fresh IDs on every invocation and spread copies over recent timestamps. Restarts without `SEED` do not add data. Lens excludes activity received in the last two minutes, so wait two minutes after seeding before checking investigation previews. `LENS_DEV_SEED_COPIES` overrides total copies. Large seeds test data volume and pagination, rather than concurrent ingestion throughput or review accuracy. They can use substantial disk space; adjust `--copies` for your machine. Seeding expects the generated local tracing configuration. The old `run_tracing_proxy_local.sh --seed` command forwards to Lens dev, using its ports and saved master key
The Rust receiver bounds each upload and its decompressed body to 16 MiB and permits two ingestion requests at once per replica. Exporters should split large batches and retry backpressure. `LENS_DEV_SEED_COPIES` and `LENS_DEV_SEED_TIMEOUT_SECONDS` control the seeder; the receiver's limits are compiled into the service
## Quality evaluation
Run the checked-in cases against a configured real model. Expected labels are used only for scoring, never passed to the model. Dev and held-out cases include missing outcomes, failed tools, recovery, handoffs, unsupported claims, repeated work, long evidence and prompt injection. The background option adds clean arithmetic traces to test rare-issue discovery at scale; those repeated synthetic cases do not establish accuracy on every production workload
```bash
cargo build --manifest-path litellm-rust/Cargo.toml -p litellm-lens --example worker_once --locked
python -m tests.proxy_behavior.lens.evaluate --api-base "$LITELLM_URL" \
--model your-model-alias --split all --background 1000 --concurrency 16 \
--output /tmp/lens-quality.json
```
Set `LITELLM_API_KEY` privately. This makes paid model calls. Inspect missed and unexpected per-run labels, final findings and coverage; do not equate a passing dataset with guaranteed detection on arbitrary traces
The default workspace retrieves trace content on demand. Python calls have temporary scratch space that is removed after execution. The Docker command supplies a writable temporary mount while keeping the application filesystem read-only
To check that accepted behavior stays accepted without hiding new problems, run the evaluator with `--dataset tests/proxy_behavior/lens/feedback_cases.json`. Reports include elapsed time, model call count, reported cost when the proxy provides it, missed checks, unexpected checks, and inconclusive candidates
## Release compatibility
Gateway and worker builds carry the same `LITELLM_RELEASE_TAG`. A worker announces its release and protocol before claiming an investigation. A mismatch returns HTTP 409 with the required image, leaving queued investigations untouched
PostgreSQL stores complete review checkpoints in `LiteLLM_LensReview`, alongside Lens records and run history. Schema migrations preserve saved investigations, findings, history, worker credentials and billing assignments without rewriting stored Lens records
Drain active scans, stop workers, back up the database, and deploy all gateway replicas as a coordinated replacement or traffic cutover. Keep traffic paused until every gateway replica uses the selected build and its migrations have completed. Mixed gateway versions sharing Lens data are not supported because every replica must understand the stored records. Pausing workers alone does not prevent dashboard or API writes. Recreate workers with the matching image and their existing tokens, then resume traffic and schedules. Scanning waits until a compatible worker connects
Use the shipped migration history for databases containing Lens data. The schema guard stops `--use_prisma_db_push` before changes if it detects Lens tables that require renaming; run the shipped rename migration against the configured schema before retrying. Fresh databases and databases with the current table names can use database push
A rollback to a gateway that cannot read saved Lens records requires restoring a compatible database backup. Database push from such a build can also remove the review table. Test recovery on a separate database and account for all gateway data written after the backup
The dashboard reads its image from the running gateway. `LENS_WORKER_IMAGE` overrides the registry/image for private deployments. Set an explicit `LENS_WORKER_IMAGE` for worker-only Compose. Verify that the image exists and matches the gateway before deploying it
For source development, use `make lens-dev`, which gives the proxy and source worker the same commit identity. For custom containers, build both from the same checkout with `--build-arg LITELLM_RELEASE_TAG=sha-$(git rev-parse HEAD)` and set the proxy's `LENS_WORKER_IMAGE` to the worker image you built. An unlabelled custom build refuses worker setup and claims instead of guessing from the Python package version. Normal package-index installations use their installed release version
The hourly development pipeline pins all component images to the same selected commit and publishes its chart only after every build and worker smoke test succeeds. The public commit-tagged worker workflow publishes to `ghcr.io/berriai/litellm-lens-worker-dev` on Lens-related changes, so an arbitrary `main` commit may require building your own pair; do not substitute the newest available worker
## Worker dependencies
The service builds from the workspace Cargo.lock with a pinned Rust toolchain and a digest-pinned Wolfi runtime. It has no Python package dependencies. CPython and libseccomp support the confined calculation tool. CI builds, runs, and scans native amd64 and arm64 images
## Python analysis boundary
The `python` tool runs ordinary CPython with the standard library in a fresh child process inside the existing worker container. It receives the selected evidence as `data` over stdin and has its own temporary working directory. It creates no additional container or service. Read and search tools remain available independently of Python
The native worker image builds a syscall policy with libseccomp and includes the full `setpriv` launcher. Each child starts with no inherited worker secrets or open worker files, isolated Python startup, Landlock filesystem restrictions and a default-deny seccomp filter. It can read the Python runtime and its own scratch files. Worker source, installed worker packages, other jobs' files and `/proc` contents are unavailable. Network sockets, child processes, cross-process memory operations, signals to other processes and filesystem metadata mutation are denied, including calls made through `ctypes`. Some metadata inspection, such as `stat`, `access` and `readlink` of known paths, remains possible
Python execution requires a native Linux worker with Landlock ABI 3 or later and seccomp filtering. Build the image for the host architecture. Missing policy files, an incompatible kernel, or an unsupported host such as a macOS source worker returns a clear tool error. There is no unrestricted execution fallback. Keep the container's non-root user, dropped capabilities, no-new-privileges setting, read-only root and writable temporary mount
The worker permits two Python children at once across all investigations. Queued calls consume no child process or scratch directory; cancelling a queued call does not start it. Model, read and search concurrency are separate
| Per-call resource | Default |
| --- | --- |
| Elapsed execution time | 60 seconds |
| CPU time | 30 seconds |
| Process address space | 512 MiB |
| Captured stdout or stderr | 4 MiB per stream |
| Individual scratch file size | 16 MiB |
| Monitored scratch storage | 64 MiB |
| Monitored scratch entries | 2,048 |
| Scratch directory depth | 128 |
| Open file descriptors | 64 |
Evidence is streamed from gateway pages into the confined child without building another complete selection in worker memory. The child decodes the selected data under its memory limit before running the code. The execution wall clock starts after input delivery; gateway fetches keep their HTTP timeouts and remain cancellable. CPU, address-space and file-size limits apply during input decoding as well as computation. Scratch usage is monitored every 50 milliseconds, so a call can temporarily overshoot its scratch allowance. The worker's shared temporary mount supplies the hard aggregate storage ceiling, 1 GiB by default. Accounting includes unlinked open files and files retained only by memory mappings. A mapped scratch inode without an open descriptor or directory entry is conservatively charged at the individual file-size limit, which may overcount small files. Cancellation and limit failures kill and reap the child before removing its scratch directory
Results include `stdout`, `stderr`, `exit_code`, `error` and `output_complete`. Nonzero interpreter exits, confinement failures and resource failures set `error` and `output_complete=false`. Available traceback output is retained. An output-size failure delivers no partial stdout/stderr; the agent can narrow its computation and retry. A successful result retains all captured output without truncation
This is a process boundary sharing the worker's Linux kernel. The checked-in smoke test verifies useful Python operations, filesystem and process restrictions, raw syscall attempts, resource failures, mapping accounting, cleanup and cancellation in the actual image. Run it on the deployment's native architecture and kernel:
```bash
docker build --target smoke --build-arg LITELLM_RELEASE_TAG=lens-python-test \
-f deploy/lens/Dockerfile -t lens-worker:smoke .
docker run --rm --read-only --cap-drop ALL \
--security-opt no-new-privileges --network none \
--tmpfs /tmp:rw,noexec,nosuid,size=1g lens-worker:smoke
```
The smoke target runs the Rust sandbox integration tests. The production image contains neither Cargo nor the test executable

8
deploy/lens/compose.build.yaml
View file

@ -1,8 +0,0 @@
services:
lens-worker:
build:
context: ../..
dockerfile: deploy/lens/Dockerfile
args:
LITELLM_RELEASE_TAG: ${LITELLM_RELEASE_TAG:?Set the release tag used by the gateway}
image: litellm-lens-worker:local

21
deploy/lens/compose.yaml
View file

@ -1,21 +0,0 @@
services:
lens-worker:
image: ${LENS_WORKER_IMAGE:-${LITELLM_VERSION:+ghcr.io/berriai/litellm-lens-worker:v}${LITELLM_VERSION:-}}
environment:
LITELLM_URL: ${LITELLM_URL:?Set the URL reachable from this container}
LENS_WORKER_TOKEN: ${LENS_WORKER_TOKEN:-}
LITELLM_LENS_SERVICE_TOKEN: ${LITELLM_LENS_SERVICE_TOKEN:?Set the same secret on LiteLLM and Lens}
CLICKHOUSE_URL: ${CLICKHOUSE_URL:?Set the ClickHouse URL reachable from Lens}
CLICKHOUSE_DATABASE: ${CLICKHOUSE_DATABASE:-litellm}
AGENT_TRACING_RETENTION_DAYS: ${AGENT_TRACING_RETENTION_DAYS:-14}
ports:
- "127.0.0.1:${LENS_PORT:-4318}:4318"
mem_limit: 2g
cpus: 2
pids_limit: 64
restart: unless-stopped
read_only: true
tmpfs:
- /tmp:rw,noexec,nosuid,size=${LENS_WORKER_TMP_SIZE:-1g}
cap_drop: [ALL]
security_opt: [no-new-privileges:true]

7
deploy/lens/config.yaml
View file

@ -1,7 +0,0 @@
model_list: []
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
tracing:
store:
type: lens

73
deploy/lens/configure.py
View file

@ -1,73 +0,0 @@
from __future__ import annotations
import argparse
import os
import re
import secrets
import shlex
import sys
from pathlib import Path
from typing import Final
SECRET_NAMES: Final = (
"LITELLM_MASTER_KEY",
"LITELLM_SALT_KEY",
"LITELLM_LENS_SERVICE_TOKEN",
"POSTGRES_PASSWORD",
"CLICKHOUSE_PASSWORD",
)
def environment_content(path: Path) -> str:
if not path.exists():
return "".join(
f"{name}={'sk-' if name.endswith('KEY') else ''}{secrets.token_hex(32)}\n" for name in SECRET_NAMES
)
saved: Final = path.read_text().splitlines()
values: Final = dict(line.split("=", 1) for line in saved if "=" in line)
if any(not values.get(name) for name in SECRET_NAMES):
raise ValueError(f"{path} is incomplete. Restore your saved credentials before continuing")
return "\n".join(line for line in saved if not line.startswith("LITELLM_VERSION=")) + "\n"
def configure(path: Path, version: str) -> None:
release: Final = version.removeprefix("v")
if not re.fullmatch(r"[0-9]+\.[0-9]+\.[0-9]+(?:[-.][a-zA-Z0-9.-]+)?", release):
raise ValueError("Use a published release version, such as 1.82.0 or 1.82.0-nightly")
if path.is_symlink():
raise ValueError(f"Refusing to replace a symlink: {path}")
existing: Final = path.exists()
content: Final = environment_content(path)
temporary: Final = path.with_name(f".{path.name}.{secrets.token_hex(8)}")
descriptor: Final = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
try:
with os.fdopen(descriptor, "w") as output:
output.write(content + f"LITELLM_VERSION={release}\n")
if existing:
os.replace(temporary, path)
else:
os.link(temporary, path)
finally:
temporary.unlink(missing_ok=True)
def main() -> None:
parser: Final = argparse.ArgumentParser(description="Create or update the configuration for the Lens Compose stack")
parser.add_argument("--version", required=True, help="Published LiteLLM release; Lens uses the matching version")
parser.add_argument("--env-file", type=Path, default=Path(__file__).with_name(".env"))
arguments: Final = parser.parse_args()
try:
configure(arguments.env_file, arguments.version)
except (OSError, ValueError) as error:
parser.exit(1, f"Could not configure Lens: {error}\n")
sys.stdout.write(
f"Saved {arguments.env_file}. Existing keys and database passwords are preserved\n"
f"Start with: docker compose --env-file {shlex.quote(str(arguments.env_file))} "
"-f deploy/lens/stack.yaml up -d --wait\n"
"Open http://localhost:4000/ui/ and sign in as admin with LITELLM_MASTER_KEY from the saved file\n"
"Back up this file with your database volumes. Do not commit it\n"
)
if __name__ == "__main__":
main()

63
deploy/lens/python_policy.c
View file

@ -1,63 +0,0 @@
#include <errno.h>
#include <fcntl.h>
#include <seccomp.h>
#include <stdio.h>
#include <sys/ioctl.h>
#include <unistd.h>
static int allow(scmp_filter_ctx policy, const char *name)
{
int number = seccomp_syscall_resolve_name(name);
return number < 0 ? 0 : seccomp_rule_add(policy, SCMP_ACT_ALLOW, number, 0);
}
int main(int argc, char **argv)
{
const char *calls[] = {
"read", "write", "readv", "writev", "pread64", "pwrite64", "close", "close_range",
"open", "openat", "openat2", "fstat", "stat", "lstat", "newfstatat", "statx",
"lseek", "getdents", "getdents64", "access", "faccessat", "faccessat2",
"readlink", "readlinkat", "getcwd", "chdir", "fchdir", "statfs", "fstatfs",
"mkdir", "mkdirat", "rmdir", "unlink", "unlinkat", "rename", "renameat", "renameat2",
"link", "linkat", "symlink", "symlinkat", "truncate", "ftruncate", "fsync", "fdatasync",
"mmap", "mmap2", "mprotect", "munmap", "mremap", "madvise", "brk",
"rt_sigaction", "rt_sigprocmask", "rt_sigreturn", "rt_sigsuspend", "rt_sigtimedwait", "sigaltstack",
"getpid", "getppid", "gettid", "getuid", "geteuid", "getgid", "getegid", "getgroups",
"clock_gettime", "clock_getres", "clock_nanosleep", "gettimeofday", "time", "nanosleep",
"futex", "futex_time64", "set_tid_address", "set_robust_list", "rseq", "arch_prctl",
"sched_getaffinity", "sched_yield", "getrandom", "getrlimit", "setrlimit", "getrusage", "umask",
"dup", "dup2", "dup3", "pipe", "pipe2", "poll", "ppoll", "select", "pselect6",
"epoll_create", "epoll_create1", "epoll_ctl", "epoll_wait", "epoll_pwait", "epoll_pwait2",
"capget", "capset", "prctl", "landlock_create_ruleset", "landlock_add_rule", "landlock_restrict_self",
"execve", "exit", "exit_group", "uname", "sysinfo", "restart_syscall"
};
const int commands[] = {F_DUPFD, F_DUPFD_CLOEXEC, F_GETFD, F_SETFD, F_GETFL, F_GETLK, F_SETLK, F_SETLKW};
if (argc != 2) {
fputs("Usage: python-policy OUTPUT\n", stderr);
return 1;
}
scmp_filter_ctx policy = seccomp_init(SCMP_ACT_ERRNO(EPERM));
if (!policy)
return 1;
int result = 0;
for (size_t i = 0; i < sizeof(calls) / sizeof(calls[0]); i++)
result |= allow(policy, calls[i]);
result |= seccomp_rule_add(policy, SCMP_ACT_ALLOW, SCMP_SYS(prlimit64), 1, SCMP_A0(SCMP_CMP_EQ, 0));
for (size_t i = 0; i < sizeof(commands) / sizeof(commands[0]); i++)
result |= seccomp_rule_add(policy, SCMP_ACT_ALLOW, SCMP_SYS(fcntl), 1, SCMP_A1(SCMP_CMP_EQ, commands[i]));
result |= seccomp_rule_add(policy, SCMP_ACT_ALLOW, SCMP_SYS(fcntl), 2,
SCMP_A1(SCMP_CMP_EQ, F_SETFL), SCMP_A2(SCMP_CMP_MASKED_EQ, O_ASYNC, 0));
result |= seccomp_rule_add(policy, SCMP_ACT_ALLOW, SCMP_SYS(ioctl), 1, SCMP_A1(SCMP_CMP_EQ, FIOCLEX));
result |= seccomp_rule_add(policy, SCMP_ACT_ALLOW, SCMP_SYS(ioctl), 1, SCMP_A1(SCMP_CMP_EQ, FIONCLEX));
int output = open(argv[1], O_WRONLY | O_CREAT | O_TRUNC, 0444);
if (output < 0)
result = -1;
if (!result)
result = seccomp_export_bpf(policy, output);
if (output >= 0)
close(output);
seccomp_release(policy);
if (result)
fputs("Could not build the Python syscall policy\n", stderr);
return result ? 1 : 0;
}

41
deploy/lens/python_runtime.py
View file

@ -1,41 +0,0 @@
import json
import subprocess
import sys
import sysconfig
from itertools import chain
from pathlib import Path
from typing import Final
def dependencies(path: Path, loader: Path) -> tuple[Path, ...]:
result: Final = subprocess.run((str(loader), "--list", str(path)), capture_output=True, text=True, check=True)
if "not found" in result.stdout:
raise RuntimeError(f"Missing Python runtime library: {path}")
words: Final = tuple(result.stdout.split())
return tuple(Path(word).resolve() for word in words if word.startswith("/"))
def main() -> None:
stdlib: Final = Path(sysconfig.get_path("stdlib")).resolve()
executable: Final = Path(sys.executable).resolve()
loaders: Final = tuple(Path("/usr/lib").glob("ld-linux-*.so.*"))
if len(loaders) != 1:
raise RuntimeError("Expected one native glibc dynamic loader in the Lens worker image")
entries: Final = tuple(
path for path in stdlib.iterdir() if path.name not in ("site-packages", "dist-packages", "__pycache__")
)
extensions: Final = tuple((stdlib / "lib-dynload").glob("*.so"))
libraries: Final = frozenset(
chain.from_iterable(dependencies(binary, loaders[0]) for binary in (executable, *extensions))
)
manifest: Final = {
"executable": str(executable),
"directories": (str(stdlib),),
"read": tuple(sorted(str(path) for path in {*entries, *libraries})),
"execute": tuple(sorted(str(path) for path in (executable, *loaders))),
}
Path(sys.argv[1]).write_text(json.dumps(manifest), encoding="utf-8")
if __name__ == "__main__":
main()

40
deploy/lens/smoke.sh
View file

@ -1,40 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
image="${1:?pass the built image reference}"
release="${2:?pass the expected release tag}"
version="$(docker run --rm --network none --read-only --cap-drop ALL --security-opt no-new-privileges "$image" --version)"
test "$version" = "litellm-lens $release protocol=7"
container="$(docker run -d --network none --read-only --cap-drop ALL \
--security-opt no-new-privileges --pids-limit 64 --memory 2g --cpus 2 \
--tmpfs /tmp:rw,noexec,nosuid,size=256m \
-e LITELLM_URL=http://127.0.0.1:1 \
-e CLICKHOUSE_URL=http://127.0.0.1:1 \
-e LITELLM_LENS_SERVICE_TOKEN=isolated-runtime-smoke-secret-32-characters \
"$image")"
trap 'docker rm -f "$container" >/dev/null' EXIT
test "$(docker exec "$container" id -u)" = 65532
docker exec -i "$container" python3.13 -I -S - <<'PY'
import time
import urllib.error
import urllib.request
for attempt in range(50):
try:
with urllib.request.urlopen("http://127.0.0.1:4318/health/live", timeout=1) as response:
assert response.status == 200
break
except urllib.error.URLError:
if attempt == 49:
raise
time.sleep(0.1)
for path, expected in (("health/ready", 503), ("internal/status", 401)):
try:
urllib.request.urlopen(f"http://127.0.0.1:4318/{path}", timeout=1)
except urllib.error.HTTPError as error:
assert error.code == expected, (path, error.code)
else:
raise AssertionError(f"{path} should return {expected}")
print("Unprivileged Lens service remains live with unavailable dependencies")
PY

100
deploy/lens/stack.yaml
View file

@ -1,100 +0,0 @@
name: litellm-lens
services:
litellm:
image: ghcr.io/berriai/litellm:${LITELLM_VERSION:?Set LITELLM_VERSION to a published release, without the v prefix}
entrypoint:
- python3
- -c
- |
import os, sys
from urllib.parse import quote
postgres_password = quote(os.environ["POSTGRES_PASSWORD"], safe="")
os.environ["DATABASE_URL"] = f"postgresql://litellm:{postgres_password}@db:5432/litellm"
os.execv("docker/prod_entrypoint.sh", ["docker/prod_entrypoint.sh", *sys.argv[1:]])
command: ["--config", "/app/lens-config.yaml", "--port", "4000"]
environment:
LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY:?Set a strong master key}
LITELLM_SALT_KEY: ${LITELLM_SALT_KEY:?Set a permanent encryption key and keep it across upgrades}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Set a permanent database password}
STORE_MODEL_IN_DB: "True"
LITELLM_LENS_URL: http://lens-worker:4318
LITELLM_LENS_PUBLIC_URL: ${LITELLM_LENS_PUBLIC_URL:-http://localhost:4318}
LITELLM_LENS_SERVICE_TOKEN: ${LITELLM_LENS_SERVICE_TOKEN:?Set the shared Lens service secret}
LENS_WORKER_IMAGE: ghcr.io/berriai/litellm-lens-worker:v${LITELLM_VERSION}
volumes:
- ./config.yaml:/app/lens-config.yaml:ro
ports:
- "127.0.0.1:${LITELLM_PORT:-4000}:4000"
networks: [proxy, database]
depends_on:
db:
condition: service_healthy
restart: unless-stopped
lens-worker:
image: ghcr.io/berriai/litellm-lens-worker:v${LITELLM_VERSION}
environment:
LITELLM_URL: http://litellm:4000
LENS_WORKER_TOKEN: ${LENS_WORKER_TOKEN:-}
LITELLM_LENS_SERVICE_TOKEN: ${LITELLM_LENS_SERVICE_TOKEN}
CLICKHOUSE_HOST: clickhouse
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:?Set a permanent ClickHouse password}
CLICKHOUSE_DATABASE: ${CLICKHOUSE_DATABASE:-litellm}
AGENT_TRACING_RETENTION_DAYS: ${AGENT_TRACING_RETENTION_DAYS:-14}
depends_on: [litellm]
networks: [proxy, storage]
ports:
- "127.0.0.1:${LENS_PORT:-4318}:4318"
mem_limit: 2g
cpus: 2
pids_limit: 64
restart: unless-stopped
read_only: true
tmpfs:
- /tmp:rw,noexec,nosuid,size=${LENS_WORKER_TMP_SIZE:-1g}
cap_drop: [ALL]
security_opt: [no-new-privileges:true]
db:
image: postgres:16
environment:
POSTGRES_DB: litellm
POSTGRES_USER: litellm
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
networks: [database]
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U litellm -d litellm"]
interval: 5s
timeout: 5s
retries: 20
restart: unless-stopped
clickhouse:
image: clickhouse/clickhouse-server:26.9.6.6
environment:
CLICKHOUSE_USER: default
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD}
CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: "1"
volumes:
- clickhouse_data:/var/lib/clickhouse
healthcheck:
test: ["CMD", "clickhouse-client", "--user", "default", "--password", "${CLICKHOUSE_PASSWORD}", "--query", "SELECT 1"]
interval: 5s
timeout: 5s
retries: 20
restart: unless-stopped
networks: [storage]
networks:
proxy:
database:
internal: true
storage:
internal: true
volumes:
postgres_data:
clickhouse_data:

52
deploy/lens/test_configure.py
View file

@ -1,52 +0,0 @@
import tempfile
import unittest
from pathlib import Path
from configure import SECRET_NAMES, configure
class ComposeConfigurationTests(unittest.TestCase):
def test_restart_and_upgrade_preserve_private_credentials_and_custom_settings(self) -> None:
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / ".env"
configure(path, "v1.2.3")
original = dict(line.split("=", 1) for line in path.read_text().splitlines())
self.assertEqual(path.stat().st_mode & 0o777, 0o600)
self.assertEqual(len({original[name] for name in SECRET_NAMES}), len(SECRET_NAMES))
self.assertTrue(all(len(original[name]) >= 64 for name in SECRET_NAMES))
with path.open("a") as output:
output.write("LITELLM_LENS_PUBLIC_URL=https://traces.example/prefix\n")
configure(path, "1.2.3")
configure(path, "v1.2.4-nightly")
updated = dict(line.split("=", 1) for line in path.read_text().splitlines())
self.assertEqual({name: updated[name] for name in SECRET_NAMES}, {name: original[name] for name in SECRET_NAMES})
self.assertEqual(updated["LITELLM_VERSION"], "1.2.4-nightly")
self.assertEqual(updated["LITELLM_LENS_PUBLIC_URL"], "https://traces.example/prefix")
self.assertEqual(path.stat().st_mode & 0o777, 0o600)
def test_incomplete_configuration_is_never_replaced_with_new_database_passwords(self) -> None:
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / ".env"
original = "POSTGRES_PASSWORD=existing\n"
path.write_text(original)
with self.assertRaisesRegex(ValueError, "incomplete"):
configure(path, "1.2.3")
self.assertEqual(path.read_text(), original)
def test_invalid_release_and_symlink_leave_existing_files_untouched(self) -> None:
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / ".env"
target = Path(directory) / "saved"
target.write_text("preserve")
path.symlink_to(target)
with self.assertRaisesRegex(ValueError, "symlink"):
configure(path, "1.2.3")
self.assertEqual(target.read_text(), "preserve")
path.unlink()
with self.assertRaisesRegex(ValueError, "published release"):
configure(path, "1.2.3\nPOSTGRES_PASSWORD=replaced")
self.assertFalse(path.exists())
if __name__ == "__main__":
unittest.main()

1
docker/Dockerfile.database
View file

@ -39,6 +39,7 @@ ENV NEXT_TELEMETRY_DISABLED=1 \
WORKDIR /ui
COPY ui/litellm-dashboard/package.json ui/litellm-dashboard/package-lock.json ./
COPY ui/litellm-dashboard/vendor/ ./vendor/
RUN --mount=type=cache,target=/root/.npm npm ci --prefer-offline
COPY ui/litellm-dashboard/ ./

1
docker/Dockerfile.non_root
View file

@ -37,6 +37,7 @@ ENV NEXT_TELEMETRY_DISABLED=1 \
WORKDIR /ui
COPY ui/litellm-dashboard/package.json ui/litellm-dashboard/package-lock.json ./
COPY ui/litellm-dashboard/vendor/ ./vendor/
RUN --mount=type=cache,target=/root/.npm npm ci --prefer-offline
COPY ui/litellm-dashboard/ ./

46
docker/docker-compose.tracing.yml
View file

@ -13,11 +13,11 @@ services:
LITELLM_SALT_KEY: sk-local-tracing-salt-key
DATABASE_URL: postgresql://litellm:litellm@db:5432/litellm
STORE_MODEL_IN_DB: "True"
LITELLM_LENS_URL: http://lens-worker:4318
LITELLM_LENS_PUBLIC_URL: http://localhost:4318
LITELLM_LENS_URL: ${LITELLM_LENS_URL:?set the Lens URL reachable from the gateway container}
LITELLM_LENS_PUBLIC_URL: ${LITELLM_LENS_PUBLIC_URL:?set the Lens URL reachable from your browser and agents}
LITELLM_LENS_SERVICE_TOKEN: ${LITELLM_LENS_SERVICE_TOKEN:?set LITELLM_LENS_SERVICE_TOKEN}
LENS_GATEWAY_SECRET: ${LENS_GATEWAY_SECRET:?set the same signing secret on Lens and the gateway}
OPENAI_API_KEY: ${OPENAI_API_KEY:-}
LENS_WORKER_IMAGE: ${LENS_WORKER_IMAGE:-}
volumes:
- ./tracing-config.yaml:/app/tracing-config.yaml:ro
ports:
@ -26,29 +26,6 @@ services:
db:
condition: service_healthy
lens-worker:
build:
context: ..
dockerfile: deploy/lens/Dockerfile
args:
LITELLM_RELEASE_TAG: ${LITELLM_RELEASE_TAG:?set LITELLM_RELEASE_TAG to the source commit}
environment:
LITELLM_URL: http://litellm:4000
LITELLM_LENS_SERVICE_TOKEN: ${LITELLM_LENS_SERVICE_TOKEN:?set LITELLM_LENS_SERVICE_TOKEN}
CLICKHOUSE_URL: http://default:local-tracing@clickhouse:8123
CLICKHOUSE_DATABASE: litellm
ports:
- "127.0.0.1:4318:4318"
read_only: true
cap_drop: [ALL]
security_opt: [no-new-privileges:true]
tmpfs:
- /tmp:rw,noexec,nosuid,nodev,size=${LENS_WORKER_TMP_SIZE:-1g},mode=1777
mem_limit: 2g
cpus: 2
pids_limit: 64
restart: unless-stopped
db:
image: postgres:16
environment:
@ -65,22 +42,5 @@ services:
timeout: 5s
retries: 10
clickhouse:
image: clickhouse/clickhouse-server:26.9.6.6
environment:
CLICKHOUSE_USER: default
CLICKHOUSE_PASSWORD: local-tracing
CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: "1"
volumes:
- clickhouse_data:/var/lib/clickhouse
ports:
- "127.0.0.1:18123:8123"
healthcheck:
test: ["CMD", "clickhouse-client", "--user", "default", "--password", "local-tracing", "--query", "SELECT 1"]
interval: 5s
timeout: 5s
retries: 20
volumes:
postgres_data:
clickhouse_data:

72
enterprise/litellm_enterprise/proxy/management_endpoints/project_endpoints.py
View file

@ -12,7 +12,7 @@ Endpoints for /project operations
import json
from collections.abc import Mapping, Sequence
from typing import TYPE_CHECKING, Final
from typing import TYPE_CHECKING, Final, cast
from fastapi import APIRouter, Depends, HTTPException, Request
from pydantic import TypeAdapter
@ -31,6 +31,7 @@ from litellm.proxy.management_helpers.utils import (
management_endpoint_wrapper,
)
from litellm.proxy.utils import PrismaClient, handle_exception_on_proxy
from litellm.repositories.base_repository import record_to_dict
from litellm.repositories.budget_repository import BudgetRepository
from litellm.repositories.object_permission_repository import ObjectPermissionRepository
from litellm.repositories.prisma_protocols import TableActions
@ -57,6 +58,15 @@ def _project_table(prisma_client: PrismaClient) -> TableActions["prisma_models.L
return ProjectRepository(prisma_client).table
def _writer_project_table(
prisma_client: PrismaClient,
) -> TableActions["prisma_models.LiteLLM_ProjectTable"]:
return cast( # cast-ok: writer_db exposes generated Prisma tables through a dynamic wrapper
"TableActions[prisma_models.LiteLLM_ProjectTable]",
prisma_client.writer_db.litellm_projecttable,
)
def _verification_token_table(
prisma_client: PrismaClient,
) -> TableActions["prisma_models.LiteLLM_VerificationToken"]:
@ -824,6 +834,38 @@ async def update_project(
**({"max_budget": None} if "max_budget" in data.model_fields_set and data.max_budget is None else {}),
}
object_permission_data: Final = update_data.pop("object_permission", None)
object_permission_payload: Final = (
_OBJECT_PERMISSION_PAYLOAD.validate_python(object_permission_data) if object_permission_data else None
)
if data.team_id is not None:
current_project_record: Final = await _writer_project_table(prisma_client).find_unique(
where={"project_id": data.project_id}
)
current_project: Final = (
LiteLLM_ProjectTable.model_validate(record_to_dict(current_project_record))
if current_project_record is not None
else None
)
if current_project is not None and data.team_id != current_project.team_id:
mismatched_key_count: Final = await prisma_client.writer_db.litellm_verificationtoken.count(
where={
"project_id": data.project_id,
"OR": [{"team_id": {"not": data.team_id}}, {"team_id": None}],
}
)
if mismatched_key_count > 0:
raise ProxyException(
message=(
f"Project {data.project_id} has {mismatched_key_count} key(s) that do not belong to "
f"team {data.team_id}. Detach or delete them before moving the project."
),
type="bad_request",
code=400,
param="team_id",
)
if budget_updates and existing_project.budget_id:
# Update existing budget
await _budget_table(prisma_client).update(
@ -837,23 +879,17 @@ async def update_project(
for field in budget_updates.keys():
update_data.pop(field, None)
# Handle object permissions
if "object_permission" in update_data:
object_permission_data = update_data.pop("object_permission")
if object_permission_data:
object_permission_payload: Final = _OBJECT_PERMISSION_PAYLOAD.validate_python(object_permission_data)
if existing_project.object_permission_id:
# Update existing permission
await _object_permission_table(prisma_client).update(
where={"object_permission_id": existing_project.object_permission_id},
data=object_permission_payload,
)
else:
# Create new permission
created_permission: Final = await _object_permission_table(prisma_client).create(
data=object_permission_payload,
)
update_data["object_permission_id"] = created_permission.object_permission_id
if object_permission_payload is not None:
if existing_project.object_permission_id:
await _object_permission_table(prisma_client).update(
where={"object_permission_id": existing_project.object_permission_id},
data=object_permission_payload,
)
else:
created_permission: Final = await _object_permission_table(prisma_client).create(
data=object_permission_payload,
)
update_data["object_permission_id"] = created_permission.object_permission_id
# Handle metadata fields
for field in LiteLLM_ManagementEndpoint_MetadataFields:

7
helm/litellm-helm/Chart.lock
View file

@ -5,5 +5,8 @@ dependencies:
- name: redis
repository: oci://registry-1.docker.io/bitnamicharts
version: 18.19.1
digest: sha256:38962e231f6596b93f82a8412bbe4cf5de696caecf5775dfbbd163383eb1c009
generated: "2026-07-28T10:21:22.511401-07:00"
- name: lens
repository: oci://ghcr.io/berriai/charts
version: 0.1.0-dev.0
digest: sha256:16b92c3d7fc74632e8aa11e602c136d273c774ed39b2fe970b80273e0570898b
generated: '2026-10-09T04:18:10.643228000Z'

3
helm/litellm-helm/Chart.yaml
View file

@ -39,3 +39,6 @@ dependencies:
version: "18.19.1"
repository: oci://registry-1.docker.io/bitnamicharts
condition: redis.enabled
- name: lens
version: 0.1.0-dev.0
repository: oci://ghcr.io/berriai/charts

6
helm/litellm-helm/README.md
View file

@ -225,3 +225,9 @@ At the time of writing, the Admin UI is unable to add models. This is because
it would need to update the `config.yaml` file which is a exposed ConfigMap, and
therefore, read-only. This is a limitation of this helm chart, not the Admin UI
itself.
## Connect Lens
`lensWorker.mode` selects `disabled`, `bundled` or `external`. Bundled mode generates separate service and identity-signing credentials. External mode requires `lensWorker.gateway.secretName` and `lensWorker.serviceTokenSecret.name`; their keys default to `gateway-secret` and `service-token`. Provision distinct values matching the external Lens deployment
Follow the [Lens Helm connection guide](https://github.com/BerriAI/lens/blob/main/helm/lens/README.md#connect-a-gateway) for complete values, verification, GitOps credential requirements and existing-data migration. Source-chart installation requires a built Lens image until the first signed Lens release is published

BIN
helm/litellm-helm/charts/lens-0.1.0-dev.0.tgz Normal file
View file

Binary file not shown.

78
helm/litellm-helm/templates/_helpers.tpl
View file

@ -323,21 +323,7 @@ through an emptyDir. Empty when the sidecar is off or uses 127.0.0.1 TCP.
{{- end -}}
{{- define "litellm.lensWorker.image" -}}
{{- if .Values.lensWorker.image.digest -}}
{{- if not (regexMatch "^sha256:[0-9a-f]{64}$" .Values.lensWorker.image.digest) -}}
{{- fail "lensWorker.image.digest must be sha256 followed by 64 lowercase hex characters" -}}
{{- end -}}
{{- printf "%s@%s" .Values.lensWorker.image.repository .Values.lensWorker.image.digest -}}
{{- else -}}
{{- $backendTag := .Values.image.tag | default .Chart.AppVersion -}}
{{- $releaseTag := ternary (printf "v%s" $backendTag) $backendTag (regexMatch "^[0-9]" $backendTag) -}}
{{- $tag := .Values.lensWorker.image.tag | default $releaseTag -}}
{{- $repository := .Values.lensWorker.image.repository -}}
{{- if and (hasPrefix "sha-" $tag) (eq $repository "ghcr.io/berriai/litellm-lens-worker") -}}
{{- $repository = "ghcr.io/berriai/litellm-lens-worker-dev" -}}
{{- end -}}
{{- printf "%s:%s" $repository $tag -}}
{{- end -}}
{{- include "lens.image" (dict "Values" .Values.lensWorker "Chart" .Subcharts.lens.Chart) -}}
{{- end -}}
{{- define "litellm.gateway.collectorSocketDir" -}}
@ -376,13 +362,19 @@ shutdown drain window.
{{- .Values.lensWorker.serviceTokenSecret.name | default (printf "%s-lens-service" (include "litellm.fullname" .)) -}}
{{- end -}}
{{- define "litellm.lensWorker.gatewaySecretName" -}}
{{- .Values.lensWorker.gateway.secretName | default (printf "%s-lens-gateway" (include "litellm.fullname" . | trunc 50 | trimSuffix "-")) -}}
{{- end -}}
{{- define "litellm.lensWorker.bundledClickhouse" -}}
{{- if and .Values.lensWorker.enabled .Values.lensWorker.clickhouse.enabled (not .Values.lensWorker.clickhouseSecret.name) -}}true{{- end -}}
{{- if and (eq (include "litellm.lens.mode" .) "bundled") .Values.lensWorker.clickhouse.enabled (not .Values.lensWorker.clickhouseSecret.name) -}}true{{- end -}}
{{- end -}}
{{- define "litellm.lensWorker.publicUrl" -}}
{{- if .Values.lensWorker.publicUrl -}}
{{- .Values.lensWorker.publicUrl -}}
{{- else if eq (include "litellm.lens.mode" .) "external" -}}
{{- fail "lensWorker.publicUrl is required for external Lens" -}}
{{- else if .Values.lensWorker.ingress.enabled -}}
{{- $tls := or (not (empty .Values.lensWorker.ingress.tls)) (hasKey .Values.lensWorker.ingress.annotations "alb.ingress.kubernetes.io/certificate-arn") -}}
{{- printf "%s://%s" (ternary "https" "http" $tls) (required "lensWorker.ingress.host is required" .Values.lensWorker.ingress.host) -}}
@ -398,3 +390,57 @@ shutdown drain window.
{{- define "litellm.lensWorker.clickhouseName" -}}
{{- printf "%s-lens-clickhouse" (include "litellm.fullname" . | trunc 47 | trimSuffix "-") -}}
{{- end -}}
{{- define "litellm.lensConnectionEnv" -}}
{{- $mode := include "litellm.lens.mode" . -}}
{{- if ne $mode "disabled" }}
{{- if and (eq $mode "external") (not .Values.lensWorker.serviceTokenSecret.name) -}}
{{- fail "lensWorker.serviceTokenSecret.name is required for external Lens" -}}
{{- end }}
{{- if and (eq $mode "external") (not .Values.lensWorker.gateway.secretName) -}}
{{- fail "lensWorker.gateway.secretName is required for external Lens" -}}
{{- end -}}
- name: LITELLM_LENS_URL
value: {{ if eq $mode "external" }}{{ required "lensWorker.externalUrl is required for external Lens" .Values.lensWorker.externalUrl | quote }}{{ else }}{{ printf "http://%s-lens-worker:%v" (include "litellm.fullname" .) .Values.lensWorker.service.port | quote }}{{ end }}
- name: LITELLM_LENS_PUBLIC_URL
value: {{ include "litellm.lensWorker.publicUrl" . | quote }}
- name: LENS_GATEWAY_SECRET
valueFrom:
secretKeyRef:
name: {{ include "litellm.lensWorker.gatewaySecretName" . | quote }}
key: {{ .Values.lensWorker.gateway.secretKey | quote }}
- name: LITELLM_LENS_SERVICE_TOKEN
valueFrom:
secretKeyRef:
name: {{ include "litellm.lensWorker.serviceTokenSecretName" . | quote }}
key: {{ .Values.lensWorker.serviceTokenSecret.key | quote }}
{{- end }}
{{- end -}}
{{- define "litellm.lens.mode" -}}
{{- $mode := .Values.lensWorker.mode | default (ternary "bundled" "disabled" .Values.lensWorker.enabled) -}}
{{- if not (has $mode (list "bundled" "external" "disabled")) -}}
{{- fail "lensWorker.mode must be bundled, external or disabled" -}}
{{- end -}}
{{- $mode -}}
{{- end -}}
{{- define "litellm.lens.render" -}}
{{- $root := .root -}}
{{- $values := mergeOverwrite (deepCopy $root.Subcharts.lens.Values) (deepCopy $root.Values.lensWorker) -}}
{{- $_ := set $values "fullnameOverride" (printf "%s-lens-worker" (include "litellm.fullname" $root)) -}}
{{- $_ := set $values "component" "lens-worker" -}}
{{- $_ := set $values "nameOverride" (printf "%s-lens-worker" (include "litellm.name" $root | trunc 51 | trimSuffix "-")) -}}
{{- $_ := set $values "imagePullSecrets" $root.Values.imagePullSecrets -}}
{{- $_ := set $values.gateway "enabled" true -}}
{{- $_ := set $values.gateway "generatedName" (include "litellm.lensWorker.gatewaySecretName" $root) -}}
{{- $_ := set $values.clickhouse "nameOverride" (include "litellm.lensWorker.clickhouseName" $root) -}}
{{- $_ := set $values.serviceTokenSecret "generatedName" (include "litellm.lensWorker.serviceTokenSecretName" $root) -}}
{{- $_ := set $values "publicUrl" ($root.Values.lensWorker.standaloneUrl | default $root.Subcharts.lens.Values.publicUrl) -}}
{{- if and (eq .resource "deployment") (or $root.Values.lensWorker.publicUrl $root.Values.lensWorker.ingress.enabled $root.Values.ingress.enabled) -}}
{{- $ingestion := include "litellm.lensWorker.publicUrl" $root -}}
{{- $_ := set $values "ingestionUrl" $ingestion -}}
{{- $_ := set $values "publicUrl" ($root.Values.lensWorker.standaloneUrl | default (trimSuffix "/lens-ingest" $ingestion)) -}}
{{- end -}}
{{- include (printf "lens.%s" .resource) (dict "Values" $values "Release" $root.Release "Chart" $root.Subcharts.lens.Chart "Capabilities" $root.Capabilities) -}}
{{- end -}}

12
helm/litellm-helm/templates/deployment.yaml
View file

@ -56,17 +56,7 @@ spec:
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
env:
{{- if .Values.lensWorker.enabled }}
- name: LITELLM_LENS_URL
value: {{ printf "http://%s-lens-worker:%v" (include "litellm.fullname" .) .Values.lensWorker.service.port | quote }}
- name: LITELLM_LENS_PUBLIC_URL
value: {{ include "litellm.lensWorker.publicUrl" . | quote }}
- name: LITELLM_LENS_SERVICE_TOKEN
valueFrom:
secretKeyRef:
name: {{ include "litellm.lensWorker.serviceTokenSecretName" . | quote }}
key: {{ .Values.lensWorker.serviceTokenSecret.key | quote }}
{{- end }}
{{- include "litellm.lensConnectionEnv" . | nindent 12 }}
{{- include "litellm.proxyEnv" . | nindent 12 }}
{{- if .Values.liteadmin.enabled }}
- name: LITELLM_ADMIN_AGENT_URL

8
helm/litellm-helm/templates/ingress.yaml
View file

@ -1,6 +1,8 @@
{{- if .Values.ingress.enabled -}}
{{- $fullName := include "litellm.fullname" . -}}
{{- $svcPort := .Values.service.port -}}
{{- $lensMode := include "litellm.lens.mode" . -}}
{{- $lensService := .Values.lensWorker.externalServiceName | default (printf "%s-lens-worker" $fullName) -}}
{{- if and .Values.ingress.className (not (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion)) }}
{{- if not (hasKey .Values.ingress.annotations "kubernetes.io/ingress.class") }}
{{- $_ := set .Values.ingress.annotations "kubernetes.io/ingress.class" .Values.ingress.className}}
@ -44,17 +46,17 @@ spec:
- host: {{ .host | quote }}
http:
paths:
{{- if $.Values.lensWorker.enabled }}
{{- if or (eq $lensMode "bundled") (and (eq $lensMode "external") $.Values.lensWorker.externalServiceName) }}
- path: /lens-ingest
pathType: Prefix
backend:
{{- if semverCompare ">=1.19-0" $.Capabilities.KubeVersion.GitVersion }}
service:
name: {{ $fullName }}-lens-worker
name: {{ if eq $lensMode "external" }}{{ $lensService }}{{ else }}{{ $fullName }}-lens-worker{{ end }}
port:
number: {{ $.Values.lensWorker.service.port }}
{{- else }}
serviceName: {{ $fullName }}-lens-worker
serviceName: {{ if eq $lensMode "external" }}{{ $lensService }}{{ else }}{{ $fullName }}-lens-worker{{ end }}
servicePort: {{ $.Values.lensWorker.service.port }}
{{- end }}
{{- end }}

99
helm/litellm-helm/templates/lens/clickhouse.yaml
View file

@ -1,98 +1,3 @@
{{- if include "litellm.lensWorker.bundledClickhouse" . }}
{{- $name := include "litellm.lensWorker.clickhouseName" . }}
apiVersion: v1
kind: Service
metadata:
name: {{ $name }}
spec:
clusterIP: None
selector:
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: lens-clickhouse
ports:
- name: http
port: 8123
targetPort: http
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ $name }}
spec:
serviceName: {{ $name }}
replicas: 1
selector:
matchLabels:
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: lens-clickhouse
template:
metadata:
labels:
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: lens-clickhouse
spec:
automountServiceAccountToken: false
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
securityContext:
runAsNonRoot: true
runAsUser: 101
runAsGroup: 101
fsGroup: 101
seccompProfile:
type: RuntimeDefault
containers:
- name: clickhouse
image: {{ .Values.lensWorker.clickhouse.image | quote }}
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
env:
- name: CLICKHOUSE_USER
value: default
- name: CLICKHOUSE_PASSWORD
valueFrom:
secretKeyRef:
name: {{ $name }}
key: password
- name: CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT
value: "1"
ports:
- name: http
containerPort: 8123
startupProbe:
httpGet:
path: /ping
port: http
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 60
readinessProbe:
httpGet:
path: /ping
port: http
livenessProbe:
httpGet:
path: /ping
port: http
timeoutSeconds: 3
resources:
{{- toYaml .Values.lensWorker.clickhouse.resources | nindent 12 }}
volumeMounts:
- name: data
mountPath: /var/lib/clickhouse
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ReadWriteOnce]
{{- if ne .Values.lensWorker.clickhouse.storageClassName nil }}
storageClassName: {{ .Values.lensWorker.clickhouse.storageClassName | quote }}
{{- end }}
resources:
requests:
storage: {{ .Values.lensWorker.clickhouse.storage | quote }}
{{- if eq (include "litellm.lens.mode" .) "bundled" }}
{{ include "litellm.lens.render" (dict "root" . "resource" "clickhouse") }}
{{- end }}

110
helm/litellm-helm/templates/lens/deployment.yaml
View file

@ -1,109 +1,3 @@
{{- if .Values.lensWorker.enabled }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "litellm.fullname" . }}-lens-worker
labels:
{{- include "litellm.lensWorker.labels" . | nindent 4 }}
app.kubernetes.io/component: lens-worker
spec:
replicas: {{ .Values.lensWorker.replicaCount }}
selector:
matchLabels:
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: lens-worker
template:
metadata:
labels:
{{- include "litellm.lensWorker.labels" . | nindent 8 }}
app.kubernetes.io/component: lens-worker
spec:
automountServiceAccountToken: false
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
securityContext:
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
fsGroup: 65532
seccompProfile:
type: RuntimeDefault
containers:
- name: lens-worker
image: {{ include "litellm.lensWorker.image" . | quote }}
imagePullPolicy: {{ .Values.lensWorker.image.pullPolicy }}
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: [ALL]
env:
- name: LITELLM_URL
value: {{ .Values.lensWorker.url | default (printf "http://%s:%v" (include "litellm.fullname" .) .Values.service.port) | quote }}
- name: LITELLM_LENS_SERVICE_TOKEN
valueFrom:
secretKeyRef:
name: {{ include "litellm.lensWorker.serviceTokenSecretName" . | quote }}
key: {{ .Values.lensWorker.serviceTokenSecret.key | quote }}
{{- if include "litellm.lensWorker.bundledClickhouse" . }}
- name: CLICKHOUSE_HOST
value: {{ include "litellm.lensWorker.clickhouseName" . | quote }}
- name: CLICKHOUSE_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "litellm.lensWorker.clickhouseName" . | quote }}
key: password
{{- else }}
- name: CLICKHOUSE_URL
valueFrom:
secretKeyRef:
name: {{ required "lensWorker.clickhouseSecret.name is required" .Values.lensWorker.clickhouseSecret.name | quote }}
key: {{ .Values.lensWorker.clickhouseSecret.key | quote }}
{{- end }}
- name: CLICKHOUSE_DATABASE
value: {{ .Values.lensWorker.clickhouseDatabase | quote }}
- name: AGENT_TRACING_RETENTION_DAYS
value: {{ .Values.lensWorker.retentionDays | quote }}
{{- if .Values.lensWorker.tokenSecret.name }}
- name: LENS_WORKER_TOKEN
valueFrom:
secretKeyRef:
name: {{ .Values.lensWorker.tokenSecret.name | quote }}
key: {{ .Values.lensWorker.tokenSecret.key | quote }}
{{- end }}
ports:
- name: otlp
containerPort: 4318
livenessProbe:
httpGet:
path: /health/live
port: otlp
readinessProbe:
httpGet:
path: /health/ready
port: otlp
resources:
{{- toYaml .Values.lensWorker.resources | nindent 12 }}
volumeMounts:
- name: tmp
mountPath: /tmp
volumes:
- name: tmp
emptyDir:
medium: Memory
sizeLimit: {{ .Values.lensWorker.tmpSizeLimit }}
{{- with .Values.lensWorker.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.lensWorker.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.lensWorker.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- if eq (include "litellm.lens.mode" .) "bundled" }}
{{ include "litellm.lens.render" (dict "root" . "resource" "deployment") }}
{{- end }}

30
helm/litellm-helm/templates/lens/ingress.yaml
View file

@ -1,29 +1,3 @@
{{- if and .Values.lensWorker.enabled .Values.lensWorker.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "litellm.fullname" . }}-lens-worker
{{- with .Values.lensWorker.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
{{- with .Values.lensWorker.ingress.className }}
ingressClassName: {{ . | quote }}
{{- end }}
{{- with .Values.lensWorker.ingress.tls }}
tls:
{{- toYaml . | nindent 4 }}
{{- end }}
rules:
- host: {{ required "lensWorker.ingress.host is required" .Values.lensWorker.ingress.host | quote }}
http:
paths:
- path: /v1/
pathType: Prefix
backend:
service:
name: {{ include "litellm.fullname" . }}-lens-worker
port:
name: otlp
{{- if eq (include "litellm.lens.mode" .) "bundled" }}
{{ include "litellm.lens.render" (dict "root" . "resource" "ingress") }}
{{- end }}

28
helm/litellm-helm/templates/lens/secrets.yaml
View file

@ -1,27 +1,3 @@
{{- if and .Values.lensWorker.enabled (not .Values.lensWorker.serviceTokenSecret.name) }}
{{- $name := include "litellm.lensWorker.serviceTokenSecretName" . }}
{{- $existing := lookup "v1" "Secret" .Release.Namespace $name }}
apiVersion: v1
kind: Secret
metadata:
name: {{ $name }}
annotations:
helm.sh/resource-policy: keep
type: Opaque
data:
{{ .Values.lensWorker.serviceTokenSecret.key }}: {{ if $existing }}{{ required "Saved Lens service secret is missing its key" (index $existing.data .Values.lensWorker.serviceTokenSecret.key) | quote }}{{ else }}{{ randAlphaNum 64 | b64enc | quote }}{{ end }}
{{- end }}
{{- if include "litellm.lensWorker.bundledClickhouse" . }}
{{- $name := include "litellm.lensWorker.clickhouseName" . }}
{{- $existing := lookup "v1" "Secret" .Release.Namespace $name }}
---
apiVersion: v1
kind: Secret
metadata:
name: {{ $name }}
annotations:
helm.sh/resource-policy: keep
type: Opaque
data:
password: {{ if $existing }}{{ required "Saved Lens ClickHouse secret is missing its password" (index $existing.data "password") | quote }}{{ else }}{{ randAlphaNum 64 | b64enc | quote }}{{ end }}
{{- if eq (include "litellm.lens.mode" .) "bundled" }}
{{ include "litellm.lens.render" (dict "root" . "resource" "secrets") }}
{{- end }}

19
helm/litellm-helm/templates/lens/service.yaml
View file

@ -1,18 +1,3 @@
{{- if .Values.lensWorker.enabled }}
apiVersion: v1
kind: Service
metadata:
name: {{ include "litellm.fullname" . }}-lens-worker
{{- with .Values.lensWorker.service.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
selector:
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: lens-worker
ports:
- name: otlp
port: {{ .Values.lensWorker.service.port }}
targetPort: otlp
{{- if eq (include "litellm.lens.mode" .) "bundled" }}
{{ include "litellm.lens.render" (dict "root" . "resource" "service") }}
{{- end }}

187
helm/litellm-helm/tests/lens_modes_tests.yaml Normal file
View file

@ -0,0 +1,187 @@
---
suite: Lens deployment modes
templates:
- ingress.yaml
- configmap-litellm.yaml
- deployment.yaml
- lens/deployment.yaml
- lens/service.yaml
- lens/clickhouse.yaml
- lens/secrets.yaml
set:
lensWorker.publicUrl: https://traces.example
tests:
- it: retains the existing deployment before transferring its release ownership
template: lens/deployment.yaml
set:
lensWorker.mode: bundled
lensWorker.retainResources: true
asserts:
- equal:
path: metadata.annotations["helm.sh/resource-policy"]
value: keep
- it: connects an external deployment without installing Lens resources
templates:
- lens/deployment.yaml
- lens/service.yaml
- lens/clickhouse.yaml
- lens/secrets.yaml
set:
lensWorker.mode: external
lensWorker.externalUrl: http://lens.other-namespace:4318
lensWorker.serviceTokenSecret.name: external-lens-signing
lensWorker.gateway.secretName: external-lens-signing
asserts:
- hasDocuments:
count: 0
- it: sends delegated requests to the explicitly selected external Lens
template: deployment.yaml
set:
lensWorker.mode: external
lensWorker.externalUrl: http://lens.other-namespace:4318
lensWorker.serviceTokenSecret.name: external-lens-signing
lensWorker.gateway.secretName: external-lens-signing
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_LENS_URL
value: http://lens.other-namespace:4318
- contains:
path: spec.template.spec.containers[0].env
content:
name: LENS_GATEWAY_SECRET
valueFrom:
secretKeyRef:
name: external-lens-signing
key: gateway-secret
- it: disables Lens explicitly even with a retained legacy enable flag
template: lens/deployment.yaml
set:
lensWorker.mode: disabled
lensWorker.enabled: true
asserts:
- hasDocuments:
count: 0
- it: bundles Lens through explicit mode selection
template: lens/deployment.yaml
set:
lensWorker.mode: bundled
asserts:
- hasDocuments:
count: 1
- contains:
path: spec.template.spec.containers[0].env
content:
name: LENS_MODE
value: standalone
- it: uses separate credentials for external identity and service access
template: deployment.yaml
set:
lensWorker.mode: external
lensWorker.externalUrl: http://existing-lens:4318
lensWorker.serviceTokenSecret: {name: worker-service, key: token}
lensWorker.gateway: {secretName: delegated-identity, secretKey: signature}
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LENS_GATEWAY_SECRET
valueFrom:
secretKeyRef: {name: delegated-identity, key: signature}
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_LENS_SERVICE_TOKEN
valueFrom:
secretKeyRef: {name: worker-service, key: token}
- it: supplies the same separate signing secret to the bundled service
template: lens/deployment.yaml
set:
lensWorker.mode: bundled
lensWorker.gateway: {secretName: delegated-identity, secretKey: signature}
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LENS_GATEWAY_SECRET
valueFrom:
secretKeyRef: {name: delegated-identity, key: signature}
- it: routes explicit bundled mode with the enable flag off
template: ingress.yaml
set:
fullnameOverride: gateway
ingress.enabled: true
ingress.hosts: [{host: gateway.example, paths: [{path: /, pathType: Prefix}]}]
lensWorker.mode: bundled
lensWorker.enabled: false
lensWorker.externalUrl: http://existing-lens:4318
lensWorker.externalServiceName: ""
lensWorker.serviceTokenSecret.name: worker-service
lensWorker.gateway.secretName: external-lens-signing
asserts:
- contains:
path: spec.rules[0].http.paths
content:
path: /lens-ingest
pathType: Prefix
backend:
service:
name: gateway-lens-worker
port: {number: 4318}
- it: routes external mode to its selected service
template: ingress.yaml
set:
fullnameOverride: gateway
ingress.enabled: true
ingress.hosts: [{host: gateway.example, paths: [{path: /, pathType: Prefix}]}]
lensWorker.mode: external
lensWorker.enabled: true
lensWorker.externalUrl: http://existing-lens:4318
lensWorker.externalServiceName: "existing-lens"
lensWorker.serviceTokenSecret.name: worker-service
lensWorker.gateway.secretName: external-lens-signing
asserts:
- contains:
path: spec.rules[0].http.paths
content:
path: /lens-ingest
pathType: Prefix
backend:
service:
name: existing-lens
port: {number: 4318}
- it: omits ingestion when an external service has no ingress backend
template: ingress.yaml
set:
fullnameOverride: gateway
ingress.enabled: true
ingress.hosts: [{host: gateway.example, paths: [{path: /, pathType: Prefix}]}]
lensWorker.mode: external
lensWorker.enabled: true
lensWorker.externalUrl: http://existing-lens:4318
lensWorker.externalServiceName: ""
lensWorker.serviceTokenSecret.name: worker-service
lensWorker.gateway.secretName: external-lens-signing
asserts:
- notContains:
path: spec.rules[0].http.paths
content: {path: /lens-ingest}
any: true
- it: omits ingestion for disabled mode with the enable flag on
template: ingress.yaml
set:
fullnameOverride: gateway
ingress.enabled: true
ingress.hosts: [{host: gateway.example, paths: [{path: /, pathType: Prefix}]}]
lensWorker.mode: disabled
lensWorker.enabled: true
lensWorker.externalUrl: http://existing-lens:4318
lensWorker.externalServiceName: ""
lensWorker.serviceTokenSecret.name: worker-service
lensWorker.gateway.secretName: external-lens-signing
asserts:
- notContains:
path: spec.rules[0].http.paths
content: {path: /lens-ingest}
any: true

52
helm/litellm-helm/tests/lens_modes_validation_tests.yaml Normal file
View file

@ -0,0 +1,52 @@
---
suite: Lens deployment mode validation
templates:
- configmap-litellm.yaml
- deployment.yaml
set:
lensWorker.publicUrl: https://traces.example
tests:
- it: requires a separate identity signing reference for external Lens
template: deployment.yaml
set:
lensWorker.mode: external
lensWorker.externalUrl: http://lens.other-namespace:4318
lensWorker.serviceTokenSecret.name: worker-service
asserts:
- failedTemplate:
errorMessage: lensWorker.gateway.secretName is required for external Lens
- it: requires an external service address
template: deployment.yaml
set:
lensWorker.mode: external
lensWorker.serviceTokenSecret.name: external-lens-signing
lensWorker.gateway.secretName: external-lens-signing
asserts:
- failedTemplate:
errorMessage: lensWorker.externalUrl is required for external Lens
- it: requires the shared server credential for an external service
template: deployment.yaml
set:
lensWorker.mode: external
lensWorker.externalUrl: http://lens.other-namespace:4318
asserts:
- failedTemplate:
errorMessage: lensWorker.serviceTokenSecret.name is required for external Lens
- it: requires an agent reachable URL for external ingestion
template: deployment.yaml
set:
lensWorker.mode: external
lensWorker.externalUrl: http://lens.other-namespace:4318
lensWorker.serviceTokenSecret.name: external-lens-signing
lensWorker.gateway.secretName: external-lens-signing
lensWorker.publicUrl: ''
asserts:
- failedTemplate:
errorMessage: lensWorker.publicUrl is required for external Lens
- it: rejects unsupported deployment modes
template: deployment.yaml
set:
lensWorker.mode: typo
asserts:
- failedTemplate:
errorMessage: lensWorker.mode must be bundled, external or disabled

14
helm/litellm-helm/tests/lens_saved_secrets_tests.yaml
View file

@ -16,6 +16,13 @@ kubernetesProvider:
resource: secrets
namespaced: true
objects:
- apiVersion: v1
kind: Secret
metadata:
name: lens-test-lens-gateway
namespace: lens
data:
gateway-secret: c2F2ZWQtc2lnbmluZy1rZXk=
- apiVersion: v1
kind: Secret
metadata:
@ -31,6 +38,13 @@ kubernetesProvider:
data:
password: c2F2ZWQtZGF0YWJhc2UtcGFzc3dvcmQ=
tests:
- it: preserves the separate signing credential across upgrades
documentSelector: {path: metadata.name, value: lens-test-lens-gateway}
asserts:
- equal:
path: data.gateway-secret
value: c2F2ZWQtc2lnbmluZy1rZXk=
- notExists: {path: data.service-token}
- it: reuses the service credential instead of breaking running services
documentIndex: 0
asserts:

6
helm/litellm-helm/tests/lens_service_tests.yaml
View file

@ -29,11 +29,11 @@ tests:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_LENS_SERVICE_TOKEN
name: LENS_GATEWAY_SECRET
valueFrom:
secretKeyRef:
name: lens-service
key: service-token
name: lens-test-lens-gateway
key: gateway-secret
- it: routes uploads directly to Lens instead of the gateway
template: ingress.yaml
set:

8
helm/litellm-helm/tests/lens_setup_tests.yaml
View file

@ -11,11 +11,11 @@ templates:
- lens/clickhouse.yaml
- lens/secrets.yaml
tests:
- it: generates both private credentials for a new installation
- it: generates private login, gateway and storage credentials
template: lens/secrets.yaml
asserts:
- hasDocuments:
count: 2
count: 4
- matchRegex:
path: data.service-token
pattern: '^[A-Za-z0-9+/]{86}==$'
@ -78,6 +78,8 @@ tests:
set:
lensWorker.clickhouseSecret.name: external-clickhouse
lensWorker.serviceTokenSecret.name: external-service
lensWorker.adminTokenSecret.name: external-admin
lensWorker.gateway.secretName: external-signing
asserts:
- hasDocuments:
count: 0
@ -96,7 +98,7 @@ tests:
lensWorker.clickhouse.enabled: false
asserts:
- failedTemplate:
errorMessage: lensWorker.clickhouseSecret.name is required
errorMessage: Lens clickhouseSecret.name is required when bundled storage is disabled
- it: keeps storage names valid and uses the same name for the connection
set:
fullnameOverride: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

23
helm/litellm-helm/values.yaml
View file

@ -653,12 +653,28 @@ serviceMonitor:
matchNames: []
# - test-namespace
lens:
library: true
lensWorker:
retainResources: false
mode: ""
externalUrl: ""
externalServiceName: ""
gateway:
secretName: ""
secretKey: gateway-secret
standaloneUrl: ""
adminTokenSecret:
name: ""
key: admin-token
extraEnv: []
extraEnvFrom: []
enabled: false
replicaCount: 1
image:
repository: ghcr.io/berriai/litellm-lens-worker
tag: ""
repository: ghcr.io/berriai/lens
tag: "0.1.0-dev.0"
digest: ""
pullPolicy: IfNotPresent
tokenSecret:
@ -669,7 +685,7 @@ lensWorker:
key: service-token
clickhouse:
enabled: true
image: clickhouse/clickhouse-server:26.9.6.6
image: clickhouse/clickhouse-server:26.9.6.6@sha256:eb4870e7ca7ed70c259eebfcfbee6cf797017f6b5436c2926bbbfe3d4d28486e
storage: 20Gi
storageClassName: null
resources:
@ -689,6 +705,7 @@ lensWorker:
annotations: {}
ingress:
enabled: false
path: /v1/
className: ""
host: ""
annotations: {}

6
helm/litellm/Chart.lock Normal file
View file

@ -0,0 +1,6 @@
dependencies:
- name: lens
repository: oci://ghcr.io/berriai/charts
version: 0.1.0-dev.0
digest: sha256:6b244c878890f10a093a297d70c9e68b2c3bda00fbbb56ab287e7ecee1d7bbcf
generated: '2026-10-09T04:18:10.642706000Z'

5
helm/litellm/Chart.yaml
View file

@ -6,3 +6,8 @@ version: 0.1.0
appVersion: "0.1.0"
annotations:
org.opencontainers.image.source: "https://github.com/BerriAI/litellm"
dependencies:
- name: lens
version: 0.1.0-dev.0
repository: oci://ghcr.io/berriai/charts

7
helm/litellm/README.md Normal file
View file

@ -0,0 +1,7 @@
# LiteLLM componentized chart
This chart deploys separate gateway, backend and UI services. Configure Lens through `lensWorker.mode`: `disabled` keeps it off, `bundled` installs the Lens dependency, and `external` connects an existing Lens service
Bundled mode generates separate service and identity-signing credentials. External mode requires `lensWorker.gateway.secretName` and `lensWorker.serviceTokenSecret.name`; their keys default to `gateway-secret` and `service-token`. Provision distinct values matching the external Lens deployment
Follow the [Lens Helm connection guide](https://github.com/BerriAI/lens/blob/main/helm/lens/README.md#connect-a-gateway) for the complete values and verification flow. It also links the GitOps credential requirements and existing-data migration. Source-chart installation requires a built Lens image until the first signed Lens release is published

BIN
helm/litellm/charts/lens-0.1.0-dev.0.tgz Normal file
View file

Binary file not shown.

68
helm/litellm/templates/_helpers.tpl
View file

@ -472,21 +472,7 @@ collector containers through an emptyDir. Empty when the sidecar is off
or gateway.collector.address is a tcp://127.0.0.1:<port> address.
*/}}
{{- define "litellm.lensWorker.image" -}}
{{- if .Values.lensWorker.image.digest -}}
{{- if not (regexMatch "^sha256:[0-9a-f]{64}$" .Values.lensWorker.image.digest) -}}
{{- fail "lensWorker.image.digest must be sha256 followed by 64 lowercase hex characters" -}}
{{- end -}}
{{- printf "%s@%s" .Values.lensWorker.image.repository .Values.lensWorker.image.digest -}}
{{- else -}}
{{- $backendTag := .Values.backend.image.tag | default .Chart.AppVersion -}}
{{- $releaseTag := ternary (printf "v%s" $backendTag) $backendTag (regexMatch "^[0-9]" $backendTag) -}}
{{- $tag := .Values.lensWorker.image.tag | default $releaseTag -}}
{{- $repository := .Values.lensWorker.image.repository -}}
{{- if and (hasPrefix "sha-" $tag) (eq $repository "ghcr.io/berriai/litellm-lens-worker") -}}
{{- $repository = "ghcr.io/berriai/litellm-lens-worker-dev" -}}
{{- end -}}
{{- printf "%s:%s" $repository $tag -}}
{{- end -}}
{{- include "lens.image" (dict "Values" .Values.lensWorker "Chart" .Subcharts.lens.Chart) -}}
{{- end -}}
{{- define "litellm.gateway.collectorSocketDir" -}}
@ -516,11 +502,23 @@ shutdown drain window.
{{- end -}}
{{- define "litellm.lensConnectionEnv" -}}
{{- if .Values.lensWorker.enabled }}
{{- $mode := include "litellm.lens.mode" . -}}
{{- if ne $mode "disabled" }}
{{- if and (eq $mode "external") (not .Values.lensWorker.serviceTokenSecret.name) -}}
{{- fail "lensWorker.serviceTokenSecret.name is required for external Lens" -}}
{{- end }}
{{- if and (eq $mode "external") (not .Values.lensWorker.gateway.secretName) -}}
{{- fail "lensWorker.gateway.secretName is required for external Lens" -}}
{{- end -}}
- name: LITELLM_LENS_URL
value: {{ printf "http://%s-lens-worker:%v" (include "litellm.fullname" .) .Values.lensWorker.service.port | quote }}
value: {{ if eq $mode "external" }}{{ required "lensWorker.externalUrl is required for external Lens" .Values.lensWorker.externalUrl | quote }}{{ else }}{{ printf "http://%s-lens-worker:%v" (include "litellm.fullname" .) .Values.lensWorker.service.port | quote }}{{ end }}
- name: LITELLM_LENS_PUBLIC_URL
value: {{ include "litellm.lensWorker.publicUrl" . | quote }}
- name: LENS_GATEWAY_SECRET
valueFrom:
secretKeyRef:
name: {{ include "litellm.lensWorker.gatewaySecretName" . | quote }}
key: {{ .Values.lensWorker.gateway.secretKey | quote }}
- name: LITELLM_LENS_SERVICE_TOKEN
valueFrom:
secretKeyRef:
@ -539,13 +537,19 @@ shutdown drain window.
{{- .Values.lensWorker.serviceTokenSecret.name | default (printf "%s-lens-service" (include "litellm.fullname" .)) -}}
{{- end -}}
{{- define "litellm.lensWorker.gatewaySecretName" -}}
{{- .Values.lensWorker.gateway.secretName | default (printf "%s-lens-gateway" (include "litellm.fullname" . | trunc 50 | trimSuffix "-")) -}}
{{- end -}}
{{- define "litellm.lensWorker.bundledClickhouse" -}}
{{- if and .Values.lensWorker.enabled .Values.lensWorker.clickhouse.enabled (not .Values.lensWorker.clickhouseSecret.name) -}}true{{- end -}}
{{- if and (eq (include "litellm.lens.mode" .) "bundled") .Values.lensWorker.clickhouse.enabled (not .Values.lensWorker.clickhouseSecret.name) -}}true{{- end -}}
{{- end -}}
{{- define "litellm.lensWorker.publicUrl" -}}
{{- if .Values.lensWorker.publicUrl -}}
{{- .Values.lensWorker.publicUrl -}}
{{- else if eq (include "litellm.lens.mode" .) "external" -}}
{{- fail "lensWorker.publicUrl is required for external Lens" -}}
{{- else if .Values.lensWorker.ingress.enabled -}}
{{- $tls := or (not (empty .Values.lensWorker.ingress.tls)) (hasKey .Values.lensWorker.ingress.annotations "alb.ingress.kubernetes.io/certificate-arn") -}}
{{- printf "%s://%s" (ternary "https" "http" $tls) (required "lensWorker.ingress.host is required" .Values.lensWorker.ingress.host) -}}
@ -560,3 +564,31 @@ shutdown drain window.
{{- define "litellm.lensWorker.clickhouseName" -}}
{{- printf "%s-lens-clickhouse" (include "litellm.fullname" . | trunc 47 | trimSuffix "-") -}}
{{- end -}}
{{- define "litellm.lens.mode" -}}
{{- $mode := .Values.lensWorker.mode | default (ternary "bundled" "disabled" .Values.lensWorker.enabled) -}}
{{- if not (has $mode (list "bundled" "external" "disabled")) -}}
{{- fail "lensWorker.mode must be bundled, external or disabled" -}}
{{- end -}}
{{- $mode -}}
{{- end -}}
{{- define "litellm.lens.render" -}}
{{- $root := .root -}}
{{- $values := mergeOverwrite (deepCopy $root.Subcharts.lens.Values) (deepCopy $root.Values.lensWorker) -}}
{{- $_ := set $values "fullnameOverride" (printf "%s-lens-worker" (include "litellm.fullname" $root)) -}}
{{- $_ := set $values "component" "lens-worker" -}}
{{- $_ := set $values "nameOverride" (printf "%s-lens-worker" (include "litellm.name" $root | trunc 51 | trimSuffix "-")) -}}
{{- $_ := set $values "imagePullSecrets" $root.Values.imagePullSecrets -}}
{{- $_ := set $values.gateway "enabled" true -}}
{{- $_ := set $values.gateway "generatedName" (include "litellm.lensWorker.gatewaySecretName" $root) -}}
{{- $_ := set $values.clickhouse "nameOverride" (include "litellm.lensWorker.clickhouseName" $root) -}}
{{- $_ := set $values.serviceTokenSecret "generatedName" (include "litellm.lensWorker.serviceTokenSecretName" $root) -}}
{{- $_ := set $values "publicUrl" ($root.Values.lensWorker.standaloneUrl | default $root.Subcharts.lens.Values.publicUrl) -}}
{{- if and (eq .resource "deployment") (or $root.Values.lensWorker.publicUrl $root.Values.lensWorker.ingress.enabled $root.Values.ingress.enabled) -}}
{{- $ingestion := include "litellm.lensWorker.publicUrl" $root -}}
{{- $_ := set $values "ingestionUrl" $ingestion -}}
{{- $_ := set $values "publicUrl" ($root.Values.lensWorker.standaloneUrl | default (trimSuffix "/lens-ingest" $ingestion)) -}}
{{- end -}}
{{- include (printf "lens.%s" .resource) (dict "Values" $values "Release" $root.Release "Chart" $root.Subcharts.lens.Chart "Capabilities" $root.Capabilities) -}}
{{- end -}}

2
helm/litellm/templates/backend/deployment.yaml
View file

@ -58,8 +58,6 @@ spec:
protocol: TCP
env:
{{- include "litellm.lensConnectionEnv" . | nindent 12 }}
- name: LENS_WORKER_IMAGE
value: {{ include "litellm.lensWorker.image" . | quote }}
{{- include "litellm.serverEnv" (dict "root" $ "component" .Values.backend) | nindent 12 }}
{{- if .Values.gateway.config.create }}
- name: CONFIG_FILE_PATH

4
helm/litellm/templates/ingress.yaml
View file

@ -156,13 +156,13 @@ spec:
port:
number: {{ $gatewayPort }}
{{- end }}
{{- if .Values.lensWorker.enabled }}
{{- if or (eq (include "litellm.lens.mode" .) "bundled") (and (eq (include "litellm.lens.mode" .) "external") .Values.lensWorker.externalServiceName) }}
{{- $builtinPathKeys = append $builtinPathKeys "/lens-ingest|Prefix" }}
- path: /lens-ingest
pathType: Prefix
backend:
service:
name: {{ include "litellm.fullname" . }}-lens-worker
name: {{ .Values.lensWorker.externalServiceName | default (printf "%s-lens-worker" (include "litellm.fullname" .)) }}
port:
number: {{ .Values.lensWorker.service.port }}
{{- end }}

99
helm/litellm/templates/lens/clickhouse.yaml
View file

@ -1,98 +1,3 @@
{{- if include "litellm.lensWorker.bundledClickhouse" . }}
{{- $name := include "litellm.lensWorker.clickhouseName" . }}
apiVersion: v1
kind: Service
metadata:
name: {{ $name }}
spec:
clusterIP: None
selector:
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: lens-clickhouse
ports:
- name: http
port: 8123
targetPort: http
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ $name }}
spec:
serviceName: {{ $name }}
replicas: 1
selector:
matchLabels:
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: lens-clickhouse
template:
metadata:
labels:
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: lens-clickhouse
spec:
automountServiceAccountToken: false
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
securityContext:
runAsNonRoot: true
runAsUser: 101
runAsGroup: 101
fsGroup: 101
seccompProfile:
type: RuntimeDefault
containers:
- name: clickhouse
image: {{ .Values.lensWorker.clickhouse.image | quote }}
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
env:
- name: CLICKHOUSE_USER
value: default
- name: CLICKHOUSE_PASSWORD
valueFrom:
secretKeyRef:
name: {{ $name }}
key: password
- name: CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT
value: "1"
ports:
- name: http
containerPort: 8123
startupProbe:
httpGet:
path: /ping
port: http
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 60
readinessProbe:
httpGet:
path: /ping
port: http
livenessProbe:
httpGet:
path: /ping
port: http
timeoutSeconds: 3
resources:
{{- toYaml .Values.lensWorker.clickhouse.resources | nindent 12 }}
volumeMounts:
- name: data
mountPath: /var/lib/clickhouse
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ReadWriteOnce]
{{- if ne .Values.lensWorker.clickhouse.storageClassName nil }}
storageClassName: {{ .Values.lensWorker.clickhouse.storageClassName | quote }}
{{- end }}
resources:
requests:
storage: {{ .Values.lensWorker.clickhouse.storage | quote }}
{{- if eq (include "litellm.lens.mode" .) "bundled" }}
{{ include "litellm.lens.render" (dict "root" . "resource" "clickhouse") }}
{{- end }}

110
helm/litellm/templates/lens/deployment.yaml
View file

@ -1,109 +1,3 @@
{{- if .Values.lensWorker.enabled }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "litellm.fullname" . }}-lens-worker
labels:
{{- include "litellm.lensWorker.labels" . | nindent 4 }}
app.kubernetes.io/component: lens-worker
spec:
replicas: {{ .Values.lensWorker.replicaCount }}
selector:
matchLabels:
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: lens-worker
template:
metadata:
labels:
{{- include "litellm.lensWorker.labels" . | nindent 8 }}
app.kubernetes.io/component: lens-worker
spec:
automountServiceAccountToken: false
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
securityContext:
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
fsGroup: 65532
seccompProfile:
type: RuntimeDefault
containers:
- name: lens-worker
image: {{ include "litellm.lensWorker.image" . | quote }}
imagePullPolicy: {{ .Values.lensWorker.image.pullPolicy }}
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: [ALL]
env:
- name: LITELLM_URL
value: {{ .Values.lensWorker.url | default (printf "http://%s:%v" (include "litellm.backend.fullname" .) .Values.backend.service.port) | quote }}
- name: LITELLM_LENS_SERVICE_TOKEN
valueFrom:
secretKeyRef:
name: {{ include "litellm.lensWorker.serviceTokenSecretName" . | quote }}
key: {{ .Values.lensWorker.serviceTokenSecret.key | quote }}
{{- if include "litellm.lensWorker.bundledClickhouse" . }}
- name: CLICKHOUSE_HOST
value: {{ include "litellm.lensWorker.clickhouseName" . | quote }}
- name: CLICKHOUSE_PASSWORD
valueFrom:
secretKeyRef:
name: {{ include "litellm.lensWorker.clickhouseName" . | quote }}
key: password
{{- else }}
- name: CLICKHOUSE_URL
valueFrom:
secretKeyRef:
name: {{ required "lensWorker.clickhouseSecret.name is required" .Values.lensWorker.clickhouseSecret.name | quote }}
key: {{ .Values.lensWorker.clickhouseSecret.key | quote }}
{{- end }}
- name: CLICKHOUSE_DATABASE
value: {{ .Values.lensWorker.clickhouseDatabase | quote }}
- name: AGENT_TRACING_RETENTION_DAYS
value: {{ .Values.lensWorker.retentionDays | quote }}
{{- if .Values.lensWorker.tokenSecret.name }}
- name: LENS_WORKER_TOKEN
valueFrom:
secretKeyRef:
name: {{ .Values.lensWorker.tokenSecret.name | quote }}
key: {{ .Values.lensWorker.tokenSecret.key | quote }}
{{- end }}
ports:
- name: otlp
containerPort: 4318
livenessProbe:
httpGet:
path: /health/live
port: otlp
readinessProbe:
httpGet:
path: /health/ready
port: otlp
resources:
{{- toYaml .Values.lensWorker.resources | nindent 12 }}
volumeMounts:
- name: tmp
mountPath: /tmp
volumes:
- name: tmp
emptyDir:
medium: Memory
sizeLimit: {{ .Values.lensWorker.tmpSizeLimit }}
{{- with .Values.lensWorker.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.lensWorker.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.lensWorker.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- if eq (include "litellm.lens.mode" .) "bundled" }}
{{ include "litellm.lens.render" (dict "root" . "resource" "deployment") }}
{{- end }}

30
helm/litellm/templates/lens/ingress.yaml
View file

@ -1,29 +1,3 @@
{{- if and .Values.lensWorker.enabled .Values.lensWorker.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "litellm.fullname" . }}-lens-worker
{{- with .Values.lensWorker.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
{{- with .Values.lensWorker.ingress.className }}
ingressClassName: {{ . | quote }}
{{- end }}
{{- with .Values.lensWorker.ingress.tls }}
tls:
{{- toYaml . | nindent 4 }}
{{- end }}
rules:
- host: {{ required "lensWorker.ingress.host is required" .Values.lensWorker.ingress.host | quote }}
http:
paths:
- path: /v1/
pathType: Prefix
backend:
service:
name: {{ include "litellm.fullname" . }}-lens-worker
port:
name: otlp
{{- if eq (include "litellm.lens.mode" .) "bundled" }}
{{ include "litellm.lens.render" (dict "root" . "resource" "ingress") }}
{{- end }}

28
helm/litellm/templates/lens/secrets.yaml
View file

@ -1,27 +1,3 @@
{{- if and .Values.lensWorker.enabled (not .Values.lensWorker.serviceTokenSecret.name) }}
{{- $name := include "litellm.lensWorker.serviceTokenSecretName" . }}
{{- $existing := lookup "v1" "Secret" .Release.Namespace $name }}
apiVersion: v1
kind: Secret
metadata:
name: {{ $name }}
annotations:
helm.sh/resource-policy: keep
type: Opaque
data:
{{ .Values.lensWorker.serviceTokenSecret.key }}: {{ if $existing }}{{ required "Saved Lens service secret is missing its key" (index $existing.data .Values.lensWorker.serviceTokenSecret.key) | quote }}{{ else }}{{ randAlphaNum 64 | b64enc | quote }}{{ end }}
{{- end }}
{{- if include "litellm.lensWorker.bundledClickhouse" . }}
{{- $name := include "litellm.lensWorker.clickhouseName" . }}
{{- $existing := lookup "v1" "Secret" .Release.Namespace $name }}
---
apiVersion: v1
kind: Secret
metadata:
name: {{ $name }}
annotations:
helm.sh/resource-policy: keep
type: Opaque
data:
password: {{ if $existing }}{{ required "Saved Lens ClickHouse secret is missing its password" (index $existing.data "password") | quote }}{{ else }}{{ randAlphaNum 64 | b64enc | quote }}{{ end }}
{{- if eq (include "litellm.lens.mode" .) "bundled" }}
{{ include "litellm.lens.render" (dict "root" . "resource" "secrets") }}
{{- end }}

19
helm/litellm/templates/lens/service.yaml
View file

@ -1,18 +1,3 @@
{{- if .Values.lensWorker.enabled }}
apiVersion: v1
kind: Service
metadata:
name: {{ include "litellm.fullname" . }}-lens-worker
{{- with .Values.lensWorker.service.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
selector:
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: lens-worker
ports:
- name: otlp
port: {{ .Values.lensWorker.service.port }}
targetPort: otlp
{{- if eq (include "litellm.lens.mode" .) "bundled" }}
{{ include "litellm.lens.render" (dict "root" . "resource" "service") }}
{{- end }}

208
helm/litellm/tests/lens_modes_tests.yaml Normal file
View file

@ -0,0 +1,208 @@
---
suite: Lens deployment modes
templates:
- gateway/configmap.yaml
- backend/deployment.yaml
- lens/deployment.yaml
- lens/service.yaml
- lens/clickhouse.yaml
- lens/secrets.yaml
- ingress.yaml
values:
- "./values/required.yaml"
set:
lensWorker.publicUrl: https://traces.example
tests:
- it: retains the existing deployment before transferring its release ownership
template: lens/deployment.yaml
set:
lensWorker.mode: bundled
lensWorker.retainResources: true
asserts:
- equal:
path: metadata.annotations["helm.sh/resource-policy"]
value: keep
- it: keeps the ingestion endpoint on an independently deployed Lens service
template: ingress.yaml
set:
ingress.enabled: true
lensWorker.mode: external
lensWorker.externalUrl: http://existing-lens:4318
lensWorker.externalServiceName: existing-lens
lensWorker.serviceTokenSecret.name: external-lens-signing
lensWorker.gateway.secretName: external-lens-signing
asserts:
- contains:
path: spec.rules[0].http.paths
content:
path: /lens-ingest
pathType: Prefix
backend:
service:
name: existing-lens
port: {number: 4318}
- it: connects an external deployment without installing Lens resources
templates:
- lens/deployment.yaml
- lens/service.yaml
- lens/clickhouse.yaml
- lens/secrets.yaml
set:
lensWorker.mode: external
lensWorker.externalUrl: http://lens.other-namespace:4318
lensWorker.serviceTokenSecret.name: external-lens-signing
lensWorker.gateway.secretName: external-lens-signing
asserts:
- hasDocuments:
count: 0
- it: sends delegated requests to the explicitly selected external Lens
template: backend/deployment.yaml
set:
lensWorker.mode: external
lensWorker.externalUrl: http://lens.other-namespace:4318
lensWorker.serviceTokenSecret.name: external-lens-signing
lensWorker.gateway.secretName: external-lens-signing
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_LENS_URL
value: http://lens.other-namespace:4318
- contains:
path: spec.template.spec.containers[0].env
content:
name: LENS_GATEWAY_SECRET
valueFrom:
secretKeyRef:
name: external-lens-signing
key: gateway-secret
- it: disables Lens explicitly even with a retained legacy enable flag
template: lens/deployment.yaml
set:
lensWorker.mode: disabled
lensWorker.enabled: true
asserts:
- hasDocuments:
count: 0
- it: bundles Lens through explicit mode selection
template: lens/deployment.yaml
set:
lensWorker.mode: bundled
asserts:
- hasDocuments:
count: 1
- contains:
path: spec.template.spec.containers[0].env
content:
name: LENS_MODE
value: standalone
- it: uses separate credentials for external identity and service access
template: backend/deployment.yaml
set:
lensWorker.mode: external
lensWorker.externalUrl: http://existing-lens:4318
lensWorker.serviceTokenSecret: {name: worker-service, key: token}
lensWorker.gateway: {secretName: delegated-identity, secretKey: signature}
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LENS_GATEWAY_SECRET
valueFrom:
secretKeyRef: {name: delegated-identity, key: signature}
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_LENS_SERVICE_TOKEN
valueFrom:
secretKeyRef: {name: worker-service, key: token}
- it: supplies the same separate signing secret to the bundled service
template: lens/deployment.yaml
set:
lensWorker.mode: bundled
lensWorker.gateway: {secretName: delegated-identity, secretKey: signature}
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LENS_GATEWAY_SECRET
valueFrom:
secretKeyRef: {name: delegated-identity, key: signature}
- it: routes explicit bundled mode with the enable flag off
template: ingress.yaml
set:
fullnameOverride: gateway
ingress.enabled: true
ingress.host: gateway.example
lensWorker.mode: bundled
lensWorker.enabled: false
lensWorker.externalUrl: http://existing-lens:4318
lensWorker.externalServiceName: ""
lensWorker.serviceTokenSecret.name: worker-service
lensWorker.gateway.secretName: external-lens-signing
asserts:
- contains:
path: spec.rules[0].http.paths
content:
path: /lens-ingest
pathType: Prefix
backend:
service:
name: gateway-lens-worker
port: {number: 4318}
- it: routes external mode to its selected service
template: ingress.yaml
set:
fullnameOverride: gateway
ingress.enabled: true
ingress.host: gateway.example
lensWorker.mode: external
lensWorker.enabled: true
lensWorker.externalUrl: http://existing-lens:4318
lensWorker.externalServiceName: "existing-lens"
lensWorker.serviceTokenSecret.name: worker-service
lensWorker.gateway.secretName: external-lens-signing
asserts:
- contains:
path: spec.rules[0].http.paths
content:
path: /lens-ingest
pathType: Prefix
backend:
service:
name: existing-lens
port: {number: 4318}
- it: omits ingestion when an external service has no ingress backend
template: ingress.yaml
set:
fullnameOverride: gateway
ingress.enabled: true
ingress.host: gateway.example
lensWorker.mode: external
lensWorker.enabled: true
lensWorker.externalUrl: http://existing-lens:4318
lensWorker.externalServiceName: ""
lensWorker.serviceTokenSecret.name: worker-service
lensWorker.gateway.secretName: external-lens-signing
asserts:
- notContains:
path: spec.rules[0].http.paths
content: {path: /lens-ingest}
any: true
- it: omits ingestion for disabled mode with the enable flag on
template: ingress.yaml
set:
fullnameOverride: gateway
ingress.enabled: true
ingress.host: gateway.example
lensWorker.mode: disabled
lensWorker.enabled: true
lensWorker.externalUrl: http://existing-lens:4318
lensWorker.externalServiceName: ""
lensWorker.serviceTokenSecret.name: worker-service
lensWorker.gateway.secretName: external-lens-signing
asserts:
- notContains:
path: spec.rules[0].http.paths
content: {path: /lens-ingest}
any: true

54
helm/litellm/tests/lens_modes_validation_tests.yaml Normal file
View file

@ -0,0 +1,54 @@
---
suite: Lens deployment mode validation
templates:
- gateway/configmap.yaml
- backend/deployment.yaml
values:
- "./values/required.yaml"
set:
lensWorker.publicUrl: https://traces.example
tests:
- it: requires a separate identity signing reference for external Lens
template: backend/deployment.yaml
set:
lensWorker.mode: external
lensWorker.externalUrl: http://lens.other-namespace:4318
lensWorker.serviceTokenSecret.name: worker-service
asserts:
- failedTemplate:
errorMessage: lensWorker.gateway.secretName is required for external Lens
- it: requires an external service address
template: backend/deployment.yaml
set:
lensWorker.mode: external
lensWorker.serviceTokenSecret.name: external-lens-signing
lensWorker.gateway.secretName: external-lens-signing
asserts:
- failedTemplate:
errorMessage: lensWorker.externalUrl is required for external Lens
- it: requires the shared server credential for an external service
template: backend/deployment.yaml
set:
lensWorker.mode: external
lensWorker.externalUrl: http://lens.other-namespace:4318
asserts:
- failedTemplate:
errorMessage: lensWorker.serviceTokenSecret.name is required for external Lens
- it: requires an agent reachable URL for external ingestion
template: backend/deployment.yaml
set:
lensWorker.mode: external
lensWorker.externalUrl: http://lens.other-namespace:4318
lensWorker.serviceTokenSecret.name: external-lens-signing
lensWorker.gateway.secretName: external-lens-signing
lensWorker.publicUrl: ''
asserts:
- failedTemplate:
errorMessage: lensWorker.publicUrl is required for external Lens
- it: rejects unsupported deployment modes
template: backend/deployment.yaml
set:
lensWorker.mode: typo
asserts:
- failedTemplate:
errorMessage: lensWorker.mode must be bundled, external or disabled

14
helm/litellm/tests/lens_saved_secrets_tests.yaml
View file

@ -16,6 +16,13 @@ kubernetesProvider:
resource: secrets
namespaced: true
objects:
- apiVersion: v1
kind: Secret
metadata:
name: lens-test-lens-gateway
namespace: lens
data:
gateway-secret: c2F2ZWQtc2lnbmluZy1rZXk=
- apiVersion: v1
kind: Secret
metadata:
@ -31,6 +38,13 @@ kubernetesProvider:
data:
password: c2F2ZWQtZGF0YWJhc2UtcGFzc3dvcmQ=
tests:
- it: preserves the separate signing credential across upgrades
documentSelector: {path: metadata.name, value: lens-test-lens-gateway}
asserts:
- equal:
path: data.gateway-secret
value: c2F2ZWQtc2lnbmluZy1rZXk=
- notExists: {path: data.service-token}
- it: reuses the service credential instead of breaking running services
documentIndex: 0
asserts:

12
helm/litellm/tests/lens_service_tests.yaml
View file

@ -30,11 +30,11 @@ tests:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_LENS_SERVICE_TOKEN
name: LENS_GATEWAY_SECRET
valueFrom:
secretKeyRef:
name: lens-service
key: service-token
name: lens-test-lens-gateway
key: gateway-secret
- it: connects backend/deployment.yaml to the shared Lens service
template: backend/deployment.yaml
set: *id001
@ -52,11 +52,11 @@ tests:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_LENS_SERVICE_TOKEN
name: LENS_GATEWAY_SECRET
valueFrom:
secretKeyRef:
name: lens-service
key: service-token
name: lens-test-lens-gateway
key: gateway-secret
- it: routes uploads directly to Lens instead of the gateway
template: ingress.yaml
set:

8
helm/litellm/tests/lens_setup_tests.yaml
View file

@ -11,11 +11,11 @@ templates:
- lens/clickhouse.yaml
- lens/secrets.yaml
tests:
- it: generates both private credentials for a new installation
- it: generates private login, gateway and storage credentials
template: lens/secrets.yaml
asserts:
- hasDocuments:
count: 2
count: 4
- matchRegex:
path: data.service-token
pattern: '^[A-Za-z0-9+/]{86}==$'
@ -78,6 +78,8 @@ tests:
set:
lensWorker.clickhouseSecret.name: external-clickhouse
lensWorker.serviceTokenSecret.name: external-service
lensWorker.adminTokenSecret.name: external-admin
lensWorker.gateway.secretName: external-signing
asserts:
- hasDocuments:
count: 0
@ -96,7 +98,7 @@ tests:
lensWorker.clickhouse.enabled: false
asserts:
- failedTemplate:
errorMessage: lensWorker.clickhouseSecret.name is required
errorMessage: Lens clickhouseSecret.name is required when bundled storage is disabled
- it: keeps storage names valid and uses the same name for the connection
set:
fullnameOverride: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

204
helm/litellm/tests/lens_worker_tests.yaml
View file

@ -1,114 +1,73 @@
suite: Lens worker release and credentials
suite: Independent Lens release and credentials
templates:
- lens/deployment.yaml
- backend/deployment.yaml
- gateway/configmap.yaml
values:
- ./values/required.yaml
set:
fullnameOverride: lens-test
lensWorker.enabled: true
lensWorker.publicUrl: https://traces.example
lensWorker.serviceTokenSecret.name: lens-service
lensWorker.clickhouseSecret.name: lens-storage
tests:
- it: installs the development package for a source commit
template: lens/deployment.yaml
- it: selects a Lens release independently of the gateway
chart:
appVersion: 9.8.7
set:
backend.image.tag: sha-0123456789abcdef
lensWorker.enabled: true
lensWorker.publicUrl: https://traces.example
lensWorker.serviceTokenSecret.name: lens-service
lensWorker.clickhouseSecret.name: lens-storage
backend.image.tag: gateway-9.8.7
lensWorker.image.tag: 2.3.4
asserts:
- equal:
path: spec.template.spec.containers[0].image
value: ghcr.io/berriai/litellm-lens-worker-dev:sha-0123456789abcdef
- it: advertises the development package for standalone source workers
template: backend/deployment.yaml
value: ghcr.io/berriai/lens:2.3.4
- it: retains the pinned Lens chart default when only the gateway changes
chart:
appVersion: 9.8.7
set:
backend.image.tag: sha-0123456789abcdef
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LENS_WORKER_IMAGE
value: ghcr.io/berriai/litellm-lens-worker-dev:sha-0123456789abcdef
- it: preserves an explicit private source image repository
template: lens/deployment.yaml
set:
backend.image.tag: sha-0123456789abcdef
lensWorker.enabled: true
lensWorker.publicUrl: https://traces.example
lensWorker.serviceTokenSecret.name: lens-service
lensWorker.clickhouseSecret.name: lens-storage
lensWorker.image.repository: registry.example/lens-worker
backend.image.tag: gateway-9.8.7
asserts:
- equal:
path: spec.template.spec.containers[0].image
value: registry.example/lens-worker:sha-0123456789abcdef
- it: pins the worker to its approved digest even when its tag changes
template: lens/deployment.yaml
value: ghcr.io/berriai/lens:0.1.0-dev.0
- it: selects development publications explicitly
set:
lensWorker.enabled: true
lensWorker.publicUrl: https://traces.example
lensWorker.serviceTokenSecret.name: lens-service
lensWorker.clickhouseSecret.name: lens-storage
lensWorker.image.tag: replaced-release
lensWorker.image.repository: ghcr.io/berriai/lens-dev
lensWorker.image.tag: sha-0123456789abcdef
asserts:
- equal:
path: spec.template.spec.containers[0].image
value: ghcr.io/berriai/lens-dev:sha-0123456789abcdef
- it: preserves a verified registry mirror and digest
set:
lensWorker.image.repository: registry.example/lens
lensWorker.image.tag: replaced-label
lensWorker.image.digest: sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
asserts:
- equal:
path: spec.template.spec.containers[0].image
value: ghcr.io/berriai/litellm-lens-worker@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
- it: advertises the approved digest to standalone installers
template: backend/deployment.yaml
set:
lensWorker.image.tag: replaced-release
lensWorker.image.digest: sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LENS_WORKER_IMAGE
value: ghcr.io/berriai/litellm-lens-worker@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
- it: refuses a malformed digest instead of falling back to the tag
template: backend/deployment.yaml
value: registry.example/lens@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
- it: rejects a malformed digest
set:
lensWorker.image.digest: sha256:invalid
asserts:
- failedTemplate:
errorMessage: lensWorker.image.digest must be sha256 followed by 64 lowercase hex characters
- it: keeps the worker opt in
template: lens/deployment.yaml
asserts:
- hasDocuments:
count: 0
- it: uses the managed service secret when none is supplied
template: lens/deployment.yaml
errorMessage: Lens image.digest must be sha256 followed by 64 lowercase hex characters
- it: keeps the legacy enablement switch working
set:
lensWorker.enabled: true
lensWorker.publicUrl: https://traces.example
lensWorker.enabled: false
asserts:
- hasDocuments: {count: 0}
- it: keeps isolated local analysis with delegated gateway access
asserts:
- contains:
path: spec.template.spec.containers[0].env
content: {name: LENS_MODE, value: standalone}
- contains:
path: spec.template.spec.containers[0].env
content:
name: LITELLM_LENS_SERVICE_TOKEN
name: LENS_GATEWAY_SECRET
valueFrom:
secretKeyRef:
name: RELEASE-NAME-litellm-lens-service
key: service-token
- it: uses the chart release and a secret without granting Kubernetes access
template: lens/deployment.yaml
chart:
appVersion: v1.2.3
set:
lensWorker.enabled: true
lensWorker.publicUrl: https://traces.example
lensWorker.serviceTokenSecret.name: lens-service
lensWorker.clickhouseSecret.name: lens-storage
asserts:
- equal:
path: spec.template.spec.containers[0].image
value: ghcr.io/berriai/litellm-lens-worker:v1.2.3
- equal:
path: spec.template.spec.containers[0].env[1].valueFrom.secretKeyRef
value:
name: lens-service
key: service-token
secretKeyRef: {name: lens-test-lens-gateway, key: gateway-secret}
- equal:
path: spec.template.spec.automountServiceAccountToken
value: false
@ -117,81 +76,4 @@ tests:
value: true
- equal:
path: spec.template.spec.volumes[0].emptyDir
value:
medium: Memory
sizeLimit: 1Gi
- it: advertises the same private dev image to standalone installers
template: backend/deployment.yaml
set:
lensWorker.image.repository: registry.example/lens-worker
lensWorker.image.tag: branch-main-1234567
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LENS_WORKER_IMAGE
value: registry.example/lens-worker:branch-main-1234567
- it: supports an external gateway and a registry override
template: lens/deployment.yaml
set:
lensWorker.enabled: true
lensWorker.publicUrl: https://traces.example
lensWorker.serviceTokenSecret.name: lens-service
lensWorker.clickhouseSecret.name: lens-storage
lensWorker.url: https://gateway.example/proxy
lensWorker.image.repository: registry.example/lens-worker
lensWorker.image.tag: branch-main-1234567
asserts:
- equal:
path: spec.template.spec.containers[0].image
value: registry.example/lens-worker:branch-main-1234567
- equal:
path: spec.template.spec.containers[0].env[0].value
value: https://gateway.example/proxy
- it: prefixes a numeric chart release with v
template: lens/deployment.yaml
chart:
appVersion: 1.2.3-rc.4
set:
lensWorker.enabled: true
lensWorker.publicUrl: https://traces.example
lensWorker.serviceTokenSecret.name: lens-service
lensWorker.clickhouseSecret.name: lens-storage
asserts:
- equal:
path: spec.template.spec.containers[0].image
value: ghcr.io/berriai/litellm-lens-worker:v1.2.3-rc.4
- it: follows a backend image override when no worker tag is set
template: lens/deployment.yaml
set:
backend.image.tag: branch-main-1234567
lensWorker.enabled: true
lensWorker.publicUrl: https://traces.example
lensWorker.serviceTokenSecret.name: lens-service
lensWorker.clickhouseSecret.name: lens-storage
asserts:
- equal:
path: spec.template.spec.containers[0].image
value: ghcr.io/berriai/litellm-lens-worker:branch-main-1234567
- it: recommends the overridden backend release for standalone installers
template: backend/deployment.yaml
set:
backend.image.tag: v1.2.3-dev.4
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: LENS_WORKER_IMAGE
value: ghcr.io/berriai/litellm-lens-worker:v1.2.3-dev.4
- it: normalizes a numeric backend tag to the published worker tag
template: lens/deployment.yaml
set:
backend.image.tag: 1.2.3-dev.4
lensWorker.enabled: true
lensWorker.publicUrl: https://traces.example
lensWorker.serviceTokenSecret.name: lens-service
lensWorker.clickhouseSecret.name: lens-storage
asserts:
- equal:
path: spec.template.spec.containers[0].image
value: ghcr.io/berriai/litellm-lens-worker:v1.2.3-dev.4
value: {medium: Memory, sizeLimit: 1Gi}

23
helm/litellm/values.yaml
View file

@ -630,12 +630,28 @@ ui:
# Same shape as gateway.topologySpreadConstraints.
topologySpreadConstraints: []
lens:
library: true
lensWorker:
retainResources: false
mode: ""
externalUrl: ""
externalServiceName: ""
gateway:
secretName: ""
secretKey: gateway-secret
standaloneUrl: ""
adminTokenSecret:
name: ""
key: admin-token
extraEnv: []
extraEnvFrom: []
enabled: false
replicaCount: 1
image:
repository: ghcr.io/berriai/litellm-lens-worker
tag: ""
repository: ghcr.io/berriai/lens
tag: "0.1.0-dev.0"
digest: ""
pullPolicy: IfNotPresent
tokenSecret:
@ -646,7 +662,7 @@ lensWorker:
key: service-token
clickhouse:
enabled: true
image: clickhouse/clickhouse-server:26.9.6.6
image: clickhouse/clickhouse-server:26.9.6.6@sha256:eb4870e7ca7ed70c259eebfcfbee6cf797017f6b5436c2926bbbfe3d4d28486e
storage: 20Gi
storageClassName: null
resources:
@ -666,6 +682,7 @@ lensWorker:
annotations: {}
ingress:
enabled: false
path: /v1/
className: ""
host: ""
annotations: {}

15
litellm-proxy-extras/litellm_proxy_extras/migrations/20261008010000_add_daily_gateway_failed_requests/migration.sql Normal file
View file

@ -0,0 +1,15 @@
-- CreateTable
CREATE TABLE IF NOT EXISTS "LiteLLM_DailyGatewayFailedRequests" (
"date" TEXT NOT NULL,
"category" TEXT NOT NULL,
"route" TEXT NOT NULL,
"status_code" INTEGER NOT NULL,
"failed_requests" BIGINT NOT NULL DEFAULT 0,
"created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updated_at" TIMESTAMP(3) NOT NULL,
CONSTRAINT "LiteLLM_DailyGatewayFailedRequests_pkey" PRIMARY KEY ("date","category","route","status_code")
);
-- CreateIndex
CREATE INDEX IF NOT EXISTS "LiteLLM_DailyGatewayFailedRequests_date_idx" ON "LiteLLM_DailyGatewayFailedRequests"("date");

17
litellm-proxy-extras/litellm_proxy_extras/migrations/20261008120000_add_daily_request_errors/migration.sql Normal file
View file

@ -0,0 +1,17 @@
-- CreateTable
CREATE TABLE IF NOT EXISTS "LiteLLM_DailyRequestErrors" (
"date" TEXT NOT NULL,
"api_key" TEXT NOT NULL,
"team_id" TEXT NOT NULL DEFAULT '',
"user_id" TEXT NOT NULL DEFAULT '',
"model_group" TEXT NOT NULL DEFAULT '',
"status_code" INTEGER NOT NULL,
"failed_requests" BIGINT NOT NULL DEFAULT 0,
"created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updated_at" TIMESTAMP(3) NOT NULL,
CONSTRAINT "LiteLLM_DailyRequestErrors_pkey" PRIMARY KEY ("date","api_key","team_id","user_id","model_group","status_code")
);
-- CreateIndex
CREATE INDEX IF NOT EXISTS "LiteLLM_DailyRequestErrors_date_idx" ON "LiteLLM_DailyRequestErrors"("date");

18
litellm-proxy-extras/litellm_proxy_extras/migrations/20261008120000_user_provider_credentials/migration.sql Normal file
View file

@ -0,0 +1,18 @@
-- CreateTable
CREATE TABLE IF NOT EXISTS "LiteLLM_UserProviderCredentials" (
"id" TEXT NOT NULL,
"user_id" TEXT NOT NULL,
"credential_name" TEXT NOT NULL,
"provider" TEXT NOT NULL,
"credential_b64" TEXT NOT NULL,
"created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
"updated_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT "LiteLLM_UserProviderCredentials_pkey" PRIMARY KEY ("id")
);
-- CreateIndex
CREATE UNIQUE INDEX IF NOT EXISTS "LiteLLM_UserProviderCredentials_user_id_credential_name_key" ON "LiteLLM_UserProviderCredentials"("user_id", "credential_name");
-- CreateIndex
CREATE INDEX IF NOT EXISTS "LiteLLM_UserProviderCredentials_credential_name_idx" ON "LiteLLM_UserProviderCredentials"("credential_name");

42
litellm-proxy-extras/litellm_proxy_extras/schema.prisma
View file

@ -453,6 +453,20 @@ model LiteLLM_MCPUserCredentials {
@@unique([user_id, server_id])
}
// Per-user provider connections (e.g. GitHub Copilot OAuth) keyed by credential name
model LiteLLM_UserProviderCredentials {
id String @id @default(uuid())
user_id String
credential_name String
provider String
credential_b64 String
created_at DateTime @default(now()) @map("created_at")
updated_at DateTime @default(now()) @updatedAt @map("updated_at")
@@unique([user_id, credential_name])
@@index([credential_name])
}
// Per-user environment variable values for MCP servers.
// values_b64 is an encrypted JSON object: {VAR_NAME: "value", ...}.
model LiteLLM_MCPUserEnvVars {
@ -1361,6 +1375,34 @@ model LiteLLM_DailyGatewayRequests {
@@index([date])
}
model LiteLLM_DailyGatewayFailedRequests {
date String
category String
route String
status_code Int
failed_requests BigInt @default(0)
created_at DateTime @default(now())
updated_at DateTime @updatedAt
@@id([date, category, route, status_code])
@@index([date])
}
model LiteLLM_DailyRequestErrors {
date String
api_key String
team_id String @default("")
user_id String @default("")
model_group String @default("")
status_code Int
failed_requests BigInt @default(0)
created_at DateTime @default(now())
updated_at DateTime @updatedAt
@@id([date, api_key, team_id, user_id, model_group, status_code])
@@index([date])
}
// Prompt table for storing prompt configurations
model LiteLLM_PromptTable {
id String @id @default(uuid())

10
litellm-rust/AGENTS.md
View file

@ -4,7 +4,7 @@ For diagnostic tracing changes, follow [.agents/skills/rust-tracing/SKILL.md](.a
For string-valued enums and their Serde conversions, follow [.agents/skills/rust-string-enums/SKILL.md](.agents/skills/rust-string-enums/SKILL.md)
For fieldless enums, derive `strum::VariantArray` and use `VARIANTS` instead of a hand-listed `ALL` array; derive Strum string conversions instead of hand-written variant-to-string matches
For fieldless enums, derive `strum::VariantArray` and use `VARIANTS` instead of a hand-listed `ALL` array. Derive Strum string conversions instead of hand-written variant-to-string matches
Use the derived conversions directly (`<&'static str>::from(x)` / `.into()`, `str::parse`) with no `as_str`/`parse` wrapper that only delegates, and spell each variant with explicit `#[strum(serialize = "...")]` instead of `serialize_all`
@ -15,20 +15,20 @@ Use the derived conversions directly (`<&'static str>::from(x)` / `.into()`, `st
- A test that only uses the crate's public API lives in `crates/<crate>/tests/<subject>.rs`, next to `src/`
- Split a mixed test file along that line instead of widening visibility to move it
- A test for another crate's item belongs in that crate, not in a downstream one
- Never set `autotests = false` or hand-list `[[test]]` targets; every file directly under `tests/` is discovered by cargo, and a shared helper goes in `tests/<name>/mod.rs` or `tests/<subject>/support.rs` so it is not picked up as a test crate of its own
- Never set `autotests = false` or hand-list `[[test]]` targets. Every file directly under `tests/` is discovered by cargo, and a shared helper goes in `tests/<name>/mod.rs` or `tests/<subject>/support.rs` so it is not picked up as a test crate of its own
## Test fixtures and cases
Use [`#[rstest]`](https://docs.rs/rstest/latest/rstest/attr.rstest.html) for new and updated tests and [`#[fixture]`](https://docs.rs/rstest/latest/rstest/attr.fixture.html) for reusable setup, injected through typed test arguments. Express input variations as named `#[case::name(...)]` cases instead of loops or duplicated tests so each failure identifies its case. Keep behavior assertions in the test body and fixtures focused on setup. Use the workspace `rstest` dependency
- Never loop over inputs (`for`, `.iter().for_each`, `.all`) inside a test body; give each input its own `#[case::name(...)]`, or use `#[values(...)]` for a cross product
- Never loop over inputs (`for`, `.iter().for_each`, `.all`) inside a test body. Give each input its own `#[case::name(...)]`, or use `#[values(...)]` for a cross product
- Exception: a test pinning a Rust table against a repo-owned data file (for example `include_str!` of a JSON config) may iterate that file's entries
## Error definitions
- A crate's errors live in `src/error.rs`, defined with `thiserror`, and re-exported from `lib.rs`
- Put message templates in the variant's `#[error(...)]` declaration. Callers pass only the small typed arguments needed to fill them, never `Error::Variant(format!(...))` or a preformatted message. Keep the smallest set of neutral variants that callers need to distinguish; different wording or providers do not justify new variants
- Default to one top-level `Error` enum per crate, with one variant per failure mode and a `#[error(...)]` message on each. A failure mode is something a caller handles differently (phase, status code, retry, a message Python parity pins exactly); failures no caller tells apart share one variant and differ only in its message
- Put message templates in the variant's `#[error(...)]` declaration. Callers pass only the small typed arguments needed to fill them, never `Error::Variant(format!(...))` or a preformatted message. Keep the smallest set of neutral variants that callers need to distinguish. Different wording or providers do not justify new variants
- Default to one top-level `Error` enum per crate, with one variant per failure mode and a `#[error(...)]` message on each. A failure mode is something a caller handles differently (phase, status code, retry, a message Python parity pins exactly). Failures no caller tells apart share one variant and differ only in its message
- Keep shared error enums minimal and provider-neutral. Provider names, credential types, configuration fields, and setup guidance belong in caller-supplied data, not dedicated variants or hardcoded shared messages. Reuse a variant for the same failure mode across providers, such as `MissingApiBase { provider: "Azure", guidance: "..." }`. An exact parity message does not justify a provider-specific variant when caller-supplied context can preserve it
- Wrap a lower-level error as a variant with `#[from]` or `#[source]` instead of flattening it to a string
- Exception: split into separate types when different functions fail in disjoint ways, especially when different callers see them. A shared enum would force every caller to match variants its function can never return

401
litellm-rust/Cargo.lock generated
View file

@ -97,53 +97,6 @@ version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "03918c3dbd7701a85c6b9887732e2921175f26c350b4563841d0958c21d57e6d"
[[package]]
name = "askama"
version = "0.16.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6024d73179f43f15ccd2b881bfea6fee7f3a46ec53f33b52210dea749ebebaa4"
dependencies = [
"askama_macros",
"itoa",
"percent-encoding",
"serde",
"serde_json",
]
[[package]]
name = "askama_derive"
version = "0.16.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "071ee5ebf2138e3ad180e0aacf6940c2cab5e6d8333741d9925c7bee2b153f39"
dependencies = [
"askama_parser",
"memchr",
"proc-macro2",
"quote",
"rustc-hash",
"syn 3.0.6",
]
[[package]]
name = "askama_macros"
version = "0.16.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "643e1c7cbb6aec1d920332fe51a7c0d8219e273dcb8602db03f5263e4d16487b"
dependencies = [
"askama_derive",
]
[[package]]
name = "askama_parser"
version = "0.16.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2c5ae75772275d268b03ab8bdccdd12117b6169ee23256942b34e46c9f476583"
dependencies = [
"rustc-hash",
"unicode-ident",
"winnow 1.0.4",
]
[[package]]
name = "asn1-rs"
version = "0.7.2"
@ -1100,6 +1053,15 @@ version = "0.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dc0b364ead1874514c8c2855ab558056ebfeb775653e7ae45ff72f28f8f3166c"
[[package]]
name = "bs58"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bf88ba1141d185c399bee5288d850d63b8369520c1eafc32a0430b5b6c287bf4"
dependencies = [
"tinyvec",
]
[[package]]
name = "bstr"
version = "1.13.1"
@ -1321,18 +1283,6 @@ version = "0.4.33"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e8ccc4ea9f6acc32d102c0f6d471d11d913ad15f20c04de743374861fa1d414"
[[package]]
name = "const-hex"
version = "1.19.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0e59eef12462b0f9b0a3620219be5d639afd79fe39dff0a42c3997061f9298b4"
dependencies = [
"cfg-if",
"cpufeatures 0.2.17",
"proptest",
"serde_core",
]
[[package]]
name = "const-oid"
version = "0.9.6"
@ -1608,12 +1558,12 @@ dependencies = [
[[package]]
name = "darling"
version = "0.21.3"
version = "0.23.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9cdf337090841a411e2a7f3deb9187445851f91b309c0c0a29e05f74a00a48c0"
checksum = "25ae13da2f202d56bd7f91c25fba009e7717a1e4a1cc98a76d844b65ae912e9d"
dependencies = [
"darling_core 0.21.3",
"darling_macro 0.21.3",
"darling_core 0.23.0",
"darling_macro 0.23.0",
]
[[package]]
@ -1632,11 +1582,10 @@ dependencies = [
[[package]]
name = "darling_core"
version = "0.21.3"
version = "0.23.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1247195ecd7e3c85f83c8d2a366e4210d588e802133e1e355180a9870b517ea4"
checksum = "9865a50f7c335f53564bb694ef660825eb8610e0a53d3e11bf1b0d3df31e03b0"
dependencies = [
"fnv",
"ident_case",
"proc-macro2",
"quote",
@ -1657,11 +1606,11 @@ dependencies = [
[[package]]
name = "darling_macro"
version = "0.21.3"
version = "0.23.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d38308df82d1080de0afee5d069fa14b0326a88c14f15c5ccda35b4a6c414c81"
checksum = "ac3984ec7bd6cfa798e62b4a642426a5be0e68f9401cfc2a01e3fa9ea2fcdb8d"
dependencies = [
"darling_core 0.21.3",
"darling_core 0.23.0",
"quote",
"syn 2.0.119",
]
@ -2431,9 +2380,9 @@ dependencies = [
"http-body-util",
"hyper 1.10.1",
"lazy_static",
"opentelemetry 0.32.0",
"opentelemetry",
"opentelemetry-semantic-conventions",
"opentelemetry_sdk 0.32.1",
"opentelemetry_sdk",
"percent-encoding",
"pin-project",
"prost",
@ -3429,10 +3378,12 @@ dependencies = [
"moka",
"reqwest 0.12.28",
"rstest",
"serde",
"serde_json",
"sha2 0.10.9",
"thiserror 2.0.19",
"tokio",
"veil",
]
[[package]]
@ -3461,9 +3412,12 @@ dependencies = [
"litellm-auth-types",
"moka",
"rstest",
"serde",
"serde_json",
"sha2 0.10.9",
"tempfile",
"tokio",
"veil",
]
[[package]]
@ -3472,6 +3426,7 @@ version = "0.1.0"
dependencies = [
"rstest",
"serde",
"serde_json",
"subtle",
"thiserror 2.0.19",
"tokio",
@ -3691,6 +3646,7 @@ name = "litellm-config"
version = "0.1.0"
dependencies = [
"litellm-auth-types",
"litellm-router-types",
"rstest",
"serde",
"serde_json",
@ -4075,6 +4031,7 @@ dependencies = [
"litellm-inference-testing",
"litellm-llms",
"litellm-llms-types",
"litellm-router-types",
"litellm-secrets",
"litellm-tracing",
"reqwest 0.12.28",
@ -4175,45 +4132,6 @@ dependencies = [
"wiremock",
]
[[package]]
name = "litellm-lens"
version = "0.1.0"
dependencies = [
"axum",
"bytes",
"chrono",
"flate2",
"futures-util",
"http 1.4.2",
"jsonschema",
"libc",
"litellm-http",
"litellm-storage-clickhouse",
"litellm-traces",
"litellm-traces-cache",
"litellm-traces-clickhouse",
"litellm-tracing",
"prettyplease",
"prost",
"reqwest 0.12.28",
"rstest",
"serde",
"serde_json",
"sha2 0.10.9",
"subtle",
"syn 2.0.119",
"tempfile",
"thiserror 2.0.19",
"tokio",
"tower-http",
"tracing",
"typify",
"unicode-casefold",
"url",
"uuid",
"wiremock",
]
[[package]]
name = "litellm-llms"
version = "0.1.0"
@ -4234,6 +4152,7 @@ dependencies = [
"litellm-http",
"litellm-llms-types",
"litellm-python-compat",
"litellm-router-types",
"litellm-secrets",
"reqwest 0.12.28",
"rstest",
@ -4305,14 +4224,13 @@ dependencies = [
"litellm-inference-transcription",
"litellm-llms",
"litellm-llms-types",
"litellm-router-types",
"litellm-secrets",
"litellm-secrets-aws",
"litellm-secrets-types",
"litellm-spend-clickhouse",
"litellm-storage-clickhouse",
"litellm-token-counter",
"litellm-traces",
"litellm-traces-cache",
"litellm-traces-clickhouse",
"litellm-tracing",
"prost",
"pyo3",
@ -4355,9 +4273,21 @@ dependencies = [
"litellm-config",
"litellm-inference",
"litellm-inference-messages",
"litellm-router-types",
"rstest",
]
[[package]]
name = "litellm-router-types"
version = "0.1.0"
dependencies = [
"litellm-auth-types",
"rstest",
"serde",
"serde_json",
"serde_with",
]
[[package]]
name = "litellm-secrets"
version = "0.1.0"
@ -4513,6 +4443,26 @@ dependencies = [
"veil",
]
[[package]]
name = "litellm-spend-clickhouse"
version = "0.1.0"
dependencies = [
"flate2",
"litellm-http",
"litellm-storage-clickhouse",
"rstest",
"serde",
"serde_json",
"sha2 0.10.9",
"sqlx",
"testcontainers-modules",
"thiserror 2.0.19",
"time",
"tokio",
"uuid",
"wiremock",
]
[[package]]
name = "litellm-storage-clickhouse"
version = "0.1.0"
@ -4608,76 +4558,6 @@ dependencies = [
"tiktoken-rs",
]
[[package]]
name = "litellm-traces"
version = "0.1.0"
dependencies = [
"askama",
"base64 0.22.1",
"criterion",
"indexmap 2.14.0",
"litellm-llms-types",
"macro_rules_attribute",
"opentelemetry-proto",
"prost",
"rstest",
"schemars 1.2.2",
"serde",
"serde_json",
"serde_with",
"sha2 0.10.9",
"strum",
"thiserror 2.0.19",
"time",
]
[[package]]
name = "litellm-traces-cache"
version = "0.1.0"
dependencies = [
"base64 0.22.1",
"litellm-traces",
"moka",
"rstest",
"serde",
"serde_json",
"sha2 0.10.9",
"thiserror 2.0.19",
"time",
"tokio",
"tracing",
]
[[package]]
name = "litellm-traces-clickhouse"
version = "0.1.0"
dependencies = [
"askama",
"flate2",
"futures-util",
"hmac 0.12.1",
"jsonschema",
"litellm-http",
"litellm-storage-clickhouse",
"litellm-traces",
"litellm-traces-cache",
"macro_rules_attribute",
"moka",
"rstest",
"schemars 1.2.2",
"serde",
"serde_json",
"sha2 0.10.9",
"sqlx",
"strum",
"testcontainers-modules",
"thiserror 2.0.19",
"time",
"tokio",
"url",
"wiremock",
]
[[package]]
name = "litellm-tracing"
version = "0.1.0"
@ -5087,36 +4967,6 @@ dependencies = [
"tracing",
]
[[package]]
name = "opentelemetry"
version = "0.33.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6cdb0b1b267eb9db3331b434ed9ddab10d50e280a9adf9d13e5233e2002b61b5"
dependencies = [
"futures-core",
"futures-sink",
"js-sys",
"pin-project-lite",
"thiserror 2.0.19",
"tracing",
]
[[package]]
name = "opentelemetry-proto"
version = "0.33.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "25da1ac11a0aeccf38d7f77ee0348715adaf8340f65ad46c94a02c6b20e2f65d"
dependencies = [
"base64 0.22.1",
"const-hex",
"opentelemetry 0.33.0",
"opentelemetry_sdk 0.33.0",
"prost",
"serde",
"tonic",
"tonic-prost",
]
[[package]]
name = "opentelemetry-semantic-conventions"
version = "0.32.1"
@ -5132,23 +4982,7 @@ dependencies = [
"futures-channel",
"futures-executor",
"futures-util",
"opentelemetry 0.32.0",
"percent-encoding",
"portable-atomic",
"rand 0.9.5",
"thiserror 2.0.19",
]
[[package]]
name = "opentelemetry_sdk"
version = "0.33.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cb39533d9d1c912123efd7d41d7e0c29d16917b60ce15b4c8d87cb1af7f67520"
dependencies = [
"futures-channel",
"futures-executor",
"futures-util",
"opentelemetry 0.33.0",
"opentelemetry",
"percent-encoding",
"portable-atomic",
"rand 0.9.5",
@ -5454,16 +5288,6 @@ dependencies = [
"zerocopy",
]
[[package]]
name = "prettyplease"
version = "0.2.37"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b"
dependencies = [
"proc-macro2",
"syn 2.0.119",
]
[[package]]
name = "primeorder"
version = "0.13.6"
@ -6030,16 +5854,6 @@ version = "0.8.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4"
[[package]]
name = "regress"
version = "0.11.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "158a764437582235e3501f683b93a0a6f8d825d04a789dbe5ed30b8799b8908a"
dependencies = [
"hashbrown 0.16.1",
"memchr",
]
[[package]]
name = "relative-path"
version = "1.9.3"
@ -6486,18 +6300,6 @@ dependencies = [
"parking_lot",
]
[[package]]
name = "schemars"
version = "0.8.22"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3fbf2ae1b8bc8e02df939598064d22402220cd5bbcca1c76f7d6a310974d5615"
dependencies = [
"dyn-clone",
"schemars_derive 0.8.22",
"serde",
"serde_json",
]
[[package]]
name = "schemars"
version = "0.9.0"
@ -6520,23 +6322,11 @@ dependencies = [
"dyn-clone",
"indexmap 2.14.0",
"ref-cast",
"schemars_derive 1.2.2",
"schemars_derive",
"serde",
"serde_json",
]
[[package]]
name = "schemars_derive"
version = "0.8.22"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32e265784ad618884abaea0600a9adf15393368d840e0222d101a072f3f7534d"
dependencies = [
"proc-macro2",
"quote",
"serde_derive_internals 0.29.1",
"syn 2.0.119",
]
[[package]]
name = "schemars_derive"
version = "1.2.2"
@ -6545,7 +6335,7 @@ checksum = "d98c67716b46af2f0b8cf752abc930f6f9aecfbf671ecfb531db8a31dbe4e2ba"
dependencies = [
"proc-macro2",
"quote",
"serde_derive_internals 0.30.0",
"serde_derive_internals",
"syn 3.0.6",
]
@ -6651,17 +6441,6 @@ dependencies = [
"syn 3.0.6",
]
[[package]]
name = "serde_derive_internals"
version = "0.29.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "18d26a20a969b9e3fdf2fc2d9f21eda6c40e2de84c9408bb5d3b05d499aae711"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
]
[[package]]
name = "serde_derive_internals"
version = "0.30.0"
@ -6732,11 +6511,12 @@ dependencies = [
[[package]]
name = "serde_with"
version = "3.16.1"
version = "3.21.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4fa237f2807440d238e0364a218270b98f767a00d3dada77b1c53ae88940e2e7"
checksum = "76a5c54c7310e7b8b9577c286d7e399ddd876c3e12b3ed917a8aabc4b96e9e8c"
dependencies = [
"base64 0.22.1",
"bs58",
"chrono",
"hex",
"indexmap 1.9.3",
@ -6751,11 +6531,11 @@ dependencies = [
[[package]]
name = "serde_with_macros"
version = "3.16.1"
version = "3.21.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "52a8e3ca0ca629121f70ab50f95249e5a6f925cc0f6ffe8256c45b728875706c"
checksum = "84d57bc0c8b9a17920c178daa6bb924850d54a9c97ab45194bb8c17ad66bb660"
dependencies = [
"darling 0.21.3",
"darling 0.23.0",
"proc-macro2",
"quote",
"syn 2.0.119",
@ -7926,7 +7706,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "adbc64cba7137545b8044cb1fe9814f7aacf3c6b5f9b45be8bb5db538befdb26"
dependencies = [
"js-sys",
"opentelemetry 0.32.0",
"opentelemetry",
"tracing",
"tracing-core",
"tracing-subscriber",
@ -8028,35 +7808,6 @@ dependencies = [
"syn 2.0.119",
]
[[package]]
name = "typify"
version = "0.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b715573a376585888b742ead9be5f4826105e622169180662e2c81bed4a149c3"
dependencies = [
"typify-impl",
]
[[package]]
name = "typify-impl"
version = "0.6.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fa7b026f540b148b81043c720889dbb942b08659aa8a43f624ac4f04dbfc1861"
dependencies = [
"heck",
"log",
"proc-macro2",
"quote",
"regress",
"schemars 0.8.22",
"semver",
"serde",
"serde_json",
"syn 2.0.119",
"thiserror 2.0.19",
"unicode-ident",
]
[[package]]
name = "ucd-trie"
version = "0.1.7"
@ -8081,12 +7832,6 @@ version = "0.3.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5c1cb5db39152898a79168971543b1cb5020dff7fe43c8dc468b0885f5e29df5"
[[package]]
name = "unicode-casefold"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b7f66b1c8f8caa2ab31dc6d3f35386f16efdab89668f93411e565ac368908e8f"
[[package]]
name = "unicode-general-category"
version = "1.1.0"

7
litellm-rust/Cargo.toml
View file

@ -11,10 +11,9 @@ repository = "https://github.com/BerriAI/litellm"
[workspace.dependencies]
litellm-config = { path = "crates/config" }
litellm-router = { path = "crates/router" }
litellm-router-types = { path = "crates/router-types" }
litellm-tracing = { path = "crates/tracing" }
litellm-traces = { path = "crates/traces" }
litellm-traces-cache = { path = "crates/traces-cache" }
litellm-traces-clickhouse = { path = "crates/traces-clickhouse" }
litellm-spend-clickhouse = { path = "crates/spend-clickhouse" }
litellm-storage-clickhouse = { path = "crates/storage-clickhouse" }
litellm-inference = { path = "crates/inference" }
litellm-inference-transcription = { path = "crates/inference-transcription" }
@ -100,7 +99,7 @@ vaultrs = { version = "=0.8.0", default-features = false, features = ["rustls"]
rustls-native-certs = "0.8"
serde = { version = "1.0", features = ["derive"] }
serde_json = { version = "1.0", features = ["float_roundtrip"] }
serde_with = { version = "=3.16.1", default-features = false, features = ["std", "macros"] }
serde_with = { version = "=3.21.0", default-features = false, features = ["std", "macros"] }
sha2 = "0.10"
syn = { version = "2", default-features = false }
sqlx = { version = "0.9.0", default-features = false, features = ["json", "macros", "postgres", "runtime-tokio", "chrono", "tls-rustls-ring-native-roots"] }

4
litellm-rust/crates/auth-aws/Cargo.toml
View file

@ -10,9 +10,11 @@ litellm-auth-types.workspace = true
litellm-http.workspace = true
moka = { workspace = true, features = ["sync"] }
serde_json.workspace = true
serde.workspace = true
serde_json = { workspace = true, features = ["preserve_order"] }
sha2.workspace = true
thiserror.workspace = true
veil.workspace = true
aws-config = { version = "1.9.0", default-features = false, features = ["rustls", "rt-tokio"] }
aws-credential-types = { version = "1.3.0", features = ["hardcoded-credentials"] }

164
litellm-rust/crates/auth-aws/src/aws.rs
View file

@ -3,7 +3,6 @@ use std::time::Duration;
use std::time::{SystemTime, UNIX_EPOCH};
use moka::sync::Cache;
use serde_json::{Map, Value};
use sha2::{Digest, Sha256};
use aws_credential_types::Credentials;
@ -14,10 +13,12 @@ use aws_sigv4::http_request::{
use aws_sigv4::sign::v4;
use aws_smithy_runtime_api::client::identity::Identity;
use litellm_auth_types::AwsParams;
use super::Error;
use super::constants::{
AWS_ACCESS_KEY_ID, AWS_EXTERNAL_ID, AWS_PROFILE_NAME, AWS_REGION, AWS_REGION_NAME,
AWS_ROLE_ARN, AWS_ROLE_NAME, AWS_SECRET_ACCESS_KEY, AWS_SESSION_NAME, AWS_SESSION_TOKEN,
AWS_ACCESS_KEY_ID, AWS_EXTERNAL_ID, AWS_PROFILE_NAME, AWS_REGION_NAME, AWS_ROLE_ARN,
AWS_ROLE_NAME, AWS_SECRET_ACCESS_KEY, AWS_SESSION_NAME, AWS_SESSION_TOKEN,
AWS_SIGNED_HEADER_NAMES, AWS_STS_ENDPOINT, AWS_WEB_IDENTITY_TOKEN, AWS_WEB_IDENTITY_TOKEN_FILE,
DEFAULT_BEDROCK_REGION, DEFAULT_SESSION_NAME_PREFIX, SIGV4_COMPUTED_HEADER_NAMES,
};
@ -543,50 +544,39 @@ fn is_bedrock_region(value: &str) -> bool {
/// region, then the environment. Each service decides what a missing one means.
pub fn resolve_aws_region(
model_region: Option<&str>,
optional_params: &Map<String, Value>,
params: &AwsParams,
env_lookup: &dyn Fn(&str) -> Option<String>,
) -> Option<String> {
optional_params
.get("aws_region_name")
.and_then(Value::as_str)
.or(model_region)
.map(str::to_string)
.or_else(|| env_lookup(AWS_REGION_NAME))
.or_else(|| env_lookup(AWS_REGION))
params
.resolve(&AwsParams::REGION, &|_| None)
.or_else(|| model_region.map(str::to_string))
.or_else(|| AwsParams::REGION.resolve(&|_| None, env_lookup))
}
pub fn resolve_bedrock_region(
model_region: Option<&str>,
optional_params: &Map<String, Value>,
params: &AwsParams,
env_lookup: &dyn Fn(&str) -> Option<String>,
) -> String {
resolve_aws_region(model_region, optional_params, env_lookup)
resolve_aws_region(model_region, params, env_lookup)
.unwrap_or_else(|| DEFAULT_BEDROCK_REGION.to_string())
}
pub fn aws_auth_config(
optional_params: &Map<String, Value>,
params: &AwsParams,
env_lookup: &dyn Fn(&str) -> Option<String>,
) -> AwsAuthConfig {
let value = |key: &str| {
optional_params
.get(key)
.and_then(Value::as_str)
.map(str::to_string)
};
let env = |key: &str| env_lookup(key);
AwsAuthConfig {
access_key_id: value("aws_access_key_id").or_else(|| env("AWS_ACCESS_KEY_ID")),
secret_access_key: value("aws_secret_access_key").or_else(|| env("AWS_SECRET_ACCESS_KEY")),
session_token: value("aws_session_token").or_else(|| env("AWS_SESSION_TOKEN")),
region_name: value("aws_region_name").or_else(|| env(AWS_REGION_NAME)),
session_name: value("aws_session_name").or_else(|| env("AWS_SESSION_NAME")),
profile_name: value("aws_profile_name").or_else(|| env("AWS_PROFILE_NAME")),
role_name: value("aws_role_name").or_else(|| env("AWS_ROLE_NAME")),
web_identity_token: value("aws_web_identity_token")
.or_else(|| env("AWS_WEB_IDENTITY_TOKEN")),
sts_endpoint: value("aws_sts_endpoint").or_else(|| env("AWS_STS_ENDPOINT")),
external_id: value("aws_external_id").or_else(|| env("AWS_EXTERNAL_ID")),
access_key_id: params.resolve(&AwsParams::ACCESS_KEY_ID, env_lookup),
secret_access_key: params.resolve(&AwsParams::SECRET_ACCESS_KEY, env_lookup),
session_token: params.resolve(&AwsParams::SESSION_TOKEN, env_lookup),
region_name: params.resolve(&AwsParams::REGION, env_lookup),
session_name: params.resolve(&AwsParams::SESSION_NAME, env_lookup),
profile_name: params.resolve(&AwsParams::PROFILE_NAME, env_lookup),
role_name: params.resolve(&AwsParams::ROLE_NAME, env_lookup),
web_identity_token: params.resolve(&AwsParams::WEB_IDENTITY_TOKEN, env_lookup),
sts_endpoint: params.resolve(&AwsParams::STS_ENDPOINT, env_lookup),
external_id: params.resolve(&AwsParams::EXTERNAL_ID, env_lookup),
}
}
@ -599,13 +589,10 @@ pub enum AwsCredentialSource {
}
impl AwsCredentialSource {
pub fn from_params(
optional_params: &Map<String, Value>,
env_lookup: &dyn Fn(&str) -> Option<String>,
) -> Self {
match host_supplied_credentials(optional_params) {
pub fn from_params(params: &AwsParams, env_lookup: &dyn Fn(&str) -> Option<String>) -> Self {
match host_supplied_credentials(params) {
Some(credentials) => Self::HostSupplied(credentials),
None => Self::Chain(aws_auth_config(optional_params, env_lookup)),
None => Self::Chain(aws_auth_config(params, env_lookup)),
}
}
@ -629,20 +616,20 @@ impl AwsCredentialSource {
/// state, where an unrelated `AWS_ROLE_NAME` or `AWS_PROFILE_NAME` in the
/// environment outranks explicit keys in [`classify_auth`] and the two sides
/// would sign as different principals.
pub fn host_supplied_credentials(optional_params: &Map<String, Value>) -> Option<Credentials> {
let value = |key: &str| {
optional_params
.get(key)
.and_then(Value::as_str)
pub fn host_supplied_credentials(params: &AwsParams) -> Option<Credentials> {
let value = |value: &Option<String>| {
value
.as_deref()
.map(str::trim)
.filter(|value| !value.is_empty())
.map(str::to_string)
};
let access_key_id = value("aws_access_key_id")?;
let secret_access_key = value("aws_secret_access_key")?;
let access_key_id = value(&params.aws_access_key_id)?;
let secret_access_key = value(&params.aws_secret_access_key)?;
Some(Credentials::new(
access_key_id,
secret_access_key,
value("aws_session_token").map(str::to_string),
value(&params.aws_session_token),
None,
"litellm-host-supplied",
))
@ -651,7 +638,7 @@ pub fn host_supplied_credentials(optional_params: &Map<String, Value>) -> Option
#[cfg(test)]
mod tests {
use super::*;
use crate::constants::BEDROCK_SERVICE;
use crate::constants::{AWS_REGION, BEDROCK_SERVICE};
fn no_env(_: &str) -> Option<String> {
None
@ -667,37 +654,80 @@ mod tests {
recorded.lock().unwrap().insert(name.to_string());
None
};
resolve_aws_region(None, &Map::new(), &env);
aws_auth_config(&Map::new(), &env);
resolve_aws_region(None, &AwsParams::default(), &env);
aws_auth_config(&AwsParams::default(), &env);
assert!(
seen.lock()
.unwrap()
.iter()
.all(|name| crate::constants::SECRET_NAMES.contains(&name.as_str()))
.all(|name| AwsParams::secret_names().contains(&name.as_str()))
);
}
#[test]
fn a_region_comes_from_the_call_then_the_model_then_the_environment() {
let params = Map::from_iter([("aws_region_name".to_string(), Value::from("eu-west-1"))]);
let region_name = |key: &str| (key == AWS_REGION_NAME).then(|| "ap-south-1".to_string());
let region = |key: &str| (key == AWS_REGION).then(|| "sa-east-1".to_string());
let resolved = [
resolve_aws_region(Some("us-east-2"), &params, &region_name),
resolve_aws_region(Some("us-east-2"), &Map::new(), &region_name),
resolve_aws_region(None, &Map::new(), &region_name),
resolve_aws_region(None, &Map::new(), &region),
resolve_aws_region(None, &Map::new(), &no_env),
];
#[rstest::rstest]
#[case::param_wins(Some("from-params"), &[(AWS_SECRET_ACCESS_KEY, "from-env")], Some("from-params"))]
#[case::blank_param_falls_to_environment(Some(" "), &[(AWS_SECRET_ACCESS_KEY, "from-env")], Some("from-env"))]
#[case::absent_param_falls_to_environment(None, &[(AWS_SECRET_ACCESS_KEY, "from-env")], Some("from-env"))]
#[case::nothing(None, &[], None)]
fn auth_config_reads_each_spec_from_params_then_the_environment(
#[case] aws_secret_access_key: Option<&str>,
#[case] environment: &[(&str, &str)],
#[case] expected: Option<&str>,
) {
let params = AwsParams {
aws_secret_access_key: aws_secret_access_key.map(str::to_string),
..AwsParams::default()
};
let env = |key: &str| {
environment
.iter()
.find(|(name, _)| *name == key)
.map(|(_, value)| value.to_string())
};
assert_eq!(
resolved.map(|region| region.unwrap_or_else(|| "none".into())),
["eu-west-1", "us-east-2", "ap-south-1", "sa-east-1", "none"]
aws_auth_config(&params, &env).secret_access_key.as_deref(),
expected
);
assert_eq!(
resolve_bedrock_region(None, &Map::new(), &no_env),
DEFAULT_BEDROCK_REGION
aws_auth_config(&AwsParams::default(), &|key: &str| (key == AWS_REGION)
.then(|| "sa-east-1".to_string()))
.region_name
.as_deref(),
Some("sa-east-1")
);
}
#[rstest::rstest]
#[case::call_params(Some("us-east-2"), Some("eu-west-1"), &[(AWS_REGION_NAME, "ap-south-1")], Some("eu-west-1"))]
#[case::model_region(Some("us-east-2"), None, &[(AWS_REGION_NAME, "ap-south-1")], Some("us-east-2"))]
#[case::aws_region_name(None, None, &[(AWS_REGION_NAME, "ap-south-1")], Some("ap-south-1"))]
#[case::aws_region(None, None, &[(AWS_REGION, "sa-east-1")], Some("sa-east-1"))]
#[case::nothing(None, None, &[], None)]
fn a_region_comes_from_the_call_then_the_model_then_the_environment(
#[case] model_region: Option<&str>,
#[case] aws_region_name: Option<&str>,
#[case] environment: &[(&str, &str)],
#[case] expected: Option<&str>,
) {
let params = AwsParams {
aws_region_name: aws_region_name.map(str::to_string),
..AwsParams::default()
};
let env = |key: &str| {
environment
.iter()
.find(|(name, _)| *name == key)
.map(|(_, value)| value.to_string())
};
assert_eq!(
resolve_aws_region(model_region, &params, &env).as_deref(),
expected
);
assert_eq!(
resolve_bedrock_region(model_region, &params, &env),
expected.unwrap_or(DEFAULT_BEDROCK_REGION)
);
}

40
litellm-rust/crates/auth-aws/src/constants.rs
View file

@ -1,33 +1,21 @@
pub const AWS_ACCESS_KEY_ID: &str = "AWS_ACCESS_KEY_ID";
pub const AWS_SECRET_ACCESS_KEY: &str = "AWS_SECRET_ACCESS_KEY";
pub const AWS_SESSION_TOKEN: &str = "AWS_SESSION_TOKEN";
pub const AWS_REGION_NAME: &str = "AWS_REGION_NAME";
pub const AWS_REGION: &str = "AWS_REGION";
use litellm_auth_types::AwsParams;
pub const AWS_ACCESS_KEY_ID: &str = AwsParams::ACCESS_KEY_ID.env[0];
pub const AWS_SECRET_ACCESS_KEY: &str = AwsParams::SECRET_ACCESS_KEY.env[0];
pub const AWS_SESSION_TOKEN: &str = AwsParams::SESSION_TOKEN.env[0];
pub const AWS_REGION_NAME: &str = AwsParams::REGION.env[0];
pub const AWS_REGION: &str = AwsParams::REGION.env[1];
pub const AWS_DEFAULT_REGION: &str = "AWS_DEFAULT_REGION";
pub const AWS_BEDROCK_RUNTIME_ENDPOINT: &str = "AWS_BEDROCK_RUNTIME_ENDPOINT";
pub const AWS_SESSION_NAME: &str = "AWS_SESSION_NAME";
pub const AWS_PROFILE_NAME: &str = "AWS_PROFILE_NAME";
pub const AWS_ROLE_NAME: &str = "AWS_ROLE_NAME";
pub const AWS_WEB_IDENTITY_TOKEN: &str = "AWS_WEB_IDENTITY_TOKEN";
pub const AWS_BEDROCK_RUNTIME_ENDPOINT: &str = AwsParams::BEDROCK_RUNTIME_ENDPOINT.env[0];
pub const AWS_SESSION_NAME: &str = AwsParams::SESSION_NAME.env[0];
pub const AWS_PROFILE_NAME: &str = AwsParams::PROFILE_NAME.env[0];
pub const AWS_ROLE_NAME: &str = AwsParams::ROLE_NAME.env[0];
pub const AWS_WEB_IDENTITY_TOKEN: &str = AwsParams::WEB_IDENTITY_TOKEN.env[0];
pub const AWS_ROLE_ARN: &str = "AWS_ROLE_ARN";
pub const AWS_WEB_IDENTITY_TOKEN_FILE: &str = "AWS_WEB_IDENTITY_TOKEN_FILE";
pub const AWS_STS_ENDPOINT: &str = "AWS_STS_ENDPOINT";
pub const AWS_EXTERNAL_ID: &str = "AWS_EXTERNAL_ID";
pub const AWS_STS_ENDPOINT: &str = AwsParams::STS_ENDPOINT.env[0];
pub const AWS_EXTERNAL_ID: &str = AwsParams::EXTERNAL_ID.env[0];
pub const AWS_BEARER_TOKEN_BEDROCK: &str = "AWS_BEARER_TOKEN_BEDROCK";
pub const SECRET_NAMES: &[&str] = &[
AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY,
AWS_SESSION_TOKEN,
AWS_REGION_NAME,
AWS_REGION,
AWS_SESSION_NAME,
AWS_PROFILE_NAME,
AWS_ROLE_NAME,
AWS_WEB_IDENTITY_TOKEN,
AWS_STS_ENDPOINT,
AWS_EXTERNAL_ID,
];
/// Headers SigV4 covers, beyond the `x-amz-` / `x-amzn-` prefixes. Mirrors
/// Python's `_filter_headers_for_aws_signature` allowlist.
pub const AWS_SIGNED_HEADER_NAMES: &[&str] = &[

3
litellm-rust/crates/auth-gcp/Cargo.toml
View file

@ -12,9 +12,11 @@ google-sdk = ["dep:google-cloud-auth", "dep:http"]
litellm-auth-types.workspace = true
moka.workspace = true
serde.workspace = true
serde_json.workspace = true
sha2.workspace = true
tokio.workspace = true
veil.workspace = true
gcp_auth = "0.12.7"
google-cloud-auth = { workspace = true, optional = true }
@ -22,3 +24,4 @@ http = { workspace = true, optional = true }
[dev-dependencies]
rstest.workspace = true
tempfile.workspace = true

274
litellm-rust/crates/auth-gcp/src/auth.rs Normal file
View file

@ -0,0 +1,274 @@
use std::{future::Future, path::Path, pin::Pin, sync::Arc};
use gcp_auth::{CustomServiceAccount, TokenProvider};
use litellm_auth_types::{CredentialPlacement, Error, ErrorDetail, http::apply_credential};
use moka::future::Cache;
use serde_json::Value;
use sha2::{Digest, Sha256};
use crate::{
config::{
CredentialSource, VertexConfig, credential_source, get_vertex_ai_project, non_empty_env,
},
constants::{
CLOUD_PLATFORM_SCOPE, GOOGLE_OAUTH_TOKEN_ENDPOINT, VERTEX_AI_API_KEY_ENV,
VERTEXAI_API_KEY_ENV,
},
};
pub struct VertexEnvironment {
pub headers: Vec<(String, String)>,
pub project_id: String,
}
struct VertexAccessToken {
token: String,
project_id: String,
}
#[derive(Clone)]
pub struct VertexAuth {
providers: Cache<CredentialCacheKey, Arc<dyn VertexTokenSource>>,
loader: Arc<dyn VertexProviderLoader>,
}
impl Default for VertexAuth {
fn default() -> Self {
Self::new(Arc::new(GcpProviderLoader))
}
}
impl VertexAuth {
pub fn new(loader: Arc<dyn VertexProviderLoader>) -> Self {
Self {
providers: Cache::builder().max_capacity(64).build(),
loader,
}
}
pub async fn access_token(
&self,
config: &VertexConfig,
env_lookup: &(dyn Fn(&str) -> Option<String> + Sync),
) -> Result<String, Error> {
self.load_provider(config, env_lookup).await?.token().await
}
pub async fn validate_environment(
&self,
headers: Vec<(String, String)>,
api_key: Option<&str>,
config: &VertexConfig,
env_lookup: &(dyn Fn(&str) -> Option<String> + Sync),
) -> Result<VertexEnvironment, Error> {
let has_authorization = headers
.iter()
.any(|(name, _)| name.eq_ignore_ascii_case("Authorization"));
let static_token = api_key
.map(str::trim)
.filter(|value| !value.is_empty())
.map(str::to_string)
.or_else(|| non_empty_env(env_lookup, VERTEX_AI_API_KEY_ENV))
.or_else(|| non_empty_env(env_lookup, VERTEXAI_API_KEY_ENV));
let project_id = get_vertex_ai_project(config, env_lookup);
if !has_authorization && static_token.is_none() {
let access = self.get_access_token(config, env_lookup).await?;
return Ok(VertexEnvironment {
headers: apply_credential(headers, &access.token, CredentialPlacement::Bearer)?,
project_id: project_id.unwrap_or(access.project_id),
});
}
let project_id = match project_id {
Some(project_id) => project_id,
None => {
self.load_provider(config, env_lookup)
.await?
.project_id()
.await?
}
};
let headers = if has_authorization {
headers
} else {
apply_credential(
headers,
static_token.as_deref().expect("static token was checked"),
CredentialPlacement::Bearer,
)?
};
Ok(VertexEnvironment {
headers,
project_id,
})
}
async fn get_access_token(
&self,
config: &VertexConfig,
env_lookup: &(dyn Fn(&str) -> Option<String> + Sync),
) -> Result<VertexAccessToken, Error> {
let provider = self.load_provider(config, env_lookup).await?;
let (token, project_id) = tokio::try_join!(provider.token(), provider.project_id())?;
Ok(VertexAccessToken { token, project_id })
}
async fn load_provider(
&self,
config: &VertexConfig,
env_lookup: &(dyn Fn(&str) -> Option<String> + Sync),
) -> Result<Arc<dyn VertexTokenSource>, Error> {
let source = credential_source(config, env_lookup);
let key = source.cache_key();
self.providers
.try_get_with(key, self.loader.load(source))
.await
.map_err(|error| (*error).clone())
}
}
pub trait VertexTokenSource: Send + Sync {
fn project_id(&self) -> VertexAuthFuture<'_, String>;
fn token(&self) -> VertexAuthFuture<'_, String>;
}
pub trait VertexProviderLoader: Send + Sync {
fn load(&self, source: CredentialSource) -> VertexAuthFuture<'_, Arc<dyn VertexTokenSource>>;
}
pub type VertexAuthFuture<'a, T> = Pin<Box<dyn Future<Output = Result<T, Error>> + Send + 'a>>;
struct GcpTokenSource(Arc<dyn TokenProvider>);
impl VertexTokenSource for GcpTokenSource {
fn project_id(&self) -> VertexAuthFuture<'_, String> {
Box::pin(async move {
self.0
.project_id()
.await
.map(|project| project.to_string())
.map_err(auth_acquisition_error)
})
}
fn token(&self) -> VertexAuthFuture<'_, String> {
Box::pin(async move {
self.0
.token(&[CLOUD_PLATFORM_SCOPE])
.await
.map(|token| token.as_str().to_string())
.map_err(auth_acquisition_error)
})
}
}
struct GcpProviderLoader;
impl VertexProviderLoader for GcpProviderLoader {
fn load(&self, source: CredentialSource) -> VertexAuthFuture<'_, Arc<dyn VertexTokenSource>> {
Box::pin(async move {
let provider: Arc<dyn TokenProvider> = match source {
CredentialSource::Inline(configured) => Arc::new(
CustomServiceAccount::from_json(validate_request_credentials(
configured.expose(),
)?)
.map_err(auth_acquisition_error)?,
),
CredentialSource::Trusted(configured) => {
let configured = configured.expose();
let service_account = if Path::new(configured).is_file() {
CustomServiceAccount::from_file(configured)
} else {
CustomServiceAccount::from_json(configured)
}
.map_err(auth_acquisition_error)?;
Arc::new(service_account)
}
CredentialSource::ApplicationCredentials(path) => {
Arc::new(CustomServiceAccount::from_file(path).map_err(auth_acquisition_error)?)
}
CredentialSource::Adc => {
gcp_auth::provider().await.map_err(auth_acquisition_error)?
}
};
Ok(Arc::new(GcpTokenSource(provider)) as Arc<dyn VertexTokenSource>)
})
}
}
fn validate_request_credentials(configured: &str) -> Result<&str, Error> {
let token_uri = serde_json::from_str::<Value>(configured)
.ok()
.and_then(|credentials| {
credentials
.get("token_uri")
.and_then(Value::as_str)
.map(str::to_string)
});
if token_uri.as_deref() != Some(GOOGLE_OAUTH_TOKEN_ENDPOINT) {
return Err(Error::InvalidConfiguration(ErrorDetail::InvalidType {
field: "request-controlled vertex_credentials token_uri".into(),
expected: "the canonical Google OAuth token endpoint",
}));
}
Ok(configured)
}
impl CredentialSource {
fn cache_key(&self) -> CredentialCacheKey {
match self {
Self::Inline(configured) => {
CredentialCacheKey::Inline(Sha256::digest(configured.expose()).into())
}
Self::Trusted(configured) => {
CredentialCacheKey::Trusted(Sha256::digest(configured.expose()).into())
}
Self::ApplicationCredentials(path) => {
CredentialCacheKey::ApplicationCredentials(path.clone())
}
Self::Adc => CredentialCacheKey::Adc,
}
}
}
#[derive(Clone, Debug, Hash, PartialEq, Eq)]
enum CredentialCacheKey {
Inline([u8; 32]),
Trusted([u8; 32]),
ApplicationCredentials(String),
Adc,
}
fn auth_acquisition_error(error: gcp_auth::Error) -> Error {
Error::CredentialAcquisition(litellm_auth_types::ErrorDetail::failed(
"Vertex AI credentials",
error,
))
}
#[cfg(test)]
mod tests {
use litellm_auth_types::SecretValue;
use super::*;
#[rstest::rstest]
#[case::canonical_endpoint(r#"{"token_uri":"https://oauth2.googleapis.com/token"}"#, true)]
#[case::noncanonical_endpoint(r#"{"token_uri":"http://127.0.0.1/token"}"#, false)]
#[case::missing_endpoint("{}", false)]
fn request_credentials_require_canonical_token_endpoint(
#[case] credentials: &str,
#[case] accepted: bool,
) {
assert_eq!(validate_request_credentials(credentials).is_ok(), accepted);
}
#[rstest::rstest]
fn inline_and_trusted_credentials_never_share_a_cache_entry() {
assert_ne!(
CredentialSource::Inline(SecretValue::new("same-value")).cache_key(),
CredentialSource::Trusted(SecretValue::new("same-value")).cache_key()
);
}
}

311
litellm-rust/crates/auth-gcp/src/config.rs Normal file
View file

@ -0,0 +1,311 @@
use std::{collections::BTreeMap, path::Path};
use litellm_auth_types::{Error, ErrorDetail, InputSource, SecretValue, Sourced};
use serde_json::{Map, Value};
use litellm_auth_types::VertexParams;
use crate::constants::GOOGLE_APPLICATION_CREDENTIALS_ENV;
#[derive(Clone, Debug, Default)]
pub struct VertexConfig {
credentials: Option<Sourced<SecretValue>>,
project_id: Option<String>,
location: Option<String>,
}
impl VertexConfig {
pub fn new(
credentials: Option<Sourced<SecretValue>>,
project_id: Option<String>,
location: Option<String>,
) -> Self {
Self {
credentials: credentials.filter(|value| !value.value().expose().trim().is_empty()),
project_id: project_id.filter(|value| !value.trim().is_empty()),
location: location.filter(|value| !value.trim().is_empty()),
}
}
pub fn from_sourced_optional_params(
params: &Map<String, Value>,
sources: &BTreeMap<String, InputSource>,
) -> Result<Self, Error> {
Ok(Self::new(
optional_credentials(params, sources, VertexParams::CREDENTIALS.wire)?,
optional_string(params, VertexParams::PROJECT.wire)?,
optional_string(params, VertexParams::LOCATION.wire)?,
))
}
/// The deployment's `litellm_params`, which a host already trusts the way it trusts its
/// own configuration.
pub fn from_params(params: &VertexParams) -> Self {
Self::new(
params
.credentials()
.map(|value| Sourced::new(SecretValue::new(value), InputSource::Deployment)),
params.project(),
params.location(),
)
}
pub fn project_id(&self) -> Option<&str> {
self.project_id.as_deref()
}
pub fn location(&self) -> Option<&str> {
self.location.as_deref()
}
}
pub fn get_vertex_ai_project(
config: &VertexConfig,
env_lookup: &dyn Fn(&str) -> Option<String>,
) -> Option<String> {
config
.project_id()
.map(str::to_string)
.or_else(|| VertexParams::PROJECT.resolve(&|_| None, env_lookup))
}
/// The project a request is billed to when nothing names it: the `project_id` inside the
/// service account the call would authenticate with. Application default credentials carry
/// no project that can be read without a token exchange, so they resolve to `None`.
pub fn get_vertex_ai_project_from_credentials(
config: &VertexConfig,
env_lookup: &dyn Fn(&str) -> Option<String>,
) -> Option<String> {
let text = match credential_source(config, env_lookup) {
CredentialSource::Inline(configured) => configured.expose().to_string(),
CredentialSource::Trusted(configured) => {
let configured = configured.expose();
match std::fs::read_to_string(configured) {
Ok(contents) if Path::new(configured).is_file() => contents,
_ => configured.to_string(),
}
}
CredentialSource::ApplicationCredentials(path) => std::fs::read_to_string(path).ok()?,
CredentialSource::Adc => return None,
};
serde_json::from_str::<Value>(&text)
.ok()?
.get("project_id")
.and_then(Value::as_str)
.map(str::to_string)
}
pub fn get_vertex_ai_location(
config: &VertexConfig,
env_lookup: &dyn Fn(&str) -> Option<String>,
) -> Option<String> {
config
.location()
.map(str::to_string)
.or_else(|| VertexParams::LOCATION.resolve(&|_| None, env_lookup))
}
#[derive(Clone, Debug)]
pub enum CredentialSource {
Inline(SecretValue),
Trusted(SecretValue),
ApplicationCredentials(String),
Adc,
}
pub(crate) fn credential_source(
config: &VertexConfig,
env_lookup: &dyn Fn(&str) -> Option<String>,
) -> CredentialSource {
if let Some(configured) = config.credentials.clone() {
return match configured.source() {
InputSource::Request => CredentialSource::Inline(configured.into_value()),
InputSource::Deployment | InputSource::Environment => {
CredentialSource::Trusted(configured.into_value())
}
};
}
if let Some(configured) = VertexParams::CREDENTIALS.resolve(&|_| None, env_lookup) {
return CredentialSource::Trusted(SecretValue::new(configured));
}
non_empty_env(env_lookup, GOOGLE_APPLICATION_CREDENTIALS_ENV)
.map(CredentialSource::ApplicationCredentials)
.unwrap_or(CredentialSource::Adc)
}
fn optional_credentials(
params: &Map<String, Value>,
sources: &BTreeMap<String, InputSource>,
names: &[&str],
) -> Result<Option<Sourced<SecretValue>>, Error> {
for name in names {
let source = source_for(sources, name);
match params.get(*name) {
None | Some(Value::Null) => continue,
Some(Value::String(value)) if value.trim().is_empty() => continue,
Some(Value::String(value)) => {
return Ok(Some(Sourced::new(SecretValue::new(value), source)));
}
Some(Value::Object(value)) if value.is_empty() => continue,
Some(Value::Object(value)) => {
return serde_json::to_string(value)
.map(SecretValue::new)
.map(|value| Sourced::new(value, source))
.map(Some)
.map_err(|error| {
Error::InvalidConfiguration(ErrorDetail::failed(
"credential serialization",
error,
))
});
}
Some(_) => {
return Err(Error::InvalidConfiguration(ErrorDetail::InvalidType {
field: names[0].into(),
expected: "a string or null",
}));
}
}
}
Ok(None)
}
fn source_for(sources: &BTreeMap<String, InputSource>, name: &str) -> InputSource {
sources.get(name).copied().unwrap_or_default()
}
fn optional_string(params: &Map<String, Value>, names: &[&str]) -> Result<Option<String>, Error> {
for name in names {
match params.get(*name) {
None | Some(Value::Null) => continue,
Some(Value::String(value)) if value.trim().is_empty() => continue,
Some(Value::String(value)) => return Ok(Some(value.clone())),
Some(_) => {
return Err(Error::InvalidConfiguration(ErrorDetail::InvalidType {
field: names[0].into(),
expected: "a string or null",
}));
}
}
}
Ok(None)
}
pub(crate) fn non_empty_env(
env_lookup: &dyn Fn(&str) -> Option<String>,
name: &str,
) -> Option<String> {
env_lookup(name)
.map(|value| value.trim().to_string())
.filter(|value| !value.is_empty())
}
#[cfg(test)]
mod tests {
use serde_json::json;
use super::*;
use crate::constants::VERTEXAI_CREDENTIALS_ENV;
fn config(value: Value) -> VertexConfig {
VertexConfig::from_sourced_optional_params(value.as_object().unwrap(), &BTreeMap::new())
.unwrap()
}
#[rstest::rstest]
fn empty_primary_values_fall_back_to_python_aliases() {
let config = config(json!({
"vertex_credentials": null,
"vertex_ai_credentials": "alias-credentials",
"vertex_project": " ",
"vertex_ai_project": "alias-project",
"vertex_location": null,
"vertex_ai_location": "alias-location"
}));
assert_eq!(
config.credentials.as_ref().unwrap().value().expose(),
"alias-credentials"
);
assert_eq!(config.project_id(), Some("alias-project"));
assert_eq!(config.location(), Some("alias-location"));
}
#[rstest::rstest]
fn typed_config_preserves_source_and_empty_value_fallback() {
let configured = VertexConfig::new(
Some(Sourced::new(
SecretValue::new("inline-json"),
InputSource::Request,
)),
Some("project".into()),
Some("location".into()),
);
assert!(matches!(
credential_source(&configured, &|_| Some("environment-json".into())),
CredentialSource::Inline(value) if value.expose() == "inline-json"
));
let empty = VertexConfig::new(
Some(Sourced::new(SecretValue::new(" "), InputSource::Request)),
Some(" ".into()),
Some(" ".into()),
);
assert!(matches!(
credential_source(&empty, &|_| None),
CredentialSource::Adc
));
assert_eq!(
get_vertex_ai_project(&empty, &|_| Some("env-project".into())).as_deref(),
Some("env-project")
);
assert_eq!(
get_vertex_ai_location(&empty, &|_| Some("env-location".into())).as_deref(),
Some("env-location")
);
}
#[rstest::rstest]
fn credential_discovery_prefers_input_then_environment_then_adc() {
let params = json!({"vertex_credentials":"input-json"});
let sources = BTreeMap::from([("vertex_credentials".to_string(), InputSource::Request)]);
let configured =
VertexConfig::from_sourced_optional_params(params.as_object().unwrap(), &sources)
.unwrap();
assert!(
matches!(credential_source(&configured, &|_| Some("environment-value".into())), CredentialSource::Inline(value) if value.expose() == "input-json")
);
let empty = VertexConfig::default();
assert!(
matches!(credential_source(&empty, &|name| (name == VERTEXAI_CREDENTIALS_ENV).then(|| "environment-json".into())), CredentialSource::Trusted(value) if value.expose() == "environment-json")
);
assert!(
matches!(credential_source(&empty, &|name| (name == GOOGLE_APPLICATION_CREDENTIALS_ENV).then(|| "adc.json".into())), CredentialSource::ApplicationCredentials(path) if path == "adc.json")
);
assert!(matches!(
credential_source(&empty, &|_| None),
CredentialSource::Adc
));
}
#[rstest::rstest]
fn params_become_a_deployment_sourced_config() {
let params = VertexParams {
vertex_credentials: Some("deployment-json".into()),
vertex_project: Some("project-1".into()),
vertex_ai_location: Some("europe-west4".into()),
..VertexParams::default()
};
let config = VertexConfig::from_params(&params);
assert_eq!(config.project_id(), Some("project-1"));
assert_eq!(config.location(), Some("europe-west4"));
assert!(
matches!(credential_source(&config, &|_| Some("environment-json".into())), CredentialSource::Trusted(value) if value.expose() == "deployment-json")
);
assert!(matches!(
credential_source(
&VertexConfig::from_params(&VertexParams::default()),
&|_| None
),
CredentialSource::Adc
));
}
}

36
litellm-rust/crates/auth-gcp/src/constants.rs Normal file
View file

@ -0,0 +1,36 @@
use std::sync::LazyLock;
use litellm_auth_types::VertexParams;
pub const CLOUD_PLATFORM_SCOPE: &str = "https://www.googleapis.com/auth/cloud-platform";
pub const GOOGLE_OAUTH_TOKEN_ENDPOINT: &str = "https://oauth2.googleapis.com/token";
pub const GOOGLE_APPLICATION_CREDENTIALS_ENV: &str = "GOOGLE_APPLICATION_CREDENTIALS";
pub const VERTEX_AI_API_KEY_ENV: &str = "VERTEX_AI_API_KEY";
pub const VERTEXAI_API_KEY_ENV: &str = "VERTEXAI_API_KEY";
pub const VERTEXAI_CREDENTIALS_ENV: &str = VertexParams::CREDENTIALS.env[0];
pub const VERTEX_LOCATION_ENV: &str = VertexParams::LOCATION.env[1];
/// Every environment name the Vertex params and the token acquisition read, for hosts
/// that resolve secrets up front.
pub fn secret_names() -> &'static [&'static str] {
static NAMES: LazyLock<Vec<&'static str>> = LazyLock::new(|| {
[
VERTEX_AI_API_KEY_ENV,
VERTEXAI_API_KEY_ENV,
GOOGLE_APPLICATION_CREDENTIALS_ENV,
]
.into_iter()
.chain(
VertexParams::SPECS
.iter()
.flat_map(|spec| spec.env.iter().copied()),
)
.fold(Vec::new(), |mut names, name| {
if !names.contains(&name) {
names.push(name);
}
names
})
});
&NAMES
}

716
litellm-rust/crates/auth-gcp/src/lib.rs
View file

@ -1,708 +1,16 @@
use std::{collections::BTreeMap, future::Future, path::Path, pin::Pin, sync::Arc};
use gcp_auth::{CustomServiceAccount, TokenProvider};
use litellm_auth_types::{
CredentialPlacement, Error, InputSource, SecretValue, Sourced, http::apply_credential,
};
use moka::future::Cache;
use serde_json::{Map, Value};
use sha2::{Digest, Sha256};
mod auth;
mod config;
pub mod constants;
#[cfg(feature = "google-sdk")]
mod sdk;
pub use auth::{
VertexAuth, VertexAuthFuture, VertexEnvironment, VertexProviderLoader, VertexTokenSource,
};
pub use config::{
CredentialSource, VertexConfig, get_vertex_ai_location, get_vertex_ai_project,
get_vertex_ai_project_from_credentials,
};
pub use constants::secret_names;
#[cfg(feature = "google-sdk")]
pub use sdk::GoogleCredentials;
const CLOUD_PLATFORM_SCOPE: &str = "https://www.googleapis.com/auth/cloud-platform";
const GOOGLE_OAUTH_TOKEN_ENDPOINT: &str = "https://oauth2.googleapis.com/token";
const GOOGLE_APPLICATION_CREDENTIALS_ENV: &str = "GOOGLE_APPLICATION_CREDENTIALS";
const VERTEX_AI_API_KEY_ENV: &str = "VERTEX_AI_API_KEY";
const VERTEXAI_API_KEY_ENV: &str = "VERTEXAI_API_KEY";
const VERTEXAI_CREDENTIALS_ENV: &str = "VERTEXAI_CREDENTIALS";
const VERTEXAI_PROJECT_ENV: &str = "VERTEXAI_PROJECT";
const VERTEXAI_LOCATION_ENV: &str = "VERTEXAI_LOCATION";
const VERTEX_LOCATION_ENV: &str = "VERTEX_LOCATION";
pub const SECRET_NAMES: &[&str] = &[
VERTEX_AI_API_KEY_ENV,
VERTEXAI_API_KEY_ENV,
VERTEXAI_CREDENTIALS_ENV,
GOOGLE_APPLICATION_CREDENTIALS_ENV,
VERTEXAI_PROJECT_ENV,
VERTEXAI_LOCATION_ENV,
VERTEX_LOCATION_ENV,
];
#[derive(Clone, Debug, Default)]
pub struct VertexConfig {
credentials: Option<Sourced<SecretValue>>,
project_id: Option<String>,
location: Option<String>,
}
impl VertexConfig {
pub fn new(
credentials: Option<Sourced<SecretValue>>,
project_id: Option<String>,
location: Option<String>,
) -> Self {
Self {
credentials: credentials.filter(|value| !value.value().expose().trim().is_empty()),
project_id: project_id.filter(|value| !value.trim().is_empty()),
location: location.filter(|value| !value.trim().is_empty()),
}
}
pub fn from_sourced_optional_params(
params: &Map<String, Value>,
sources: &BTreeMap<String, InputSource>,
) -> Result<Self, Error> {
Ok(Self::new(
optional_credentials(
params,
sources,
&["vertex_credentials", "vertex_ai_credentials"],
)?,
optional_string(params, &["vertex_project", "vertex_ai_project"])?,
optional_string(params, &["vertex_location", "vertex_ai_location"])?,
))
}
pub fn or_configured(self, project_id: Option<&str>, location: Option<&str>) -> Self {
let configured =
|value: Option<&str>| value.filter(|value| !value.is_empty()).map(str::to_string);
Self {
project_id: self.project_id.or_else(|| configured(project_id)),
location: self.location.or_else(|| configured(location)),
..self
}
}
pub fn project_id(&self) -> Option<&str> {
self.project_id.as_deref()
}
pub fn location(&self) -> Option<&str> {
self.location.as_deref()
}
}
pub struct VertexEnvironment {
pub headers: Vec<(String, String)>,
pub project_id: String,
}
struct VertexAccessToken {
token: String,
project_id: String,
}
pub fn get_vertex_ai_project(
config: &VertexConfig,
env_lookup: &dyn Fn(&str) -> Option<String>,
) -> Option<String> {
config
.project_id()
.map(str::to_string)
.or_else(|| non_empty_env(env_lookup, VERTEXAI_PROJECT_ENV))
}
pub fn get_vertex_ai_location(
config: &VertexConfig,
env_lookup: &dyn Fn(&str) -> Option<String>,
) -> Option<String> {
config
.location()
.map(str::to_string)
.or_else(|| non_empty_env(env_lookup, VERTEXAI_LOCATION_ENV))
.or_else(|| non_empty_env(env_lookup, VERTEX_LOCATION_ENV))
}
#[derive(Clone)]
pub struct VertexAuth {
providers: Cache<CredentialCacheKey, Arc<dyn VertexTokenSource>>,
loader: Arc<dyn VertexProviderLoader>,
}
impl Default for VertexAuth {
fn default() -> Self {
Self::new(Arc::new(GcpProviderLoader))
}
}
impl VertexAuth {
pub fn new(loader: Arc<dyn VertexProviderLoader>) -> Self {
Self {
providers: Cache::builder().max_capacity(64).build(),
loader,
}
}
pub async fn access_token(
&self,
config: &VertexConfig,
env_lookup: &(dyn Fn(&str) -> Option<String> + Sync),
) -> Result<String, Error> {
self.load_provider(config, env_lookup).await?.token().await
}
pub async fn validate_environment(
&self,
headers: Vec<(String, String)>,
api_key: Option<&str>,
config: &VertexConfig,
env_lookup: &(dyn Fn(&str) -> Option<String> + Sync),
) -> Result<VertexEnvironment, Error> {
let has_authorization = headers
.iter()
.any(|(name, _)| name.eq_ignore_ascii_case("Authorization"));
let static_token = api_key
.map(str::trim)
.filter(|value| !value.is_empty())
.map(str::to_string)
.or_else(|| non_empty_env(env_lookup, VERTEX_AI_API_KEY_ENV))
.or_else(|| non_empty_env(env_lookup, VERTEXAI_API_KEY_ENV));
let project_id = get_vertex_ai_project(config, env_lookup);
if !has_authorization && static_token.is_none() {
let access = self.get_access_token(config, env_lookup).await?;
return Ok(VertexEnvironment {
headers: apply_credential(headers, &access.token, CredentialPlacement::Bearer)?,
project_id: project_id.unwrap_or(access.project_id),
});
}
let project_id = match project_id {
Some(project_id) => project_id,
None => {
self.load_provider(config, env_lookup)
.await?
.project_id()
.await?
}
};
let headers = if has_authorization {
headers
} else {
apply_credential(
headers,
static_token.as_deref().expect("static token was checked"),
CredentialPlacement::Bearer,
)?
};
Ok(VertexEnvironment {
headers,
project_id,
})
}
async fn get_access_token(
&self,
config: &VertexConfig,
env_lookup: &(dyn Fn(&str) -> Option<String> + Sync),
) -> Result<VertexAccessToken, Error> {
let provider = self.load_provider(config, env_lookup).await?;
let (token, project_id) = tokio::try_join!(provider.token(), provider.project_id())?;
Ok(VertexAccessToken { token, project_id })
}
async fn load_provider(
&self,
config: &VertexConfig,
env_lookup: &(dyn Fn(&str) -> Option<String> + Sync),
) -> Result<Arc<dyn VertexTokenSource>, Error> {
let source = credential_source(config, env_lookup);
let key = source.cache_key();
self.providers
.try_get_with(key, self.loader.load(source))
.await
.map_err(|error| (*error).clone())
}
}
pub trait VertexTokenSource: Send + Sync {
fn project_id(&self) -> VertexAuthFuture<'_, String>;
fn token(&self) -> VertexAuthFuture<'_, String>;
}
pub trait VertexProviderLoader: Send + Sync {
fn load(&self, source: CredentialSource) -> VertexAuthFuture<'_, Arc<dyn VertexTokenSource>>;
}
pub type VertexAuthFuture<'a, T> = Pin<Box<dyn Future<Output = Result<T, Error>> + Send + 'a>>;
struct GcpTokenSource(Arc<dyn TokenProvider>);
impl VertexTokenSource for GcpTokenSource {
fn project_id(&self) -> VertexAuthFuture<'_, String> {
Box::pin(async move {
self.0
.project_id()
.await
.map(|project| project.to_string())
.map_err(auth_acquisition_error)
})
}
fn token(&self) -> VertexAuthFuture<'_, String> {
Box::pin(async move {
self.0
.token(&[CLOUD_PLATFORM_SCOPE])
.await
.map(|token| token.as_str().to_string())
.map_err(auth_acquisition_error)
})
}
}
struct GcpProviderLoader;
impl VertexProviderLoader for GcpProviderLoader {
fn load(&self, source: CredentialSource) -> VertexAuthFuture<'_, Arc<dyn VertexTokenSource>> {
Box::pin(async move {
let provider: Arc<dyn TokenProvider> = match source {
CredentialSource::Inline(configured) => Arc::new(
CustomServiceAccount::from_json(validate_request_credentials(
configured.expose(),
)?)
.map_err(auth_acquisition_error)?,
),
CredentialSource::Trusted(configured) => {
let configured = configured.expose();
let service_account = if Path::new(configured).is_file() {
CustomServiceAccount::from_file(configured)
} else {
CustomServiceAccount::from_json(configured)
}
.map_err(auth_acquisition_error)?;
Arc::new(service_account)
}
CredentialSource::ApplicationCredentials(path) => {
Arc::new(CustomServiceAccount::from_file(path).map_err(auth_acquisition_error)?)
}
CredentialSource::Adc => {
gcp_auth::provider().await.map_err(auth_acquisition_error)?
}
};
Ok(Arc::new(GcpTokenSource(provider)) as Arc<dyn VertexTokenSource>)
})
}
}
fn validate_request_credentials(configured: &str) -> Result<&str, Error> {
let token_uri = serde_json::from_str::<Value>(configured)
.ok()
.and_then(|credentials| {
credentials
.get("token_uri")
.and_then(Value::as_str)
.map(str::to_string)
});
if token_uri.as_deref() != Some(GOOGLE_OAUTH_TOKEN_ENDPOINT) {
return Err(Error::InvalidConfiguration("request-controlled Vertex credentials must use the canonical Google OAuth token endpoint".into()));
}
Ok(configured)
}
#[derive(Clone, Debug)]
pub enum CredentialSource {
Inline(SecretValue),
Trusted(SecretValue),
ApplicationCredentials(String),
Adc,
}
impl CredentialSource {
fn cache_key(&self) -> CredentialCacheKey {
match self {
Self::Inline(configured) => {
CredentialCacheKey::Inline(Sha256::digest(configured.expose()).into())
}
Self::Trusted(configured) => {
CredentialCacheKey::Trusted(Sha256::digest(configured.expose()).into())
}
Self::ApplicationCredentials(path) => {
CredentialCacheKey::ApplicationCredentials(path.clone())
}
Self::Adc => CredentialCacheKey::Adc,
}
}
}
#[derive(Clone, Debug, Hash, PartialEq, Eq)]
enum CredentialCacheKey {
Inline([u8; 32]),
Trusted([u8; 32]),
ApplicationCredentials(String),
Adc,
}
fn credential_source(
config: &VertexConfig,
env_lookup: &dyn Fn(&str) -> Option<String>,
) -> CredentialSource {
if let Some(configured) = config.credentials.clone() {
return match configured.source() {
InputSource::Request => CredentialSource::Inline(configured.into_value()),
InputSource::Deployment | InputSource::Environment => {
CredentialSource::Trusted(configured.into_value())
}
};
}
if let Some(configured) = non_empty_env(env_lookup, VERTEXAI_CREDENTIALS_ENV) {
return CredentialSource::Trusted(SecretValue::new(configured));
}
non_empty_env(env_lookup, GOOGLE_APPLICATION_CREDENTIALS_ENV)
.map(CredentialSource::ApplicationCredentials)
.unwrap_or(CredentialSource::Adc)
}
fn optional_credentials(
params: &Map<String, Value>,
sources: &BTreeMap<String, InputSource>,
names: &[&str],
) -> Result<Option<Sourced<SecretValue>>, Error> {
for name in names {
let source = source_for(sources, name);
match params.get(*name) {
None | Some(Value::Null) => continue,
Some(Value::String(value)) if value.trim().is_empty() => continue,
Some(Value::String(value)) => {
return Ok(Some(Sourced::new(SecretValue::new(value), source)));
}
Some(Value::Object(value)) if value.is_empty() => continue,
Some(Value::Object(value)) => {
return serde_json::to_string(value)
.map(SecretValue::new)
.map(|value| Sourced::new(value, source))
.map(Some)
.map_err(|error| {
Error::InvalidConfiguration(litellm_auth_types::ErrorDetail::failed(
"credential serialization",
error,
))
});
}
Some(_) => {
return Err(Error::InvalidConfiguration(
litellm_auth_types::ErrorDetail::InvalidType {
field: names[0].into(),
expected: "a string or null",
},
));
}
}
}
Ok(None)
}
fn source_for(sources: &BTreeMap<String, InputSource>, name: &str) -> InputSource {
sources.get(name).copied().unwrap_or_default()
}
fn optional_string(params: &Map<String, Value>, names: &[&str]) -> Result<Option<String>, Error> {
for name in names {
match params.get(*name) {
None | Some(Value::Null) => continue,
Some(Value::String(value)) if value.trim().is_empty() => continue,
Some(Value::String(value)) => return Ok(Some(value.clone())),
Some(_) => {
return Err(Error::InvalidConfiguration(
litellm_auth_types::ErrorDetail::InvalidType {
field: names[0].into(),
expected: "a string or null",
},
));
}
}
}
Ok(None)
}
fn non_empty_env(env_lookup: &dyn Fn(&str) -> Option<String>, name: &str) -> Option<String> {
env_lookup(name)
.map(|value| value.trim().to_string())
.filter(|value| !value.is_empty())
}
fn auth_acquisition_error(error: gcp_auth::Error) -> Error {
Error::CredentialAcquisition(litellm_auth_types::ErrorDetail::failed(
"Vertex AI credentials",
error,
))
}
#[cfg(test)]
mod tests {
use std::collections::BTreeSet;
use std::sync::atomic::{AtomicUsize, Ordering};
use serde_json::json;
use super::*;
struct FakeProvider {
calls: Arc<AtomicUsize>,
}
impl VertexTokenSource for FakeProvider {
fn project_id(&self) -> VertexAuthFuture<'_, String> {
self.calls.fetch_add(1, Ordering::SeqCst);
Box::pin(async { Ok("adc-project".into()) })
}
fn token(&self) -> VertexAuthFuture<'_, String> {
self.calls.fetch_add(1, Ordering::SeqCst);
Box::pin(async { Ok("adc-token".into()) })
}
}
struct FakeLoader {
loads: Arc<AtomicUsize>,
provider: Arc<dyn VertexTokenSource>,
}
impl VertexProviderLoader for FakeLoader {
fn load(
&self,
_source: CredentialSource,
) -> VertexAuthFuture<'_, Arc<dyn VertexTokenSource>> {
let loads = self.loads.clone();
let provider = self.provider.clone();
Box::pin(async move {
loads.fetch_add(1, Ordering::SeqCst);
Ok(provider)
})
}
}
fn config(value: Value) -> VertexConfig {
VertexConfig::from_sourced_optional_params(value.as_object().unwrap(), &BTreeMap::new())
.unwrap()
}
fn auth(calls: Arc<AtomicUsize>, loads: Arc<AtomicUsize>) -> VertexAuth {
let provider: Arc<dyn VertexTokenSource> = Arc::new(FakeProvider { calls });
VertexAuth::new(Arc::new(FakeLoader { loads, provider }))
}
#[test]
fn config_is_typed_and_secrets_are_redacted() {
let config = config(json!({
"vertex_credentials":{"private_key":"secret-key"},
"vertex_project":"project-1",
"vertex_location":"europe-west4"
}));
assert_eq!(config.project_id(), Some("project-1"));
assert_eq!(config.location(), Some("europe-west4"));
assert!(!format!("{config:?}").contains("secret-key"));
assert!(
VertexConfig::from_sourced_optional_params(
json!({"vertex_credentials":true}).as_object().unwrap(),
&BTreeMap::new()
)
.is_err()
);
}
#[tokio::test]
async fn secret_names_cover_environment_reads() {
let seen = Arc::new(std::sync::Mutex::new(BTreeSet::<String>::new()));
let recorded = seen.clone();
let env = |name: &str| {
recorded.lock().unwrap().insert(name.to_string());
None
};
let auth = auth(Arc::new(AtomicUsize::new(0)), Arc::new(AtomicUsize::new(0)));
auth.validate_environment(Vec::new(), None, &VertexConfig::default(), &env)
.await
.unwrap();
get_vertex_ai_location(&VertexConfig::default(), &env);
assert!(
seen.lock()
.unwrap()
.iter()
.all(|name| SECRET_NAMES.contains(&name.as_str()))
);
}
#[test]
fn empty_primary_values_fall_back_to_python_aliases() {
let config = config(json!({
"vertex_credentials": null,
"vertex_ai_credentials": "alias-credentials",
"vertex_project": " ",
"vertex_ai_project": "alias-project",
"vertex_location": null,
"vertex_ai_location": "alias-location"
}));
assert_eq!(
config.credentials.as_ref().unwrap().value().expose(),
"alias-credentials"
);
assert_eq!(config.project_id(), Some("alias-project"));
assert_eq!(config.location(), Some("alias-location"));
}
#[test]
fn typed_config_preserves_source_and_empty_value_fallback() {
let configured = VertexConfig::new(
Some(Sourced::new(
SecretValue::new("inline-json"),
InputSource::Request,
)),
Some("project".into()),
Some("location".into()),
);
assert!(matches!(
credential_source(&configured, &|_| Some("environment-json".into())),
CredentialSource::Inline(value) if value.expose() == "inline-json"
));
let empty = VertexConfig::new(
Some(Sourced::new(SecretValue::new(" "), InputSource::Request)),
Some(" ".into()),
Some(" ".into()),
);
assert!(matches!(
credential_source(&empty, &|_| None),
CredentialSource::Adc
));
assert_eq!(
get_vertex_ai_project(&empty, &|_| Some("env-project".into())).as_deref(),
Some("env-project")
);
assert_eq!(
get_vertex_ai_location(&empty, &|_| Some("env-location".into())).as_deref(),
Some("env-location")
);
}
#[test]
fn project_and_location_prefer_input_then_environment() {
let configured =
config(json!({"vertex_project":"input-project","vertex_location":"input-location"}));
let env = |name: &str| Some(format!("env-{name}"));
assert_eq!(
get_vertex_ai_project(&configured, &env).as_deref(),
Some("input-project")
);
assert_eq!(
get_vertex_ai_location(&configured, &env).as_deref(),
Some("input-location")
);
let empty = VertexConfig::default();
assert_eq!(
get_vertex_ai_project(&empty, &|_| Some("env-project".into())).as_deref(),
Some("env-project")
);
assert_eq!(
get_vertex_ai_location(&empty, &|name| (name == VERTEX_LOCATION_ENV)
.then(|| "fallback-location".into()))
.as_deref(),
Some("fallback-location")
);
}
#[test]
fn credential_discovery_prefers_input_then_environment_then_adc() {
let params = json!({"vertex_credentials":"input-json"});
let sources = BTreeMap::from([("vertex_credentials".to_string(), InputSource::Request)]);
let configured =
VertexConfig::from_sourced_optional_params(params.as_object().unwrap(), &sources)
.unwrap();
assert!(
matches!(credential_source(&configured, &|_| Some("environment-value".into())), CredentialSource::Inline(value) if value.expose() == "input-json")
);
let empty = VertexConfig::default();
assert!(
matches!(credential_source(&empty, &|name| (name == VERTEXAI_CREDENTIALS_ENV).then(|| "environment-json".into())), CredentialSource::Trusted(value) if value.expose() == "environment-json")
);
assert!(
matches!(credential_source(&empty, &|name| (name == GOOGLE_APPLICATION_CREDENTIALS_ENV).then(|| "adc.json".into())), CredentialSource::ApplicationCredentials(path) if path == "adc.json")
);
assert!(matches!(
credential_source(&empty, &|_| None),
CredentialSource::Adc
));
assert_ne!(
CredentialSource::Inline(SecretValue::new("same-value")).cache_key(),
CredentialSource::Trusted(SecretValue::new("same-value")).cache_key()
);
}
#[rstest::rstest]
#[case::canonical_endpoint(r#"{"token_uri":"https://oauth2.googleapis.com/token"}"#, true)]
#[case::noncanonical_endpoint(r#"{"token_uri":"http://127.0.0.1/token"}"#, false)]
#[case::missing_endpoint("{}", false)]
fn request_credentials_require_canonical_token_endpoint(
#[case] credentials: &str,
#[case] accepted: bool,
) {
assert_eq!(validate_request_credentials(credentials).is_ok(), accepted);
}
#[tokio::test]
async fn explicit_token_and_header_do_not_acquire_adc() {
let loads = Arc::new(AtomicUsize::new(0));
let auth = auth(Arc::new(AtomicUsize::new(0)), loads.clone());
let configured = config(json!({"vertex_project":"project-1"}));
let explicit = auth
.validate_environment(Vec::new(), Some("access-token"), &configured, &|_| None)
.await
.unwrap();
assert_eq!(explicit.headers[0].1, "Bearer access-token");
let existing = auth
.validate_environment(
vec![("authorization".into(), "Bearer existing".into())],
None,
&configured,
&|_| None,
)
.await
.unwrap();
assert_eq!(existing.headers[0].1, "Bearer existing");
assert_eq!(loads.load(Ordering::SeqCst), 0);
}
#[tokio::test]
async fn provider_is_reused_across_authentication_calls() {
let calls = Arc::new(AtomicUsize::new(0));
let loads = Arc::new(AtomicUsize::new(0));
let auth = auth(calls.clone(), loads.clone());
for _ in 0..2 {
let environment = auth
.validate_environment(Vec::new(), None, &VertexConfig::default(), &|_| None)
.await
.unwrap();
assert_eq!(environment.project_id, "adc-project");
assert_eq!(environment.headers[0].1, "Bearer adc-token");
}
assert_eq!(loads.load(Ordering::SeqCst), 1);
assert_eq!(calls.load(Ordering::SeqCst), 4);
}
#[test]
fn configured_defaults_sit_between_call_params_and_the_environment() {
let env = |name: &str| Some(format!("env-{name}"));
let from_config =
VertexConfig::default().or_configured(Some("global-project"), Some("global-location"));
assert_eq!(
get_vertex_ai_project(&from_config, &env).as_deref(),
Some("global-project")
);
assert_eq!(
get_vertex_ai_location(&from_config, &env).as_deref(),
Some("global-location")
);
let from_call =
config(json!({"vertex_project":"call-project","vertex_location":"call-location"}))
.or_configured(Some("global-project"), Some("global-location"));
assert_eq!(from_call.project_id(), Some("call-project"));
assert_eq!(from_call.location(), Some("call-location"));
let empty_global = VertexConfig::default().or_configured(Some(""), None);
assert_eq!(
get_vertex_ai_project(&empty_global, &env).as_deref(),
Some("env-VERTEXAI_PROJECT")
);
}
}

120
litellm-rust/crates/auth-gcp/tests/vertex_auth.rs Normal file
View file

@ -0,0 +1,120 @@
use std::{
collections::{BTreeMap, BTreeSet},
sync::{
Arc,
atomic::{AtomicUsize, Ordering},
},
};
use litellm_auth_gcp::{
CredentialSource, VertexAuth, VertexAuthFuture, VertexConfig, VertexProviderLoader,
VertexTokenSource, get_vertex_ai_location, secret_names,
};
use rstest::rstest;
use serde_json::{Value, json};
struct FakeProvider {
calls: Arc<AtomicUsize>,
}
impl VertexTokenSource for FakeProvider {
fn project_id(&self) -> VertexAuthFuture<'_, String> {
self.calls.fetch_add(1, Ordering::SeqCst);
Box::pin(async { Ok("adc-project".into()) })
}
fn token(&self) -> VertexAuthFuture<'_, String> {
self.calls.fetch_add(1, Ordering::SeqCst);
Box::pin(async { Ok("adc-token".into()) })
}
}
struct FakeLoader {
loads: Arc<AtomicUsize>,
provider: Arc<dyn VertexTokenSource>,
}
impl VertexProviderLoader for FakeLoader {
fn load(&self, _source: CredentialSource) -> VertexAuthFuture<'_, Arc<dyn VertexTokenSource>> {
let loads = self.loads.clone();
let provider = self.provider.clone();
Box::pin(async move {
loads.fetch_add(1, Ordering::SeqCst);
Ok(provider)
})
}
}
fn config(value: Value) -> VertexConfig {
VertexConfig::from_sourced_optional_params(value.as_object().unwrap(), &BTreeMap::new())
.unwrap()
}
fn auth(calls: Arc<AtomicUsize>, loads: Arc<AtomicUsize>) -> VertexAuth {
let provider: Arc<dyn VertexTokenSource> = Arc::new(FakeProvider { calls });
VertexAuth::new(Arc::new(FakeLoader { loads, provider }))
}
#[rstest]
#[tokio::test]
async fn secret_names_cover_environment_reads() {
let seen = Arc::new(std::sync::Mutex::new(BTreeSet::<String>::new()));
let recorded = seen.clone();
let env = |name: &str| {
recorded.lock().unwrap().insert(name.to_string());
None
};
let auth = auth(Arc::new(AtomicUsize::new(0)), Arc::new(AtomicUsize::new(0)));
auth.validate_environment(Vec::new(), None, &VertexConfig::default(), &env)
.await
.unwrap();
get_vertex_ai_location(&VertexConfig::default(), &env);
assert!(
seen.lock()
.unwrap()
.iter()
.all(|name| secret_names().contains(&name.as_str()))
);
}
#[rstest]
#[tokio::test]
async fn explicit_token_and_header_do_not_acquire_adc() {
let loads = Arc::new(AtomicUsize::new(0));
let auth = auth(Arc::new(AtomicUsize::new(0)), loads.clone());
let configured = config(json!({"vertex_project":"project-1"}));
let explicit = auth
.validate_environment(Vec::new(), Some("access-token"), &configured, &|_| None)
.await
.unwrap();
assert_eq!(explicit.headers[0].1, "Bearer access-token");
let existing = auth
.validate_environment(
vec![("authorization".into(), "Bearer existing".into())],
None,
&configured,
&|_| None,
)
.await
.unwrap();
assert_eq!(existing.headers[0].1, "Bearer existing");
assert_eq!(loads.load(Ordering::SeqCst), 0);
}
#[rstest]
#[tokio::test]
async fn provider_is_reused_across_authentication_calls() {
let calls = Arc::new(AtomicUsize::new(0));
let loads = Arc::new(AtomicUsize::new(0));
let auth = auth(calls.clone(), loads.clone());
for _ in 0..2 {
let environment = auth
.validate_environment(Vec::new(), None, &VertexConfig::default(), &|_| None)
.await
.unwrap();
assert_eq!(environment.project_id, "adc-project");
assert_eq!(environment.headers[0].1, "Bearer adc-token");
}
assert_eq!(loads.load(Ordering::SeqCst), 1);
assert_eq!(calls.load(Ordering::SeqCst), 4);
}

95
litellm-rust/crates/auth-gcp/tests/vertex_config.rs Normal file
View file

@ -0,0 +1,95 @@
use std::collections::BTreeMap;
use litellm_auth_gcp::{
VertexConfig,
constants::{GOOGLE_APPLICATION_CREDENTIALS_ENV, VERTEX_LOCATION_ENV},
get_vertex_ai_location, get_vertex_ai_project, get_vertex_ai_project_from_credentials,
};
use litellm_auth_types::VertexParams;
use rstest::rstest;
use serde_json::{Value, json};
fn config(value: Value) -> VertexConfig {
VertexConfig::from_sourced_optional_params(value.as_object().unwrap(), &BTreeMap::new())
.unwrap()
}
#[rstest]
fn config_is_typed_and_secrets_are_redacted() {
let config = config(json!({
"vertex_credentials":{"private_key":"secret-key"},
"vertex_project":"project-1",
"vertex_location":"europe-west4"
}));
assert_eq!(config.project_id(), Some("project-1"));
assert_eq!(config.location(), Some("europe-west4"));
assert!(!format!("{config:?}").contains("secret-key"));
assert!(
VertexConfig::from_sourced_optional_params(
json!({"vertex_credentials":true}).as_object().unwrap(),
&BTreeMap::new()
)
.is_err()
);
}
#[rstest]
fn project_and_location_prefer_input_then_environment() {
let configured =
config(json!({"vertex_project":"input-project","vertex_location":"input-location"}));
let env = |name: &str| Some(format!("env-{name}"));
assert_eq!(
get_vertex_ai_project(&configured, &env).as_deref(),
Some("input-project")
);
assert_eq!(
get_vertex_ai_location(&configured, &env).as_deref(),
Some("input-location")
);
let empty = VertexConfig::default();
assert_eq!(
get_vertex_ai_project(&empty, &|_| Some("env-project".into())).as_deref(),
Some("env-project")
);
assert_eq!(
get_vertex_ai_location(&empty, &|name| (name == VERTEX_LOCATION_ENV)
.then(|| "fallback-location".into()))
.as_deref(),
Some("fallback-location")
);
}
#[rstest]
fn project_is_read_from_the_credentials_that_would_authenticate() {
let key = tempfile::NamedTempFile::new().unwrap();
std::fs::write(key.path(), r#"{"project_id":"file-project"}"#).unwrap();
let path = key.path().to_str().unwrap().to_string();
let inline = VertexConfig::from_params(&VertexParams {
vertex_credentials: Some(r#"{"project_id":"inline-project"}"#.into()),
..VertexParams::default()
});
assert_eq!(
get_vertex_ai_project_from_credentials(&inline, &|_| None).as_deref(),
Some("inline-project")
);
let from_file = VertexConfig::from_params(&VertexParams {
vertex_credentials: Some(path.clone()),
..VertexParams::default()
});
assert_eq!(
get_vertex_ai_project_from_credentials(&from_file, &|_| None).as_deref(),
Some("file-project")
);
let empty = VertexConfig::default();
assert_eq!(
get_vertex_ai_project_from_credentials(&empty, &|name| (name
== GOOGLE_APPLICATION_CREDENTIALS_ENV)
.then(|| path.clone()))
.as_deref(),
Some("file-project")
);
assert_eq!(
get_vertex_ai_project_from_credentials(&empty, &|_| None),
None
);
}

1
litellm-rust/crates/auth-types/Cargo.toml
View file

@ -7,6 +7,7 @@ repository.workspace = true
[dependencies]
serde.workspace = true
serde_json.workspace = true
subtle.workspace = true
thiserror.workspace = true
veil.workspace = true

5
litellm-rust/crates/auth-types/src/error.rs
View file

@ -19,6 +19,11 @@ pub enum Error {
provider: &'static str,
environment_variable: &'static str,
},
#[error("Missing {provider} {} - {}", .spec.setting, .spec.guidance())]
MissingParam {
provider: &'static str,
spec: &'static crate::ParamSpec,
},
#[error("Missing {provider} API Base - {guidance}")]
MissingApiBase {
provider: &'static str,

2
litellm-rust/crates/auth-types/src/lib.rs
View file

@ -3,6 +3,7 @@
mod credential;
mod error;
pub mod http;
mod params;
mod policy;
mod secret;
mod token;
@ -52,6 +53,7 @@ pub use credential::{
};
pub use error::{Error, ErrorDetail, ErrorSource};
pub use http::CredentialPlacement;
pub use params::{AwsParams, ParamSpec, VertexParams};
pub use policy::{CredentialPlanKind, CredentialRule, ExistingHeaderBehavior, ProviderAuthPolicy};
pub use secret::SecretValue;
pub use token::{ResolvedCredential, TokenFuture, TokenProvider, TokenProviderHandle};

174
litellm-rust/crates/auth-types/src/params/aws.rs Normal file
View file

@ -0,0 +1,174 @@
use std::sync::LazyLock;
use serde::{Deserialize, Serialize};
use serde_json::{Map, Value};
use veil::Redact;
use super::ParamSpec;
const fn spec(
setting: &'static str,
wire: &'static [&'static str],
env: &'static [&'static str],
) -> ParamSpec {
ParamSpec {
setting,
wire,
module_global: None,
env,
}
}
/// The `aws_*` fields of Python's `GenericLiteLLMParams`.
#[derive(Redact, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct AwsParams {
#[serde(default)]
pub aws_access_key_id: Option<String>,
#[serde(default)]
#[redact(with = "[REDACTED]")]
pub aws_secret_access_key: Option<String>,
#[serde(default)]
#[redact(with = "[REDACTED]")]
pub aws_session_token: Option<String>,
#[serde(default)]
pub aws_region_name: Option<String>,
#[serde(default)]
pub aws_session_name: Option<String>,
#[serde(default)]
pub aws_profile_name: Option<String>,
#[serde(default)]
pub aws_role_name: Option<String>,
#[serde(default)]
#[redact(with = "[REDACTED]")]
pub aws_web_identity_token: Option<String>,
#[serde(default)]
pub aws_sts_endpoint: Option<String>,
#[serde(default)]
#[redact(with = "[REDACTED]")]
pub aws_external_id: Option<String>,
#[serde(default)]
pub aws_bedrock_runtime_endpoint: Option<String>,
}
impl AwsParams {
pub const ACCESS_KEY_ID: ParamSpec = spec(
"access key id",
&["aws_access_key_id"],
&["AWS_ACCESS_KEY_ID"],
);
pub const SECRET_ACCESS_KEY: ParamSpec = spec(
"secret access key",
&["aws_secret_access_key"],
&["AWS_SECRET_ACCESS_KEY"],
);
pub const SESSION_TOKEN: ParamSpec = spec(
"session token",
&["aws_session_token"],
&["AWS_SESSION_TOKEN"],
);
pub const REGION: ParamSpec = spec(
"region",
&["aws_region_name"],
&["AWS_REGION_NAME", "AWS_REGION"],
);
pub const SESSION_NAME: ParamSpec =
spec("session name", &["aws_session_name"], &["AWS_SESSION_NAME"]);
pub const PROFILE_NAME: ParamSpec =
spec("profile name", &["aws_profile_name"], &["AWS_PROFILE_NAME"]);
pub const ROLE_NAME: ParamSpec = spec("role name", &["aws_role_name"], &["AWS_ROLE_NAME"]);
pub const WEB_IDENTITY_TOKEN: ParamSpec = spec(
"web identity token",
&["aws_web_identity_token"],
&["AWS_WEB_IDENTITY_TOKEN"],
);
pub const STS_ENDPOINT: ParamSpec =
spec("STS endpoint", &["aws_sts_endpoint"], &["AWS_STS_ENDPOINT"]);
pub const EXTERNAL_ID: ParamSpec =
spec("external id", &["aws_external_id"], &["AWS_EXTERNAL_ID"]);
pub const BEDROCK_RUNTIME_ENDPOINT: ParamSpec = spec(
"Bedrock runtime endpoint",
&["aws_bedrock_runtime_endpoint"],
&["AWS_BEDROCK_RUNTIME_ENDPOINT"],
);
/// Every `aws_*` param Python reads, in declaration order, with the environment names
/// each falls back to.
pub const SPECS: [ParamSpec; 11] = [
Self::ACCESS_KEY_ID,
Self::SECRET_ACCESS_KEY,
Self::SESSION_TOKEN,
Self::REGION,
Self::SESSION_NAME,
Self::PROFILE_NAME,
Self::ROLE_NAME,
Self::WEB_IDENTITY_TOKEN,
Self::STS_ENDPOINT,
Self::EXTERNAL_ID,
Self::BEDROCK_RUNTIME_ENDPOINT,
];
/// The wire names a host projects out of a caller's kwargs, derived from [`Self::SPECS`].
pub fn fields() -> impl Iterator<Item = &'static str> {
Self::SPECS
.iter()
.flat_map(|spec| spec.wire.iter().copied())
}
/// The environment names the specs read, for hosts that resolve secrets up front.
pub fn secret_names() -> &'static [&'static str] {
static NAMES: LazyLock<Vec<&'static str>> = LazyLock::new(|| {
AwsParams::SPECS
.iter()
.flat_map(|spec| spec.env.iter().copied())
.fold(Vec::new(), |mut names, name| {
if !names.contains(&name) {
names.push(name);
}
names
})
});
&NAMES
}
/// The value under a wire name, read through serde so the names can never drift from
/// the struct.
pub fn get(&self, wire: &str) -> Option<String> {
serde_json::to_value(self)
.ok()?
.get(wire)?
.as_str()
.map(str::to_string)
}
/// The spec's value from these params, then the environment.
pub fn resolve(
&self,
spec: &ParamSpec,
env_lookup: &dyn Fn(&str) -> Option<String>,
) -> Option<String> {
spec.resolve(&|name| self.get(name), env_lookup)
}
/// Reads the string-valued `aws_*` keys of an untyped params map, ignoring anything else.
pub fn from_optional_params(optional_params: &Map<String, Value>) -> Self {
let value = |key: &str| {
optional_params
.get(key)
.and_then(Value::as_str)
.map(str::to_string)
};
Self {
aws_access_key_id: value("aws_access_key_id"),
aws_secret_access_key: value("aws_secret_access_key"),
aws_session_token: value("aws_session_token"),
aws_region_name: value("aws_region_name"),
aws_session_name: value("aws_session_name"),
aws_profile_name: value("aws_profile_name"),
aws_role_name: value("aws_role_name"),
aws_web_identity_token: value("aws_web_identity_token"),
aws_sts_endpoint: value("aws_sts_endpoint"),
aws_external_id: value("aws_external_id"),
aws_bedrock_runtime_endpoint: value("aws_bedrock_runtime_endpoint"),
}
}
}

48
litellm-rust/crates/auth-types/src/params/mod.rs Normal file
View file

@ -0,0 +1,48 @@
mod aws;
mod vertex;
pub use aws::AwsParams;
pub use vertex::VertexParams;
/// One connection param of Python's `CredentialLiteLLMParams` and every place Python reads
/// it from, in order: the wire names on a call or a deployment, the `litellm.<name>` module
/// global a host folds in, then the environment names.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct ParamSpec {
pub setting: &'static str,
pub wire: &'static [&'static str],
pub module_global: Option<&'static str>,
pub env: &'static [&'static str],
}
impl ParamSpec {
/// The first non-blank value: each wire name through `params`, then each environment
/// name through `env`.
pub fn resolve(
&self,
params: &dyn Fn(&str) -> Option<String>,
env: &dyn Fn(&str) -> Option<String>,
) -> Option<String> {
self.wire
.iter()
.map(|name| params(name))
.chain(self.env.iter().map(|name| env(name)))
.find_map(|value| value.filter(|value| !value.trim().is_empty()))
}
pub fn missing(&'static self, provider: &'static str) -> crate::Error {
crate::Error::MissingParam {
provider,
spec: self,
}
}
pub(crate) fn guidance(&self) -> String {
let wire = self.wire.join(" or ");
let env = self.env.join(" or ");
match self.module_global {
Some(global) => format!("pass {wire}, set litellm.{global}, or set {env}"),
None => format!("pass {wire} or set {env}"),
}
}
}

118
litellm-rust/crates/auth-types/src/params/vertex.rs Normal file
View file

@ -0,0 +1,118 @@
use serde::{Deserialize, Deserializer, Serialize, de::Unexpected};
use serde_json::Value;
use veil::Redact;
use super::ParamSpec;
/// The Vertex AI fields of Python's `GenericLiteLLMParams`, both the current names and the
/// `vertex_ai_*` spellings `VertexBase.safe_get_vertex_ai_*` still read.
#[derive(Redact, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct VertexParams {
#[serde(default, deserialize_with = "optional_credential_text")]
#[redact(with = "[REDACTED]")]
pub vertex_credentials: Option<String>,
#[serde(default)]
pub vertex_project: Option<String>,
#[serde(default)]
pub vertex_location: Option<String>,
#[serde(default, deserialize_with = "optional_credential_text")]
#[redact(with = "[REDACTED]")]
pub vertex_ai_credentials: Option<String>,
#[serde(default)]
pub vertex_ai_project: Option<String>,
#[serde(default)]
pub vertex_ai_location: Option<String>,
}
impl VertexParams {
pub const CREDENTIALS: ParamSpec = ParamSpec {
setting: "credentials",
wire: &["vertex_credentials", "vertex_ai_credentials"],
module_global: None,
env: &["VERTEXAI_CREDENTIALS"],
};
pub const PROJECT: ParamSpec = ParamSpec {
setting: "project",
wire: &["vertex_project", "vertex_ai_project"],
module_global: Some("vertex_project"),
env: &["VERTEXAI_PROJECT"],
};
pub const LOCATION: ParamSpec = ParamSpec {
setting: "location",
wire: &["vertex_location", "vertex_ai_location"],
module_global: Some("vertex_location"),
env: &["VERTEXAI_LOCATION", "VERTEX_LOCATION"],
};
/// Every Vertex AI param Python reads, with both spellings, the module global
/// `VertexBase.get_vertex_ai_*` consults, and the environment names it falls back to.
pub const SPECS: [ParamSpec; 3] = [Self::CREDENTIALS, Self::PROJECT, Self::LOCATION];
/// The wire names a host projects out of a caller's kwargs, derived from [`Self::SPECS`].
pub fn fields() -> impl Iterator<Item = &'static str> {
Self::SPECS
.iter()
.flat_map(|spec| spec.wire.iter().copied())
}
/// The value under a wire name, read through serde so the names can never drift from
/// the struct.
pub fn get(&self, wire: &str) -> Option<String> {
serde_json::to_value(self)
.ok()?
.get(wire)?
.as_str()
.map(str::to_string)
}
/// The spec's value from these params, then the environment.
pub fn resolve(
&self,
spec: &ParamSpec,
env_lookup: &dyn Fn(&str) -> Option<String>,
) -> Option<String> {
spec.resolve(&|name| self.get(name), env_lookup)
}
pub fn credentials(&self) -> Option<String> {
self.resolve(&Self::CREDENTIALS, &|_| None)
}
pub fn project(&self) -> Option<String> {
self.resolve(&Self::PROJECT, &|_| None)
}
pub fn location(&self) -> Option<String> {
self.resolve(&Self::LOCATION, &|_| None)
}
}
/// A credential is a service account JSON text or a path to one; a YAML or kwargs mapping is
/// accepted as the JSON text it spells, the way Python passes a dict through.
fn optional_credential_text<'de, D: Deserializer<'de>>(
deserializer: D,
) -> Result<Option<String>, D::Error> {
match Option::<Value>::deserialize(deserializer)? {
None | Some(Value::Null) => Ok(None),
Some(Value::String(text)) => Ok(Some(text)),
Some(Value::Object(object)) if object.is_empty() => Ok(None),
Some(Value::Object(object)) => serde_json::to_string(&object)
.map(Some)
.map_err(serde::de::Error::custom),
Some(other) => Err(serde::de::Error::invalid_type(
Unexpected::Other(value_kind(&other)),
&"a string or an object",
)),
}
}
fn value_kind(value: &Value) -> &'static str {
match value {
Value::Null => "null",
Value::Bool(_) => "a boolean",
Value::Number(_) => "a number",
Value::String(_) => "a string",
Value::Array(_) => "a list",
Value::Object(_) => "an object",
}
}

26
litellm-rust/crates/auth-types/tests/aws_params.rs Normal file
View file

@ -0,0 +1,26 @@
use litellm_auth_types::AwsParams;
use rstest::rstest;
use serde_json::{Map, Value};
#[rstest]
fn fields_name_every_param_once_and_in_declaration_order() {
let filled: Map<String, Value> = AwsParams::fields()
.map(|name| (name.to_string(), Value::from(format!("value-of-{name}"))))
.collect();
let typed = AwsParams::from_optional_params(&filled);
let serialized = serde_json::to_value(&typed).unwrap();
assert_eq!(
serialized
.as_object()
.unwrap()
.keys()
.map(String::as_str)
.collect::<Vec<_>>(),
AwsParams::fields().collect::<Vec<_>>()
);
assert_eq!(serialized, Value::Object(filled.clone()));
assert_eq!(
serde_json::from_value::<AwsParams>(Value::Object(filled)).unwrap(),
typed
);
}

72
litellm-rust/crates/auth-types/tests/params.rs Normal file
View file

@ -0,0 +1,72 @@
use litellm_auth_types::{AwsParams, ParamSpec};
use rstest::rstest;
const LOCATION: ParamSpec = ParamSpec {
setting: "location",
wire: &["vertex_location", "vertex_ai_location"],
module_global: Some("vertex_location"),
env: &["VERTEXAI_LOCATION", "VERTEX_LOCATION"],
};
fn lookup(entries: &'static [(&'static str, &'static str)]) -> impl Fn(&str) -> Option<String> {
move |name| {
entries
.iter()
.find(|(key, _)| *key == name)
.map(|(_, value)| value.to_string())
}
}
#[rstest]
#[case::current_spelling_first(
&[("vertex_location", "eu"), ("vertex_ai_location", "us")],
&[("VERTEXAI_LOCATION", "env")],
Some("eu")
)]
#[case::legacy_spelling_before_the_environment(
&[("vertex_ai_location", "us")],
&[("VERTEXAI_LOCATION", "env")],
Some("us")
)]
#[case::blank_values_are_skipped(
&[("vertex_location", " "), ("vertex_ai_location", "")],
&[("VERTEXAI_LOCATION", " "), ("VERTEX_LOCATION", "fallback")],
Some("fallback")
)]
#[case::environment_names_in_order(
&[],
&[("VERTEX_LOCATION", "second"), ("VERTEXAI_LOCATION", "first")],
Some("first")
)]
#[case::nothing_set(&[], &[], None)]
fn resolution_walks_wire_names_then_environment_names(
#[case] params: &'static [(&'static str, &'static str)],
#[case] env: &'static [(&'static str, &'static str)],
#[case] expected: Option<&str>,
) {
assert_eq!(
LOCATION.resolve(&lookup(params), &lookup(env)).as_deref(),
expected
);
}
#[rstest]
#[case::without_a_module_global(
&AwsParams::REGION,
"Missing AWS region - pass aws_region_name or set AWS_REGION_NAME or AWS_REGION"
)]
#[case::with_a_module_global(
&LOCATION,
"Missing AWS location - pass vertex_location or vertex_ai_location, set litellm.vertex_location, or set VERTEXAI_LOCATION or VERTEX_LOCATION"
)]
fn a_missing_param_names_every_place_the_spec_reads(
#[case] spec: &'static ParamSpec,
#[case] expected: &str,
) {
let message = spec.missing("AWS").to_string();
assert_eq!(message, expected);
assert!(spec.wire.iter().all(|name| message.contains(name)));
assert!(spec.env.iter().all(|name| message.contains(name)));
assert_eq!(message.contains("litellm."), spec.module_global.is_some());
}

126
litellm-rust/crates/auth-types/tests/vertex_params.rs Normal file
View file

@ -0,0 +1,126 @@
use std::collections::BTreeSet;
use litellm_auth_types::VertexParams;
use rstest::rstest;
use serde_json::{Map, Value, json};
#[rstest]
#[case::current_spelling(
VertexParams { vertex_project: Some("p".into()), vertex_ai_project: Some("old".into()), ..VertexParams::default() },
Some("p")
)]
#[case::legacy_spelling(
VertexParams { vertex_ai_project: Some("old".into()), ..VertexParams::default() },
Some("old")
)]
#[case::blank_falls_through(
VertexParams { vertex_project: Some(" ".into()), vertex_ai_project: Some("old".into()), ..VertexParams::default() },
Some("old")
)]
#[case::unset(VertexParams::default(), None)]
fn params_prefer_the_current_spelling_over_the_legacy_one(
#[case] params: VertexParams,
#[case] project: Option<&str>,
) {
assert_eq!(params.project().as_deref(), project);
let mirrored = VertexParams {
vertex_credentials: params.vertex_project.clone(),
vertex_ai_credentials: params.vertex_ai_project.clone(),
vertex_location: params.vertex_project.clone(),
vertex_ai_location: params.vertex_ai_project.clone(),
..VertexParams::default()
};
assert_eq!(mirrored.credentials().as_deref(), project);
assert_eq!(mirrored.location().as_deref(), project);
}
#[rstest]
#[case::text(json!({"vertex_credentials": "/path/to/key.json"}), Some("/path/to/key.json"))]
#[case::object(json!({"vertex_credentials": {"type": "service_account"}}), Some(r#"{"type":"service_account"}"#))]
#[case::null(json!({"vertex_credentials": null}), None)]
#[case::empty_object_falls_through_to_the_legacy_spelling(
json!({"vertex_credentials": {}, "vertex_ai_credentials": "/legacy/key.json"}),
Some("/legacy/key.json")
)]
#[case::absent(json!({}), None)]
fn credentials_deserialize_from_text_or_an_object(
#[case] params: Value,
#[case] credentials: Option<&str>,
) {
let params: VertexParams = serde_json::from_value(params).unwrap();
assert_eq!(params.credentials().as_deref(), credentials);
}
#[rstest]
#[case::number(json!({"vertex_credentials": 7}))]
#[case::list(json!({"vertex_ai_credentials": ["a"]}))]
#[case::project(json!({"vertex_project": ["p"]}))]
fn a_param_of_the_wrong_type_is_rejected(#[case] params: Value) {
assert!(serde_json::from_value::<VertexParams>(params).is_err());
}
#[rstest]
fn fields_name_every_param_once() {
let filled: Value = VertexParams::fields()
.map(|name| (name.to_string(), Value::from(name)))
.collect::<Map<_, _>>()
.into();
let params: VertexParams = serde_json::from_value(filled).unwrap();
let expected = VertexParams {
vertex_credentials: Some("vertex_credentials".into()),
vertex_project: Some("vertex_project".into()),
vertex_location: Some("vertex_location".into()),
vertex_ai_credentials: Some("vertex_ai_credentials".into()),
vertex_ai_project: Some("vertex_ai_project".into()),
vertex_ai_location: Some("vertex_ai_location".into()),
};
assert_eq!(params, expected);
assert_eq!(
VertexParams::fields().collect::<BTreeSet<_>>().len(),
VertexParams::fields().count()
);
}
#[rstest]
fn debug_output_does_not_expose_credentials() {
let params = VertexParams {
vertex_credentials: Some("current-secret".into()),
vertex_ai_credentials: Some("legacy-secret".into()),
vertex_project: Some("project-1".into()),
..VertexParams::default()
};
let debug = format!("{params:?}");
assert!(!debug.contains("current-secret"));
assert!(!debug.contains("legacy-secret"));
assert!(debug.contains("project-1"));
}
#[rstest]
#[case::current_spelling(Some("p"), Some("legacy"), &[("VERTEXAI_PROJECT", "env")], Some("p"))]
#[case::legacy_spelling(None, Some("legacy"), &[("VERTEXAI_PROJECT", "env")], Some("legacy"))]
#[case::blank_params_fall_to_the_environment(Some(" "), None, &[("VERTEXAI_PROJECT", "env")], Some("env"))]
#[case::nothing(None, None, &[], None)]
fn a_spec_resolves_from_the_params_then_the_environment(
#[case] vertex_project: Option<&str>,
#[case] vertex_ai_project: Option<&str>,
#[case] environment: &[(&str, &str)],
#[case] expected: Option<&str>,
) {
let params = VertexParams {
vertex_project: vertex_project.map(str::to_string),
vertex_ai_project: vertex_ai_project.map(str::to_string),
..VertexParams::default()
};
let env = |name: &str| {
environment
.iter()
.find(|(key, _)| *key == name)
.map(|(_, value)| value.to_string())
};
assert_eq!(
params.resolve(&VertexParams::PROJECT, &env).as_deref(),
expected
);
}

2
litellm-rust/crates/cache-response/AGENTS.md
View file

@ -24,6 +24,6 @@ Keep unary caching independent of stream-only methods. Store streams only after
Test each contract in its owner: storage capabilities in backend tests, envelopes and freshness here, reuse and replay in core, Python callback and fallback behavior at the bridge, and HTTP behavior at the gateway. Run backend contract checks and Python response-codec fixtures before exposing a new backend
`ScopedCache` requires an explicit shared or isolated scope at construction. Per-call `CachePolicy` controls reads, writes, expiry, and freshness without replacing the attached scope or service. `CacheOptions` binds that policy to an explicit scope for storage requests and has no default sharing policy. Versioned native envelopes reject incompatible API surfaces and versions as misses; this envelope is distinct from the legacy Python response codec
`ScopedCache` requires an explicit shared or isolated scope at construction. Per-call `CachePolicy` controls reads, writes, expiry, and freshness without replacing the attached scope or service. `CacheOptions` binds that policy to an explicit scope for storage requests and has no default sharing policy. Versioned native envelopes reject incompatible API surfaces and versions as misses. This envelope is distinct from the legacy Python response codec
Response storage is not the source of budget or rate-limit coordination dependencies. Keep counters, reservations, and atomic admission operations out of `ResponseCacheService`, including when both services happen to use Redis

20
litellm-rust/crates/callbacks-legacy-python/AGENTS.md
View file

@ -1,20 +1,20 @@
- Target invariants, not completion claims
- This crate owns compatibility for all existing Python callbacks and loggers, including `CustomLogger`. `mapping.rs` owns the executable call bindings and the inventory of Python-owned hooks. A Python-owned entry records an existing path, never permission to invoke it a second time. The native call adapter preserves the `Logging` contract (`function_setup`, the deployment hooks, `pre_call`/`post_call`, the sync and async success and failure fan-out, the deferred proxy release, the argument sharing those callbacks rely on)
- Smell test: if a future callback host (`callbacks-v1-python`, WASM, in-process Rust) could share a piece of this crate, it does not belong here
- SDK request policy (credential inheritance, the budget and retry-count limits) is a separate hook supplied by `python-bridge`; compose it after this adapter so logging adopts the final keyword view before policy mutates or rejects it
- The driver in `litellm-host-python`, the routes and core see one `PythonCallHooks` using the shared `CallEvent`; they never learn which Python objects consume a call
- SDK request policy (credential inheritance, the budget and retry-count limits) is a separate hook supplied by `python-bridge`. Compose it after this adapter so logging adopts the final keyword view before policy mutates or rejects it
- The driver in `litellm-host-python`, the routes and core see one `PythonCallHooks` using the shared `CallEvent`. They never learn which Python objects consume a call
- Every litellm Python internal Rust still borrows is a variant of `LegacyPython`, grouped by subsystem, with its signature pinned in `python_contract.json`
- The enum only shrinks: when Rust owns a subsystem, delete its group rather than adding a Rust path beside it
- Calling a user's own callback directly is permanent Python surface and gets its own type outside `LegacyPython`
- `PublicCall` is the caller's call as `Logging` sees it: the positional arguments, the keyword view as the call rewrites it (setup, deployment hook, preflight) and the bound request object backing omitted keywords; shared bridge composition hands it to `LegacyLogging`; routes use the neutral call boundary
- `PublicCall` is the caller's call as `Logging` sees it: the positional arguments, the keyword view as the call rewrites it (setup, deployment hook, preflight) and the bound request object backing omitted keywords. Shared bridge composition hands it to `LegacyLogging`. Routes use the neutral call boundary
- `LoggingOperation` selects legacy logging entrypoints and response handling. It belongs here rather than in shared inference data contracts
- `setup` reuses a `Logging` passed as `litellm_logging_obj` (the proxy and Router) and otherwise builds one through `function_setup`; which callbacks run is `Logging`'s decision, never this crate's
- Callbacks receive the caller's own objects and may mutate them; this crate alone carries that obligation
- Retain complete boundary arguments, opaque values, aliases, omitted/default distinctions and deliberate copies; preserve the deployment-hook kwargs view
- `setup` reuses a `Logging` passed as `litellm_logging_obj` (the proxy and Router) and otherwise builds one through `function_setup`. Which callbacks run is `Logging`'s decision, never this crate's
- Callbacks receive the caller's own objects and may mutate them. This crate alone carries that obligation
- Retain complete boundary arguments, opaque values, aliases, omitted/default distinctions and deliberate copies. Preserve the deployment-hook kwargs view
- Before `pre_call`, re-alias every body key whose value equals the caller's argument to the caller's own object, resolved through `litellm_host_python::lookup`
- Retain body/header roots from `pre_call` to `post_call`; in-place mutation reaches the wire, envelope field replacement is visible to later callbacks only
- Retain body/header roots from `pre_call` to `post_call`. In-place mutation reaches the wire, envelope field replacement is visible to later callbacks only
- Success and failure handlers receive the exact selected public response or exception
- A failure-handler error cannot suppress the other eligible family or replace the mapped provider error; a cancellation ends the call with no further dispatch
- Dispatch errors never replay provider work or trigger the opposite outcome; the proxy releases deferred success at most once
- A failure-handler error cannot suppress the other eligible family or replace the mapped provider error. A cancellation ends the call with no further dispatch
- Dispatch errors never replay provider work or trigger the opposite outcome. The proxy releases deferred success at most once
- Delivery follows the registry, not the callable's type: direct, awaited, executor-submitted, logging-worker and deferred paths stay distinct
- Traverse every retained Python edge; `close` is idempotent and restores the correlation context once
- Traverse every retained Python edge. `close` is idempotent and restores the correlation context once

Some files were not shown because too many files have changed in this diff Show more