scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-22 00:31:44 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 51

ci: add OIDC-rooted keyless release pipeline with SLSA L3 provenance

Browse source 
Publishes LiteLLM to PyPI and GHCR entirely over OIDC, with no
long-lived signing keys or registry credentials:

- PyPI: OIDC Trusted Publisher upload with PEP 740 attestations,
  SLSA L3 build provenance via actions/attest-build-provenance,
  and detached keyless cosign signatures on the sdist and wheel.
- Docker: a reusable build-push-sign workflow for the three images
  (litellm, -database, -non-root), keyless cosign signing, and
  SLSA L3 provenance attached as an OCI referrer.
- Consumer-style verify jobs that re-check every signature and
  attestation the way a downstream user would (gh attestation
  verify, cosign verify), so a broken pipeline fails loudly.
- A regression test enforcing the supply-chain invariants:
  SHA-pinned actions, keyless-only, OIDC-only, no static key.

The static cosign.pub is removed; keyless verification roots in
Fulcio/Rekor, not a checked-in public key.
This commit is contained in:
lee-mcfaul2 2026-05-15 16:14:07 -04:00
parent 2f041a5224
commit ab81dd40ea
No known key found for this signature in database
GPG key ID: 672CE29E5348DA68
7 changed files with 916 additions and 36 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

162
.github/workflows/_publish-container.yml vendored Normal file
View file

@ -0,0 +1,162 @@
name: Reusable — build, push and keyless-sign a container image
on:
workflow_call:
inputs:
image-name:
type: string
required: true
dockerfile:
type: string
required: true
context:
type: string
default: "."
platforms:
type: string
default: "linux/amd64,linux/arm64"
tag:
type: string
required: true
commit-hash:
type: string
required: true
enable-docker-hub:
type: boolean
default: false
cosign-release:
type: string
default: "v3.0.6" # Keep in sync with publish_to_pypi.yml's COSIGN_VERSION
outputs:
digest:
description: "Image digest of the built+pushed image"
value: ${{ jobs.build.outputs.digest }}
image:
description: "Image reference (without digest)"
value: ${{ jobs.build.outputs.image }}
permissions: {}
jobs:
build:
runs-on: ubuntu-latest
timeout-minutes: 60
permissions:
id-token: write
contents: read
packages: write
attestations: write # required for actions/attest-build-provenance
outputs:
digest: ${{ steps.build.outputs.digest }}
image: ghcr.io/${{ github.repository_owner }}/${{ inputs.image-name }}
steps:
- name: Checkout source at release commit
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
ref: ${{ inputs.commit-hash }}
persist-credentials: false
- name: Set up QEMU
uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v4.0.0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12
- name: Log in to GHCR
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Log in to Docker Hub via OIDC
if: inputs.enable-docker-hub
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
with:
registry: docker.io
username: ${{ vars.DOCKERHUB_USERNAME }}
oidc-federation-id: ${{ vars.DOCKERHUB_OIDC_ID }}
- name: Build and push
id: build
uses: docker/build-push-action@0adf9959216b96bec444f325f1e493d4aa344497 # v6.14
with:
context: ${{ inputs.context }}
file: ${{ inputs.dockerfile }}
platforms: ${{ inputs.platforms }}
push: true
# Disabled deliberately: SLSA provenance is produced by the GitHub-native
# attest-build-provenance step below. BuildKit's own provenance attestation
# would alter the multi-arch manifest digest and break signature verification.
provenance: false
tags: |
ghcr.io/${{ github.repository_owner }}/${{ inputs.image-name }}:${{ inputs.tag }}
${{ inputs.enable-docker-hub && format('docker.io/litellm/{0}:{1}', inputs.image-name, inputs.tag) || '' }}
- name: Install cosign
uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2
with:
cosign-release: ${{ inputs.cosign-release }}
- name: Sign image (keyless)
env:
DIGEST: ${{ steps.build.outputs.digest }}
OWNER: ${{ github.repository_owner }}
IMAGE: ${{ inputs.image-name }}
ENABLE_DH: ${{ inputs.enable-docker-hub }}
run: |
set -euo pipefail
cosign sign --yes "ghcr.io/${OWNER}/${IMAGE}@${DIGEST}"
if [ "${ENABLE_DH}" = "true" ]; then
cosign sign --yes "docker.io/litellm/${IMAGE}@${DIGEST}"
fi
- name: SLSA build provenance (GitHub-native attestation)
uses: actions/attest-build-provenance@a2bbfa25375fe432b6a289bc6b6cd05ecd0c4c32 # v4.1.0
with:
subject-name: ghcr.io/${{ github.repository_owner }}/${{ inputs.image-name }}
subject-digest: ${{ steps.build.outputs.digest }}
push-to-registry: true
- name: Post-sign verify
env:
DIGEST: ${{ steps.build.outputs.digest }}
TAG: ${{ inputs.tag }}
IMAGE: ${{ inputs.image-name }}
OWNER: ${{ github.repository_owner }}
ENABLE_DH: ${{ inputs.enable-docker-hub }}
run: |
set -euo pipefail
TAG_ESC=$(printf '%s' "${TAG}" | sed 's/\./\\./g')
# When sign happens inside this reusable, the cert subject reflects the reusable's
# path (job_workflow_ref), not the orchestrator's (workflow_ref). Match accordingly.
IDENTITY_RE="^https://github\.com/${GITHUB_REPOSITORY}/\.github/workflows/_publish-container\.yml@refs/tags/${TAG_ESC}$"
cosign verify \
--certificate-identity-regexp="${IDENTITY_RE}" \
--certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
"ghcr.io/${OWNER}/${IMAGE}@${DIGEST}"
if [ "${ENABLE_DH}" = "true" ]; then
cosign verify \
--certificate-identity-regexp="${IDENTITY_RE}" \
--certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
"docker.io/litellm/${IMAGE}@${DIGEST}"
fi
- name: Cleanup on failure — delete just-pushed tag
if: failure()
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OWNER: ${{ github.repository_owner }}
TAG: ${{ inputs.tag }}
IMAGE: ${{ inputs.image-name }}
run: |
set +e
if gh api "/orgs/${OWNER}" --silent 2>/dev/null; then
PKG_BASE="/orgs/${OWNER}/packages/container/${IMAGE}"
else
PKG_BASE="/users/${OWNER}/packages/container/${IMAGE}"
fi
PACKAGE_VERSION_ID=$(gh api "${PKG_BASE}/versions" \
--jq ".[] | select(.metadata.container.tags[]? == \"${TAG}\") | .id" \
| head -1)
if [ -n "${PACKAGE_VERSION_ID}" ]; then
echo "Deleting partial push: ${IMAGE}:${TAG} (version_id=${PACKAGE_VERSION_ID})"
gh api -X DELETE "${PKG_BASE}/versions/${PACKAGE_VERSION_ID}"
else
echo "No matching package version found for ${IMAGE}:${TAG}; nothing to clean up."
fi
# Docker Hub cleanup (best-effort) when enabled.
if [ "${{ inputs.enable-docker-hub }}" = "true" ]; then
echo "Note: Docker Hub tag ${IMAGE}:${TAG} may also have been pushed."
echo "Docker Hub API requires its own credentials; manual cleanup may be needed."
fi

61
.github/workflows/create-release.yml vendored
View file

@ -52,40 +52,61 @@ jobs:
// are stable maintenance releases, not pre-releases.
const isPrerelease = /(?:rc|nightly|alpha|beta|[-.]dev)/i.test(tag);
const cosignSection = [
`## Verify Docker Image Signature`,
const verifySection = [
`## Verifying release artifacts`,
``,
`All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit \`0112e53\`](https://github.com/BerriAI/litellm/commit/0112e53046018d726492c814b3644b7d376029d0).`,
`All LiteLLM release artifacts (PyPI sdist+wheel and GHCR Docker images)`,
`are signed keyless via [Sigstore](https://sigstore.dev) and ship with`,
`[SLSA Build L3 provenance](https://slsa.dev). Each signature is bound to`,
`the exact GitHub Actions workflow file at the exact tag that produced it.`,
`Verification works fully offline against the public Sigstore TUF root.`,
``,
`**Verify using the pinned commit hash (recommended):**`,
``,
`A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key:`,
`### Docker image signature`,
``,
'```bash',
`cosign verify \\`,
` --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \\`,
` --certificate-identity-regexp='^https://github\\.com/BerriAI/litellm/\\.github/workflows/_publish-container\\.yml@refs/tags/${tag}$' \\`,
` --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \\`,
` ghcr.io/berriai/litellm:${tag}`,
'```',
``,
`**Verify using the release tag (convenience):**`,
``,
`Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules:`,
`### Docker image SLSA build provenance`,
``,
'```bash',
`cosign verify \\`,
` --key https://raw.githubusercontent.com/BerriAI/litellm/${tag}/cosign.pub \\`,
` ghcr.io/berriai/litellm:${tag}`,
`gh attestation verify oci://ghcr.io/berriai/litellm:${tag} --owner BerriAI`,
'```',
``,
`Expected output:`,
`### PyPI publish attestation (PEP 740)`,
``,
'```',
`The following checks were performed on each of these signatures:`,
` - The cosign claims were validated`,
` - The signatures were verified against the specified public key`,
'```bash',
`# pip 24.1+ automatically verifies PEP 740 attestations on install.`,
`pip install --index-url https://pypi.org/simple/ litellm==${tag.replace(/^v/, '')}`,
'```',
``,
`---`,
`### PyPI SLSA build provenance (GitHub native)`,
``,
'```bash',
`# Download wheel + sdist from this release first, then:`,
`gh attestation verify <downloaded-wheel-or-sdist> --owner BerriAI`,
'```',
``,
`### PyPI cosign detached signatures (offline-verifiable)`,
``,
'```bash',
`# Each .whl and .tar.gz on this release has a sibling .sigstore bundle.`,
`cosign verify-blob \\`,
` --bundle <artifact>.sigstore \\`,
` --new-bundle-format \\`,
` --certificate-identity-regexp='^https://github\\.com/BerriAI/litellm/\\.github/workflows/publish_to_pypi\\.yml@refs/tags/${tag}$' \\`,
` --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \\`,
` <artifact>`,
'```',
``,
`### Offline / airgap verification`,
``,
`Keyless verification works fully offline given the artifact, the signed`,
`bundle, and a pre-staged Sigstore TUF root (~10 KB). See`,
`[cosign offline verification](https://docs.sigstore.dev/cosign/verifying/verify/#offline-verification).`,
``,
].join('\n');
@ -101,7 +122,7 @@ jobs:
tag_name: tag,
});
const updatedBody = cosignSection + (response.data.body ?? '');
const updatedBody = verifySection + (response.data.body ?? '');
await github.rest.repos.updateRelease({
owner: context.repo.owner,
repo: context.repo.repo,

330
.github/workflows/publish_to_pypi.yml vendored Normal file
View file

@ -0,0 +1,330 @@
name: Publish to PyPI
on:
workflow_dispatch:
jobs:
preflight-checks:
name: Preflight Checks
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
# No environment — read-only checks, no approval needed
outputs:
needs_publish: ${{ steps.check-litellm.outputs.needs_publish }}
version: ${{ steps.check-litellm.outputs.version }}
steps:
- name: Enforce tag-ref dispatch
run: |
set -euo pipefail
case "${GITHUB_REF}" in
refs/tags/*) echo "Dispatch ref OK: ${GITHUB_REF}" ;;
*)
echo "::error::Dispatch this workflow from a release tag (refs/tags/*), not a branch. github.ref=${GITHUB_REF}. The keyless cosign certificate identity binds to the dispatch ref; a branch dispatch produces artifacts whose signatures fail the verification commands shipped in the release notes."
exit 1 ;;
esac
- name: Checkout repo
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"
- name: Set up uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
version: "0.10.9"
enable-cache: false
- name: Check litellm version on PyPI
id: check-litellm
run: |
VERSION=$(python - <<'PY'
import tomllib
with open("pyproject.toml", "rb") as f:
print(tomllib.load(f)["project"]["version"])
PY
)
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
echo "Checking if litellm $VERSION exists on PyPI..."
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" "https://pypi.org/pypi/litellm/$VERSION/json")
if [ "$HTTP_STATUS" = "200" ]; then
echo "litellm $VERSION already exists on PyPI. Skipping publish."
echo "needs_publish=false" >> "$GITHUB_OUTPUT"
else
echo "litellm $VERSION not found on PyPI. Publish needed."
echo "needs_publish=true" >> "$GITHUB_OUTPUT"
fi
- name: Sanity check proxy-extras version
run: |
# Read pinned version from project optional dependencies
PYPROJECT_VERSION=$(python3 - <<'PY'
import sys
import tomllib
with open("pyproject.toml", "rb") as f:
proxy_requirements = tomllib.load(f)["project"]["optional-dependencies"]["proxy"]
version = None
for requirement in proxy_requirements:
normalized = requirement.split(";", 1)[0].strip()
if not normalized.startswith("litellm-proxy-extras"):
continue
parts = normalized.split("==", 1)
if len(parts) == 2 and parts[0].strip() == "litellm-proxy-extras":
candidate = parts[1].strip()
if candidate:
version = candidate
break
if version is None:
print(
"::error::Could not find an exact litellm-proxy-extras pin in project.optional-dependencies.proxy",
file=sys.stderr,
)
sys.exit(1)
print(version)
PY
)
echo "pyproject.toml pins litellm-proxy-extras version: $PYPROJECT_VERSION"
# Check that the pinned version exists on PyPI
echo "Checking if litellm-proxy-extras $PYPROJECT_VERSION exists on PyPI..."
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" "https://pypi.org/pypi/litellm-proxy-extras/$PYPROJECT_VERSION/json")
if [ "$HTTP_STATUS" != "200" ]; then
echo "::error::litellm-proxy-extras $PYPROJECT_VERSION is not published on PyPI yet. Publish it before releasing litellm."
exit 1
fi
echo "litellm-proxy-extras $PYPROJECT_VERSION exists on PyPI. Sanity check passed."
build:
name: Build distribution
needs: preflight-checks
if: needs.preflight-checks.outputs.needs_publish == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
# No id-token / contents:write / environment here on purpose: this job runs
# untrusted build tooling (uv build, twine, project deps). Isolating it
# means a compromised build dependency cannot mint the OIDC token or reach
# the publishing environment — the trusted-publishing credential only
# exists in the separate `publish-litellm` job below.
permissions:
contents: read
steps:
- name: Checkout
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"
- name: Set up uv
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
with:
version: "0.10.9"
enable-cache: false
- name: Copy model prices backup
run: cp model_prices_and_context_window.json litellm/model_prices_and_context_window_backup.json
- name: Build package
run: |
rm -rf build dist
uv build
- name: Verify build artifacts
env:
EXPECTED_VERSION: ${{ needs.preflight-checks.outputs.version }}
run: |
set -euo pipefail
ls -la dist/
ls dist/*.tar.gz
ls dist/*.whl
ls dist/ | grep -q -- "-${EXPECTED_VERSION}" || {
echo "::error::Built artifacts do not match expected version $EXPECTED_VERSION"
exit 1
}
- name: Validate package metadata
run: uv tool run --from 'twine==6.2.0' twine check dist/*
- name: Upload built distribution
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: dist
path: dist/
if-no-files-found: error
retention-days: 1
publish-litellm:
name: Publish litellm to PyPI
needs: [preflight-checks, build]
if: needs.preflight-checks.outputs.needs_publish == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
id-token: write
contents: write
attestations: write # required for actions/attest-build-provenance
environment: pypi-publish
steps:
- name: Download built distribution
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
name: dist
path: dist/
- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e # v1.13.0
with:
attestations: true
verify-metadata: true
verbose: true
- name: Verify PEP 740 attestation landed
env:
VERSION: ${{ needs.preflight-checks.outputs.version }}
PROJECT: litellm
run: |
set -euo pipefail
# PyPI propagation can take 30-90s after publish; wait before first check.
sleep 30
HOST="https://pypi.org"
for ARTIFACT in dist/*.whl dist/*.tar.gz; do
FILENAME=$(basename "$ARTIFACT")
URL="$HOST/integrity/$PROJECT/$VERSION/$FILENAME/provenance"
echo "Checking $URL"
RESPONSE=$(curl -fsSL --retry 5 --retry-delay 60 --retry-all-errors --max-time 30 "$URL")
BUNDLE_COUNT=$(echo "$RESPONSE" | python3 -c "
import json, sys
try:
d = json.loads(sys.stdin.read())
except json.JSONDecodeError as e:
sys.stderr.write(f'::error::PyPI returned non-JSON response: {e}\n')
sys.exit(1)
bundles = d.get('attestation_bundles') or []
print(sum(len(b.get('attestations') or []) for b in bundles))
")
if [ "$BUNDLE_COUNT" -lt 1 ]; then
echo "::error::No PEP 740 attestation found for $FILENAME at $URL"
exit 1
fi
echo " attestations: $BUNDLE_COUNT"
done
- name: Upload sdist and wheel to GitHub release
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
VERSION: ${{ needs.preflight-checks.outputs.version }}
run: |
set -euo pipefail
gh release upload "v${VERSION}" dist/*.whl dist/*.tar.gz \
--repo "${GITHUB_REPOSITORY}" \
--clobber
- name: SLSA build provenance (GitHub-native attestation)
uses: actions/attest-build-provenance@a2bbfa25375fe432b6a289bc6b6cd05ecd0c4c32 # v4.1.0
with:
subject-path: |
dist/*.whl
dist/*.tar.gz
- name: Install cosign
run: |
set -euo pipefail
COSIGN_VERSION=v3.0.6
COSIGN_SHA256=c956e5dfcac53d52bcf058360d579472f0c1d2d9b69f55209e256fe7783f4c74
mkdir -p "$HOME/.local/bin"
curl -fsSL \
"https://github.com/sigstore/cosign/releases/download/${COSIGN_VERSION}/cosign-linux-amd64" \
-o "$HOME/.local/bin/cosign"
echo "${COSIGN_SHA256} $HOME/.local/bin/cosign" | sha256sum -c -
chmod +x "$HOME/.local/bin/cosign"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
"$HOME/.local/bin/cosign" version --json | head -3
- name: Sign each artifact with cosign (detached sigstore bundle)
env:
COSIGN_YES: 'true'
run: |
set -euo pipefail
for f in dist/*.whl dist/*.tar.gz; do
[ -f "${f}" ] || continue
cosign sign-blob --bundle "${f}.sigstore" --new-bundle-format "${f}"
done
- name: Upload cosign bundles to GitHub release
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
VERSION: ${{ needs.preflight-checks.outputs.version }}
run: |
set -euo pipefail
gh release upload "v${VERSION}" dist/*.whl.sigstore dist/*.tar.gz.sigstore \
--repo "${GITHUB_REPOSITORY}" \
--clobber
verify-slsa:
name: Verify provenance + cosign signatures (consumer-style)
needs: [preflight-checks, publish-litellm]
if: needs.preflight-checks.outputs.needs_publish == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
attestations: read
env:
VERSION: ${{ needs.preflight-checks.outputs.version }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- name: Install cosign
run: |
set -euo pipefail
COSIGN_VERSION=v3.0.6
COSIGN_SHA256=c956e5dfcac53d52bcf058360d579472f0c1d2d9b69f55209e256fe7783f4c74
mkdir -p "$HOME/.local/bin"
curl -fsSL \
"https://github.com/sigstore/cosign/releases/download/${COSIGN_VERSION}/cosign-linux-amd64" \
-o "$HOME/.local/bin/cosign"
echo "${COSIGN_SHA256} $HOME/.local/bin/cosign" | sha256sum -c -
chmod +x "$HOME/.local/bin/cosign"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Download artifacts + cosign bundles from GitHub release
run: |
set -euo pipefail
mkdir -p verify && cd verify
gh release download "v${VERSION}" \
--repo "$GITHUB_REPOSITORY" \
--pattern "*.whl" \
--pattern "*.tar.gz" \
--pattern "*.whl.sigstore" \
--pattern "*.tar.gz.sigstore"
ls -la
- name: Verify SLSA build provenance via gh attestation
run: |
set -euo pipefail
cd verify
for f in *.whl *.tar.gz; do
[ -f "${f}" ] || continue
gh attestation verify "${f}" --owner "${GITHUB_REPOSITORY_OWNER}"
done
- name: Verify cosign detached signatures
run: |
set -euo pipefail
cd verify
for f in *.whl *.tar.gz; do
[ -f "${f}" ] || continue
[ -f "${f}.sigstore" ] || continue
cosign verify-blob \
--bundle "${f}.sigstore" \
--new-bundle-format \
--certificate-identity-regexp="^https://github\.com/${GITHUB_REPOSITORY}/\.github/workflows/publish_to_pypi\.yml@refs/tags/.+$" \
--certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
"${f}"
done
echo "All consumer-style verifications passed."

173
.github/workflows/release-docker.yml vendored Normal file
View file

@ -0,0 +1,173 @@
name: Build, Publish and Sign LiteLLM Docker images
on:
workflow_dispatch:
inputs:
tag:
description: "Release tag (e.g. v1.83.14-stable)"
required: true
commit_hash:
description: "Source commit hash the tag must resolve to"
required: true
release_type:
description: "stable | nightly | rc | poc"
type: string
default: "stable"
enable_docker_hub:
description: "Also push to docker.io/litellm/* (requires Docker Hub OIDC federation)"
type: boolean
default: false
permissions: {}
jobs:
preflight:
name: Preflight tag/commit verification
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
environment: docker-release
# Only run in the canonical repository — never in forks.
if: github.repository == 'BerriAI/litellm'
outputs:
tag: ${{ inputs.tag }}
commit_hash: ${{ inputs.commit_hash }}
steps:
- name: Enforce tag-ref dispatch
env:
EXPECTED_REF: refs/tags/${{ inputs.tag }}
run: |
set -euo pipefail
if [ "${GITHUB_REF}" != "${EXPECTED_REF}" ]; then
echo "::error::Dispatch this workflow from the release tag, not a branch. github.ref=${GITHUB_REF}, expected ${EXPECTED_REF}. The keyless cosign certificate identity binds to the dispatch ref; a branch dispatch would push images and then fail signature verification."
exit 1
fi
echo "Dispatch ref OK: ${GITHUB_REF}"
- name: Checkout
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
fetch-depth: 0
ref: ${{ inputs.tag }}
persist-credentials: false
- name: Verify tag resolves to commit_hash
env:
TAG: ${{ inputs.tag }}
EXPECTED: ${{ inputs.commit_hash }}
run: |
set -euo pipefail
RESOLVED=$(git rev-parse "${TAG}^{commit}")
if [ "${RESOLVED}" != "${EXPECTED}" ]; then
echo "::error::Tag ${TAG} resolves to ${RESOLVED}, expected ${EXPECTED}"
exit 1
fi
echo "Tag ${TAG} -> ${RESOLVED} OK"
build-litellm:
name: Build, push, sign — litellm
needs: preflight
permissions:
id-token: write
contents: read
packages: write
attestations: write
uses: ./.github/workflows/_publish-container.yml
with:
image-name: litellm
dockerfile: ./Dockerfile
context: .
tag: ${{ needs.preflight.outputs.tag }}
commit-hash: ${{ needs.preflight.outputs.commit_hash }}
enable-docker-hub: ${{ inputs.enable_docker_hub }}
build-litellm-database:
name: Build, push, sign — litellm-database
needs: preflight
permissions:
id-token: write
contents: read
packages: write
attestations: write
uses: ./.github/workflows/_publish-container.yml
with:
image-name: litellm-database
dockerfile: ./docker/Dockerfile.database
context: .
tag: ${{ needs.preflight.outputs.tag }}
commit-hash: ${{ needs.preflight.outputs.commit_hash }}
enable-docker-hub: ${{ inputs.enable_docker_hub }}
build-litellm-non-root:
name: Build, push, sign — litellm-non-root
needs: preflight
permissions:
id-token: write
contents: read
packages: write
attestations: write
uses: ./.github/workflows/_publish-container.yml
with:
image-name: litellm-non-root
dockerfile: ./docker/Dockerfile.non_root
context: .
tag: ${{ needs.preflight.outputs.tag }}
commit-hash: ${{ needs.preflight.outputs.commit_hash }}
enable-docker-hub: ${{ inputs.enable_docker_hub }}
verify-all:
name: Final verification (consumer-style)
needs:
- preflight
- build-litellm
- build-litellm-database
- build-litellm-non-root
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: read
packages: read
attestations: read
strategy:
fail-fast: false
matrix:
include:
- name: litellm
digest: ${{ needs.build-litellm.outputs.digest }}
- name: litellm-database
digest: ${{ needs.build-litellm-database.outputs.digest }}
- name: litellm-non-root
digest: ${{ needs.build-litellm-non-root.outputs.digest }}
steps:
- name: Install cosign
uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2
with:
cosign-release: 'v3.0.6' # Keep in sync with _publish-container.yml's default
- name: Log in to GHCR (read-only, for attestation referrer pull)
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Verify cosign signature
env:
DIGEST: ${{ matrix.digest }}
TAG: ${{ needs.preflight.outputs.tag }}
run: |
set -euo pipefail
TAG_ESC=$(printf '%s' "${TAG}" | sed 's/\./\\./g')
# Cert identity reflects the reusable workflow that signed (job_workflow_ref).
IDENTITY_RE="^https://github\.com/${GITHUB_REPOSITORY}/\.github/workflows/_publish-container\.yml@refs/tags/${TAG_ESC}$"
cosign verify \
--certificate-identity-regexp="${IDENTITY_RE}" \
--certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
"ghcr.io/${{ github.repository_owner }}/${{ matrix.name }}@${DIGEST}"
- name: Verify SLSA provenance via gh attestation
env:
DIGEST: ${{ matrix.digest }}
IMAGE: ${{ matrix.name }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
set -euo pipefail
gh attestation verify \
"oci://ghcr.io/${GITHUB_REPOSITORY_OWNER}/${IMAGE}@${DIGEST}" \
--owner "${GITHUB_REPOSITORY_OWNER}"

19
README.md
View file

@ -424,29 +424,24 @@ Support for more providers. Missing a provider or LLM Platform, raise a [feature
### Verify Docker Image Signatures
All LiteLLM Docker images published to GHCR are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](https://github.com/BerriAI/litellm/commit/0112e53046018d726492c814b3644b7d376029d0).
All LiteLLM Docker images published to GHCR are **keyless-signed** with [cosign](https://docs.sigstore.dev/cosign/overview/) (Sigstore — Fulcio + Rekor) and ship with [SLSA Build L3](https://slsa.dev) provenance. There is no static public key to trust: each signature and attestation is cryptographically bound to the exact GitHub Actions workflow, repository, and release tag that produced the image. Verification works fully offline against the public Sigstore TUF root.
**Verify using the pinned commit hash (recommended):**
A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key:
**Verify the cosign signature:**
```bash
cosign verify \
--key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
--certificate-identity-regexp='^https://github\.com/BerriAI/litellm/\.github/workflows/_publish-container\.yml@refs/tags/<release-tag>$' \
--certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
ghcr.io/berriai/litellm:<release-tag>
```
**Verify using a release tag (convenience):**
Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules:
**Verify the SLSA build provenance:**
```bash
cosign verify \
--key https://raw.githubusercontent.com/BerriAI/litellm/<release-tag>/cosign.pub \
ghcr.io/berriai/litellm:<release-tag>
gh attestation verify oci://ghcr.io/berriai/litellm:<release-tag> --owner BerriAI
```
Replace `<release-tag>` with the version you are deploying (e.g. `v1.83.0-stable`).
Replace `<release-tag>` with the version you are deploying (e.g. `v1.83.0-stable`). The same two commands work for the `litellm-database` and `litellm-non-root` images.
---

4
cosign.pub
View file

@ -1,4 +0,0 @@
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEKi4ivqGpE231OGH50PKbqy1Y1Kkb
POJC8+i2Wko82gBOUCe3M0Vw86H/4rhUhfoYEti4gdJ9wZbYmK0I2EE96g==
-----END PUBLIC KEY-----

203
tests/test_litellm/test_release_workflow_hardening.py Normal file
View file

@ -0,0 +1,203 @@
"""Enforces invariants on the release/publish workflows.
This test is the regression net for the hardening introduced in the
'bulletproof release pipeline' PR. Each assertion catches a specific
class of supply-chain regression. The test runs without secrets or
network access — it inspects the workflow YAML files in the repo and
asserts file-shape invariants only.
"""
from __future__ import annotations
import re
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parents[2]
WORKFLOWS_DIR = REPO_ROOT / ".github" / "workflows"
RELEASE_WORKFLOWS = [
"publish_to_pypi.yml",
"release-docker.yml",
"create-release.yml",
"_publish-container.yml",
]
SHA_PIN_RE = re.compile(r"@[0-9a-f]{40}\b")
USES_LINE_RE = re.compile(r"^\s*-?\s*uses:\s*(\S+)")
def _read_workflow_text(name: str) -> str:
path = WORKFLOWS_DIR / name
assert path.exists(), f"Expected workflow {name} to exist at {path}"
return path.read_text(encoding="utf-8")
@pytest.mark.parametrize("workflow", RELEASE_WORKFLOWS)
def test_release_workflows_only_use_sha_pinned_actions(workflow: str) -> None:
"""Every `uses:` in a release workflow must reference a 40-hex SHA, not a tag."""
text = _read_workflow_text(workflow)
offenders: list[tuple[int, str]] = []
for lineno, line in enumerate(text.splitlines(), start=1):
m = USES_LINE_RE.match(line)
if not m:
continue
ref = m.group(1)
# Local reusable workflows (./.github/workflows/...) have no @ref
if ref.startswith("./"):
continue
if "@" not in ref:
offenders.append((lineno, line.rstrip()))
continue
if not SHA_PIN_RE.search(ref):
offenders.append((lineno, line.rstrip()))
assert not offenders, (
f"{workflow} contains non-SHA-pinned action references:\n"
+ "\n".join(f" L{n}: {ln}" for n, ln in offenders)
)
def test_publish_pypi_has_explicit_attestations_true() -> None:
"""The PyPI publish step must explicitly set attestations: true."""
text = _read_workflow_text("publish_to_pypi.yml")
assert re.search(
r"attestations:\s*true", text
), "publish_to_pypi.yml must set 'attestations: true' explicitly"
def test_publish_pypi_does_not_pass_password_to_pypi_publish() -> None:
"""No `password:` input is passed to pypa/gh-action-pypi-publish (OIDC only)."""
text = _read_workflow_text("publish_to_pypi.yml")
lines = text.splitlines()
for idx, line in enumerate(lines):
if "pypa/gh-action-pypi-publish" in line:
window = "\n".join(lines[idx : idx + 30])
assert "password:" not in window, (
"Found `password:` near pypa/gh-action-pypi-publish in publish_to_pypi.yml — "
"static credentials must not be passed; OIDC is mandatory"
)
def test_publish_container_uses_keyless_cosign() -> None:
"""The cosign sign step must NOT pass --key (asserts keyless via Fulcio)."""
text = _read_workflow_text("_publish-container.yml")
cosign_sign_matches = re.findall(r"cosign sign[^\n]*", text)
assert cosign_sign_matches, "_publish-container.yml must contain at least one `cosign sign` invocation"
for invocation in cosign_sign_matches:
assert "--key" not in invocation, (
f"cosign sign invocation contains --key: {invocation!r}. "
"Keyless signing (Fulcio + OIDC) is mandatory."
)
def test_publish_container_login_steps_have_no_password() -> None:
"""No docker login step in _publish-container.yml passes a static password."""
text = _read_workflow_text("_publish-container.yml")
# Only acceptable usage of password: is `secrets.GITHUB_TOKEN` for GHCR.
forbidden = re.findall(
r"password:\s*\$\{\{\s*secrets\.(?!GITHUB_TOKEN\b)[A-Z_][A-Z0-9_]*\s*\}\}",
text,
)
assert not forbidden, (
f"_publish-container.yml passes static secrets as docker login passwords: {forbidden}. "
"Only ${{ secrets.GITHUB_TOKEN }} (for GHCR auth) is permitted; Docker Hub must use OIDC."
)
@pytest.mark.parametrize("workflow", RELEASE_WORKFLOWS)
def test_release_workflows_do_not_use_slsa_github_generator(workflow: str) -> None:
"""slsa-github-generator's reusables are abandoned here (broken container
aggregator / public-fork misdetection). Provenance must come from
actions/attest-build-provenance instead."""
text = _read_workflow_text(workflow)
assert "slsa-framework/slsa-github-generator" not in text, (
f"{workflow} references slsa-framework/slsa-github-generator. "
"Use actions/attest-build-provenance for GitHub-native SLSA provenance."
)
def test_container_provenance_uses_attest_build_provenance() -> None:
"""The container build must generate SLSA provenance via
actions/attest-build-provenance with push-to-registry."""
text = _read_workflow_text("_publish-container.yml")
assert "actions/attest-build-provenance@" in text, (
"_publish-container.yml must use actions/attest-build-provenance "
"to generate SLSA build provenance for the pushed image"
)
assert re.search(r"push-to-registry:\s*true", text), (
"_publish-container.yml must set 'push-to-registry: true' so the "
"provenance attestation is attached as an OCI referrer"
)
@pytest.mark.parametrize("workflow", RELEASE_WORKFLOWS)
def test_cosign_identity_regexps_are_tag_scoped(workflow: str) -> None:
"""Keyless cosign identity regexps must bind signatures to a tag ref only.
Allowing refs/heads means a branch-dispatched run produces signatures that
pass CI but fail the tag-scoped verify commands shipped to consumers."""
text = _read_workflow_text(workflow)
offenders = [
line.strip()
for line in text.splitlines()
if "certificate-identity-regexp" in line
and ("refs/heads" in line or "(heads|tags)" in line or "(tags|heads)" in line)
]
assert not offenders, (
f"{workflow} has a cosign identity regexp that accepts a non-tag ref:\n"
+ "\n".join(f" {o}" for o in offenders)
+ "\nSignatures must bind to refs/tags only."
)
def test_cosign_version_is_consistent_across_release_workflows() -> None:
"""All release workflows must install the same cosign version. Mixing
cosign majors (e.g. 2.x sign vs 3.x verify) risks signature/bundle
format drift between the producing and verifying steps."""
version_re = re.compile(
r"""(?:cosign-release:\s*['"]?|COSIGN_VERSION=)(v\d+\.\d+\.\d+)"""
)
found: dict[str, set[str]] = {}
for workflow in RELEASE_WORKFLOWS:
text = _read_workflow_text(workflow)
versions = set(version_re.findall(text))
if versions:
found[workflow] = versions
all_versions = {v for vs in found.values() for v in vs}
assert len(all_versions) == 1, (
f"Inconsistent cosign versions across release workflows: {found}. "
"Pin a single cosign version everywhere."
)
def test_readme_documents_keyless_verification_only() -> None:
"""README image-verification docs must not reference a static cosign key.
Keyless signing has no checked-in public key; documenting `--key`/cosign.pub
sends users a verification command that cannot succeed."""
readme = (REPO_ROOT / "README.md").read_text(encoding="utf-8")
assert "cosign.pub" not in readme, (
"README references cosign.pub — keyless signing uses no static key"
)
assert "--key http" not in readme and "cosign verify --key" not in readme, (
"README documents keyful cosign verification; signing is keyless"
)
def test_release_body_instructions_match_asbuilt_slsa() -> None:
"""create-release.yml injects per-release verify instructions. SLSA
provenance is verified with `gh attestation verify` — the pipeline uses
actions/attest-build-provenance, not slsa-github-generator, so a
slsa-verifier instruction would fail for consumers."""
text = _read_workflow_text("create-release.yml")
assert "slsa-verifier" not in text, (
"create-release.yml still instructs consumers to use slsa-verifier; "
"as-built provenance is verified via `gh attestation verify`"
)
def test_cosign_pub_is_absent_from_repo_root() -> None:
"""The static cosign.pub key must not be present at repo root."""
assert not (REPO_ROOT / "cosign.pub").exists(), (
"cosign.pub exists at repo root. Keyless signing does not use a "
"static public key — delete cosign.pub."
)