mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-05 08:07:05 +00:00
50c6a1f344
* fix(ci): run every helm test suite, not just the first one per file helm-unittest gained support for multiple suites in one test file in v0.5.0; CI and the Makefile both pinned v0.4.4, the last release that decodes a single YAML document per file. Any suite after a `---` separator was parsed away and its assertions never ran, while the summary still reported a clean pass. Upgrading the pin to v0.8.2, the newest release that installs under the pinned helm 3.11.1, brings the litellm-helm chart from 11 suites / 90 tests to 14 suites / 93 tests with no change to any test file. All the recovered tests pass. The run step now compares the number of declared `suite:` documents against the number of suites the runner reports, so the same class of silent skip fails the job loudly instead of passing quietly. The Makefile target upgrades a stale local plugin instead of swallowing the "already installed" error and leaving the developer on an old version. * ci: install helm-unittest from a pinned, checksum-verified artifact `helm plugin install <git url>` clones the plugin repo and executes its install hook, which downloads the release tarball itself. The old integrity step then checked the cloned repo's HEAD, which happens after the hook has already run and never covers the binary that was actually downloaded. The plugin now comes from a full pinned release URL, verified against the SHA-256 the project publishes in its helm-unittest-checksum.sha sidecar, before anything is unpacked or run. Nothing remote executes ahead of the check, and a re-published release asset fails the job instead of installing silently.
50 lines
1.8 KiB
YAML
50 lines
1.8 KiB
YAML
name: Helm unit test
|
|
|
|
on:
|
|
pull_request:
|
|
push:
|
|
branches:
|
|
- main
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
unit-test:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- name: Set up Helm 3.11.1
|
|
uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4 # v4.3.1
|
|
with:
|
|
version: "3.11.1"
|
|
|
|
- name: Download and verify Helm Unit Test Plugin
|
|
run: |
|
|
curl -fsSLo "$RUNNER_TEMP/helm-unittest.tgz" https://github.com/helm-unittest/helm-unittest/releases/download/v0.8.2/helm-unittest-linux-amd64-0.8.2.tgz
|
|
echo "56ab3091e6fa52a7c92ee951def9bed957f295d9ce98483aed404e748d7b3a94 $RUNNER_TEMP/helm-unittest.tgz" | sha256sum -c -
|
|
|
|
- name: Install Helm Unit Test Plugin
|
|
run: |
|
|
PLUGIN_DIR="$(helm env HELM_PLUGINS)/helm-unittest"
|
|
mkdir -p "$PLUGIN_DIR"
|
|
tar -xzf "$RUNNER_TEMP/helm-unittest.tgz" -C "$PLUGIN_DIR"
|
|
helm plugin list
|
|
|
|
- name: Run unit tests
|
|
run: |
|
|
for chart in helm/litellm-helm helm/litellm; do
|
|
declared="$(grep -h '^suite:' "$chart"/tests/*.yaml | wc -l | tr -d '[:space:]')"
|
|
output="$(mktemp)"
|
|
helm unittest -f 'tests/*.yaml' "$chart" | tee "$output"
|
|
executed="$(sed -n 's/^Test Suites:.*[[:space:]]\([0-9][0-9]*\) total$/\1/p' "$output")"
|
|
if [ "$declared" != "$executed" ]; then
|
|
echo "::error::$chart declares $declared test suites but helm-unittest ran $executed. Suites are being skipped silently, so their assertions never execute."
|
|
exit 1
|
|
fi
|
|
echo "$chart: all $declared declared test suites ran"
|
|
done
|