scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-10-04 02:31:27 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 70
38245 commits 3112 branches 1436 tags 4.4 GiB
Commit graph

3 commits

Author SHA1 Message Date
mateo-berri
449a44dba3 compat-matrix: run from a GCP VM via systemd; drop docker + GHA 
The daily compat-matrix runs from the dedicated GCP VM
`litellm-compatibility-matrix-populator` rather than from a GitHub
Actions runner. The VM has no docker daemon, has `gh` already
authenticated against an account with `pull-requests: write` on
`BerriAI/litellm-docs`, and runs a long-lived litellm checkout we can
reuse across runs. That makes Docker, the GitHub App auth flow, and the
GHA workflow itself dead code.

Removed
-------

  * `.github/workflows/claude_code_compat_matrix.yml` — no longer
    triggers anything; the systemd timer in this PR owns the daily fire.
  * `docker_image_for_tag` + `DOCKER_IMAGE_BASE` constants and the two
    unit tests that covered them.
  * `_start_proxy(image, port)` / `_stop_proxy(container_id)` /
    `docker run` flow, replaced by direct `uv run litellm` subprocess
    management with a sigterm-the-process-group teardown.
  * `--skip-proxy` CLI flag (was only useful when the GHA workflow
    split docker-bringup from publish into separate jobs).
  * `docs_token` parameter and `DOCS_REPO_TOKEN` env var; `gh` on
    the VM is already authenticated, so we don't pass an explicit
    token through the publisher.

Added
-----

  * Persistent worktree flow in `publisher.py`. First run clones
    `BerriAI/litellm` into `~/litellm-cron-worktree/`; subsequent runs
    do `git fetch --tags && git checkout --force <stable-tag> &&
    uv sync --frozen`. Disk footprint is bounded because uv sync
    removes packages no longer pinned and `git clean -fdx -e .venv`
    wipes per-run cruft while keeping the venv around.
  * `tests/claude_code/cron_vm/` containing systemd units and a
    setup README:
    - `litellm-compat-matrix.service` (`Type=oneshot`, runs as the
      `mateo` user, sources `/etc/litellm-compat-matrix.env` for
      provider creds, hardened with `NoNewPrivileges` /
      `ProtectSystem=strict` / `PrivateTmp`);
    - `litellm-compat-matrix.timer` (`OnCalendar=*-*-* 06:00:00 UTC`,
      `Persistent=true` so a missed run fires when the VM is back up,
      `RandomizedDelaySec=10min`);
    - `.env.example` documenting the provider-credential surface;
    - `README.md` covering one-time install, daily operation,
      `journalctl` debugging, and the gotchas (proxy port `4100` to
      avoid colliding with a developer's `:4000`, `gh` token
      rotation, what to do after a Claude Code CLI upgrade).

Operator notes
--------------

  * The proxy now binds `:4100` by default so a developer SSH'd into
    the VM with their own `:4000` proxy isn't preempted by the cron.
  * The Claude Code CLI is exercised as-is from the system install;
    the populator does NOT `npm install` it. Operators upgrade the
    CLI by running `npm install -g @anthropic-ai/claude-code@latest`
    out of band, typically after watching a `--skip-publish` run to
    verify the matrix doesn't suddenly turn red.
  * 20 publisher unit tests pass (`pytest
    tests/claude_code/_publisher_unit_tests/`).
  * End-to-end validation on the VM happens after this PR lands as
    follow-up commits on the same branch — the systemd unit is
    `Type=oneshot` so a manual `systemctl start` reproduces the cron.
 2026-05-06 23:27:14 +00:00
mateo-berri
0e65ce1495 compat-matrix: open a docs-repo PR instead of direct-pushing main 
The daily Claude Code compatibility-matrix cron has been direct-pushing
`compatibility-matrix.json` to litellm-docs's main branch. Switch to
opening (or updating) a pull request so docs maintainers can review each
matrix update before it ships to readers.

Behavioural changes
-------------------

publisher.publish() now:
  * checks out a deterministic head branch
    (`compat-matrix/<litellm>-<claude>-<UTC-date>`) before staging the
    JSON, instead of committing on top of the docs branch directly;
  * `git push --force-with-lease` so a same-day rerun updates the
    existing branch (and therefore the existing PR), without
    overwriting any docs-maintainer fixup commit on the same branch;
  * shells out to `gh pr create` against `docs_repo` with a
    title/body that surfaces the resolved versions and a per-feature
    status summary, so reviewers can triage from the inbox;
  * treats 'a pull request for branch ... already exists' as success,
    so two cron runs on the same day produce one PR, not two.

Idempotency contract
--------------------

  * Same (litellm_version, claude_code_version, UTC date) -> same
    branch -> same PR. Verified by the new
    `test_pr_branch_name_is_deterministic_per_inputs` /
    `...changes_when_any_component_changes` tests.
  * Byte-identical JSON to the docs branch -> early-return before
    push, same as the previous direct-push path.
  * Empty version inputs are rejected up front so two distinct PRs
    can never silently collapse onto one branch.

Tests
-----

  * 8 new tests in `_publisher_unit_tests/test_publisher.py` cover
    `pr_branch_name`, `pr_title_for_matrix`, and `pr_body_for_matrix`
    (determinism, content, ordering, missing-provider rectangularity,
    empty-input rejection).
  * Existing 7 `commit_message_for_matrix` /
    `docker_image_for_tag` / `select_files_to_commit` tests are
    unchanged and still pass.

Workflow
--------

`.github/workflows/claude_code_compat_matrix.yml` updates only the
header doc comment to reflect that the GitHub App now needs
`pull-requests: write` in addition to `contents: write`. `gh` is
preinstalled on `ubuntu-latest` (also used by
`auto_update_price_and_context_window.yml`), so no install step is
needed.

Operator action required (one-time)
-----------------------------------

The compat-matrix GitHub App installation on `BerriAI/litellm-docs`
needs `pull-requests: write` added to its installation permissions
before the next cron run. Without it, the new `gh pr create` call
will fail with a 403; `compat-results.json` and
`compatibility-matrix.json` will still upload as workflow artifacts
for debugging.
 2026-05-06 23:27:14 +00:00
mateo-berri
797a449daf RALPH: compat matrix slice 4 - daily cron VM publishes matrix to docs (#26480, PRD #26476) 
Slice 4 of the Claude Code Compatibility Matrix: stand up the daily-cron
pipeline that publishes `compatibility-matrix.json` to the docs repo. After
this slice lands, the hand-authored matrix in the docs repo is replaced by
auto-generated output, and the docs page begins reflecting real test runs
against the latest stable LiteLLM release.

What landed:

- tests/claude_code/resolver.py
  Latest Stable LiteLLM Resolver. Calls the GitHub Releases API and
  returns the newest tag matching `v*-stable`. Sort is numeric on
  (major, minor, patch) so v1.10.0-stable correctly outranks
  v1.9.5-stable. Injectable `http_get` so tests run offline.

- tests/claude_code/publisher.py
  Daily-cron orchestrator. Resolves the latest stable tag, pulls
  `ghcr.io/berriai/litellm:<tag>`, starts it as the proxy, installs
  `@anthropic-ai/claude-code@latest`, runs `pytest tests/claude_code/`,
  invokes the Matrix JSON Builder, and direct-pushes
  `compatibility-matrix.json` to the docs repo's main branch using a
  GitHub App installation token (`DOCS_REPO_TOKEN`). Idempotent: a no-op
  if the JSON is byte-identical to what's already on main.

- tests/claude_code/_publisher_unit_tests/test_resolver.py
  test_publisher.py
  14 unit tests covering the small pure helpers — version sort,
  non-stable filtering, http-get injection, commit message determinism,
  Docker image-name builder, and the file allowlist that enforces the
  "only `compatibility-matrix.json` ever ships" guarantee. Per the PRD's
  "Testing Decisions" section, the publisher's full subprocess
  orchestration intentionally ships without a unit-test harness; the
  daily-cron failure surface is itself the test.

- .github/workflows/claude_code_compat_matrix.yml
  GitHub Actions workflow with three triggers (daily cron at 06:00 UTC,
  `release: published` filtered to `*-stable` tags, and
  `workflow_dispatch`). Mints a docs-repo installation token from a
  GitHub App scoped to `BerriAI/litellm-docs` only with `contents:
  write`, then runs the publisher.

- .gitignore
  Add `compatibility-matrix.json` (cron VM output).

Key decisions:

- "Isolated VM" is realized as a GitHub-hosted ubuntu-latest runner —
  every run gets a fresh ephemeral VM, and the always-latest Claude
  Code CLI is only ever installed inside that ephemeral environment,
  so a malicious or broken Claude Code release cannot affect the
  trusted PR-gate CI in CircleCI.
- File-level restriction on the GitHub App's broad `contents: write`
  scope is enforced by `select_files_to_commit` (script correctness),
  per the PRD's explicit acknowledgement that GitHub does not support
  file-path-scoped tokens.
- `release` runs are filtered to tags ending in `-stable` at the
  workflow level, so a `v1.84.0-rc1` release does not republish the
  matrix.
- Resolver and publisher live under `tests/claude_code/` alongside
  `matrix_builder.py` and `cli_driver.py` — production code that
  supports the test suite, kept colocated with it to match the slice
  1+2 layout.

Out of scope / blockers for next iteration:

- Provisioning the GitHub App itself (creating it under BerriAI's
  org, installing it on litellm-docs only, generating the private key
  and registering `COMPAT_MATRIX_APP_ID` / `COMPAT_MATRIX_APP_PRIVATE_KEY`
  as repo secrets) is an operator/infra step that cannot land via a
  code change in this repo.
- The first successful cron run is what removes the hand-authored
  `compatibility-matrix.json` from the docs repo and replaces it with
  generated output — that happens after this PR merges and the App is
  installed; not a code change here.

Tests: 34 -> 45 passing (added 7 resolver tests + 7 publisher helper
tests, all unit-only and offline). The 12 per-cell failures under
`tests/claude_code/basic_messaging_non_streaming/` remain by design —
they require a running proxy which the cron VM provides.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-06 23:27:14 +00:00