2 commits

Author	SHA1	Message	Date
mateo-berri	0e65ce1495	compat-matrix: open a docs-repo PR instead of direct-pushing main ... The daily Claude Code compatibility-matrix cron has been direct-pushing `compatibility-matrix.json` to litellm-docs's main branch. Switch to opening (or updating) a pull request so docs maintainers can review each matrix update before it ships to readers. Behavioural changes ------------------- publisher.publish() now: * checks out a deterministic head branch (`compat-matrix/<litellm>-<claude>-<UTC-date>`) before staging the JSON, instead of committing on top of the docs branch directly; * `git push --force-with-lease` so a same-day rerun updates the existing branch (and therefore the existing PR), without overwriting any docs-maintainer fixup commit on the same branch; * shells out to `gh pr create` against `docs_repo` with a title/body that surfaces the resolved versions and a per-feature status summary, so reviewers can triage from the inbox; * treats 'a pull request for branch ... already exists' as success, so two cron runs on the same day produce one PR, not two. Idempotency contract -------------------- * Same (litellm_version, claude_code_version, UTC date) -> same branch -> same PR. Verified by the new `test_pr_branch_name_is_deterministic_per_inputs` / `...changes_when_any_component_changes` tests. * Byte-identical JSON to the docs branch -> early-return before push, same as the previous direct-push path. * Empty version inputs are rejected up front so two distinct PRs can never silently collapse onto one branch. Tests ----- * 8 new tests in `_publisher_unit_tests/test_publisher.py` cover `pr_branch_name`, `pr_title_for_matrix`, and `pr_body_for_matrix` (determinism, content, ordering, missing-provider rectangularity, empty-input rejection). * Existing 7 `commit_message_for_matrix` / `docker_image_for_tag` / `select_files_to_commit` tests are unchanged and still pass. Workflow -------- `.github/workflows/claude_code_compat_matrix.yml` updates only the header doc comment to reflect that the GitHub App now needs `pull-requests: write` in addition to `contents: write`. `gh` is preinstalled on `ubuntu-latest` (also used by `auto_update_price_and_context_window.yml`), so no install step is needed. Operator action required (one-time) ----------------------------------- The compat-matrix GitHub App installation on `BerriAI/litellm-docs` needs `pull-requests: write` added to its installation permissions before the next cron run. Without it, the new `gh pr create` call will fail with a 403; `compat-results.json` and `compatibility-matrix.json` will still upload as workflow artifacts for debugging.	2026-05-06 23:27:14 +00:00
mateo-berri	797a449daf	RALPH: compat matrix slice 4 - daily cron VM publishes matrix to docs (#26480, PRD #26476) ... Slice 4 of the Claude Code Compatibility Matrix: stand up the daily-cron pipeline that publishes `compatibility-matrix.json` to the docs repo. After this slice lands, the hand-authored matrix in the docs repo is replaced by auto-generated output, and the docs page begins reflecting real test runs against the latest stable LiteLLM release. What landed: - tests/claude_code/resolver.py Latest Stable LiteLLM Resolver. Calls the GitHub Releases API and returns the newest tag matching `v*-stable`. Sort is numeric on (major, minor, patch) so v1.10.0-stable correctly outranks v1.9.5-stable. Injectable `http_get` so tests run offline. - tests/claude_code/publisher.py Daily-cron orchestrator. Resolves the latest stable tag, pulls `ghcr.io/berriai/litellm:<tag>`, starts it as the proxy, installs `@anthropic-ai/claude-code@latest`, runs `pytest tests/claude_code/`, invokes the Matrix JSON Builder, and direct-pushes `compatibility-matrix.json` to the docs repo's main branch using a GitHub App installation token (`DOCS_REPO_TOKEN`). Idempotent: a no-op if the JSON is byte-identical to what's already on main. - tests/claude_code/_publisher_unit_tests/test_resolver.py test_publisher.py 14 unit tests covering the small pure helpers — version sort, non-stable filtering, http-get injection, commit message determinism, Docker image-name builder, and the file allowlist that enforces the "only `compatibility-matrix.json` ever ships" guarantee. Per the PRD's "Testing Decisions" section, the publisher's full subprocess orchestration intentionally ships without a unit-test harness; the daily-cron failure surface is itself the test. - .github/workflows/claude_code_compat_matrix.yml GitHub Actions workflow with three triggers (daily cron at 06:00 UTC, `release: published` filtered to `*-stable` tags, and `workflow_dispatch`). Mints a docs-repo installation token from a GitHub App scoped to `BerriAI/litellm-docs` only with `contents: write`, then runs the publisher. - .gitignore Add `compatibility-matrix.json` (cron VM output). Key decisions: - "Isolated VM" is realized as a GitHub-hosted ubuntu-latest runner — every run gets a fresh ephemeral VM, and the always-latest Claude Code CLI is only ever installed inside that ephemeral environment, so a malicious or broken Claude Code release cannot affect the trusted PR-gate CI in CircleCI. - File-level restriction on the GitHub App's broad `contents: write` scope is enforced by `select_files_to_commit` (script correctness), per the PRD's explicit acknowledgement that GitHub does not support file-path-scoped tokens. - `release` runs are filtered to tags ending in `-stable` at the workflow level, so a `v1.84.0-rc1` release does not republish the matrix. - Resolver and publisher live under `tests/claude_code/` alongside `matrix_builder.py` and `cli_driver.py` — production code that supports the test suite, kept colocated with it to match the slice 1+2 layout. Out of scope / blockers for next iteration: - Provisioning the GitHub App itself (creating it under BerriAI's org, installing it on litellm-docs only, generating the private key and registering `COMPAT_MATRIX_APP_ID` / `COMPAT_MATRIX_APP_PRIVATE_KEY` as repo secrets) is an operator/infra step that cannot land via a code change in this repo. - The first successful cron run is what removes the hand-authored `compatibility-matrix.json` from the docs repo and replaces it with generated output — that happens after this PR merges and the App is installed; not a code change here. Tests: 34 -> 45 passing (added 7 resolver tests + 7 publisher helper tests, all unit-only and offline). The 12 per-cell failures under `tests/claude_code/basic_messaging_non_streaming/` remain by design — they require a running proxy which the cron VM provides. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-06 23:27:14 +00:00