mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-23 00:41:40 +00:00
797a449daf
Slice 4 of the Claude Code Compatibility Matrix: stand up the daily-cron pipeline that publishes `compatibility-matrix.json` to the docs repo. After this slice lands, the hand-authored matrix in the docs repo is replaced by auto-generated output, and the docs page begins reflecting real test runs against the latest stable LiteLLM release. What landed: - tests/claude_code/resolver.py Latest Stable LiteLLM Resolver. Calls the GitHub Releases API and returns the newest tag matching `v*-stable`. Sort is numeric on (major, minor, patch) so v1.10.0-stable correctly outranks v1.9.5-stable. Injectable `http_get` so tests run offline. - tests/claude_code/publisher.py Daily-cron orchestrator. Resolves the latest stable tag, pulls `ghcr.io/berriai/litellm:<tag>`, starts it as the proxy, installs `@anthropic-ai/claude-code@latest`, runs `pytest tests/claude_code/`, invokes the Matrix JSON Builder, and direct-pushes `compatibility-matrix.json` to the docs repo's main branch using a GitHub App installation token (`DOCS_REPO_TOKEN`). Idempotent: a no-op if the JSON is byte-identical to what's already on main. - tests/claude_code/_publisher_unit_tests/test_resolver.py test_publisher.py 14 unit tests covering the small pure helpers — version sort, non-stable filtering, http-get injection, commit message determinism, Docker image-name builder, and the file allowlist that enforces the "only `compatibility-matrix.json` ever ships" guarantee. Per the PRD's "Testing Decisions" section, the publisher's full subprocess orchestration intentionally ships without a unit-test harness; the daily-cron failure surface is itself the test. - .github/workflows/claude_code_compat_matrix.yml GitHub Actions workflow with three triggers (daily cron at 06:00 UTC, `release: published` filtered to `*-stable` tags, and `workflow_dispatch`). Mints a docs-repo installation token from a GitHub App scoped to `BerriAI/litellm-docs` only with `contents: write`, then runs the publisher. - .gitignore Add `compatibility-matrix.json` (cron VM output). Key decisions: - "Isolated VM" is realized as a GitHub-hosted ubuntu-latest runner — every run gets a fresh ephemeral VM, and the always-latest Claude Code CLI is only ever installed inside that ephemeral environment, so a malicious or broken Claude Code release cannot affect the trusted PR-gate CI in CircleCI. - File-level restriction on the GitHub App's broad `contents: write` scope is enforced by `select_files_to_commit` (script correctness), per the PRD's explicit acknowledgement that GitHub does not support file-path-scoped tokens. - `release` runs are filtered to tags ending in `-stable` at the workflow level, so a `v1.84.0-rc1` release does not republish the matrix. - Resolver and publisher live under `tests/claude_code/` alongside `matrix_builder.py` and `cli_driver.py` — production code that supports the test suite, kept colocated with it to match the slice 1+2 layout. Out of scope / blockers for next iteration: - Provisioning the GitHub App itself (creating it under BerriAI's org, installing it on litellm-docs only, generating the private key and registering `COMPAT_MATRIX_APP_ID` / `COMPAT_MATRIX_APP_PRIVATE_KEY` as repo secrets) is an operator/infra step that cannot land via a code change in this repo. - The first successful cron run is what removes the hand-authored `compatibility-matrix.json` from the docs repo and replaces it with generated output — that happens after this PR merges and the App is installed; not a code change here. Tests: 34 -> 45 passing (added 7 resolver tests + 7 publisher helper tests, all unit-only and offline). The 12 per-cell failures under `tests/claude_code/basic_messaging_non_streaming/` remain by design — they require a running proxy which the cron VM provides. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
108 lines
2.7 KiB
Text
108 lines
2.7 KiB
Text
.python-version
|
|
.venv
|
|
.venv_policy_test
|
|
.env
|
|
.claude
|
|
.newenv
|
|
newenv/*
|
|
litellm/proxy/myenv/*
|
|
litellm_uuid.txt
|
|
__pycache__/
|
|
*.pyc
|
|
bun.lockb
|
|
**/.DS_Store
|
|
.aider*
|
|
litellm_results.jsonl
|
|
secrets.toml
|
|
.gitignore
|
|
litellm/proxy/litellm_secrets.toml
|
|
litellm/proxy/api_log.json
|
|
.idea/
|
|
router_config.yaml
|
|
litellm_server/config.yaml
|
|
litellm/proxy/_secret_config.yaml
|
|
.aws-sam/
|
|
litellm/tests/aiologs.log
|
|
litellm/tests/exception_data.txt
|
|
litellm/tests/config_*.yaml
|
|
litellm/tests/langfuse.log
|
|
langfuse.log
|
|
.langfuse.log
|
|
litellm/tests/test_custom_logger.py
|
|
litellm/tests/langfuse.log
|
|
litellm/tests/dynamo*.log
|
|
.vscode/settings.json
|
|
litellm/proxy/log.txt
|
|
proxy_server_config_@.yaml
|
|
.gitignore
|
|
proxy_server_config_2.yaml
|
|
litellm/proxy/secret_managers/credentials.json
|
|
hosted_config.yaml
|
|
litellm/proxy/tests/node_modules
|
|
litellm/proxy/tests/package.json
|
|
litellm/proxy/tests/package-lock.json
|
|
ui/litellm-dashboard/.next
|
|
ui/litellm-dashboard/node_modules
|
|
ui/litellm-dashboard/next-env.d.ts
|
|
ui/litellm-dashboard/package.json
|
|
ui/litellm-dashboard/package-lock.json
|
|
deploy/charts/litellm/*.tgz
|
|
deploy/charts/litellm/charts/*
|
|
deploy/charts/*.tgz
|
|
litellm/proxy/vertex_key.json
|
|
**/.vim/
|
|
**/node_modules
|
|
kub.yaml
|
|
loadtest_kub.yaml
|
|
litellm/proxy/_new_secret_config.yaml
|
|
litellm/proxy/_new_secret_config.yaml
|
|
litellm/proxy/_super_secret_config.yaml
|
|
litellm/proxy/_super_secret_config.yaml
|
|
litellm/proxy/myenv/bin/activate
|
|
litellm/proxy/myenv/bin/Activate.ps1
|
|
myenv/*
|
|
litellm/tests/log.txt
|
|
litellm/tests/langfuse.log
|
|
litellm/tests/langfuse.log
|
|
litellm/proxy/google-cloud-sdk/*
|
|
tests/llm_translation/log.txt
|
|
venv/
|
|
tests/local_testing/log.txt
|
|
|
|
.codegpt
|
|
litellm/proxy/_new_new_secret_config.yaml
|
|
litellm/proxy/custom_guardrail.py
|
|
**/.mypy_cache/
|
|
litellm/proxy/application.log
|
|
tests/llm_translation/vertex_test_account.json
|
|
tests/llm_translation/test_vertex_key.json
|
|
litellm/proxy/migrations/0_init/migration.sql
|
|
litellm/proxy/db/migrations/0_init/migration.sql
|
|
litellm/proxy/db/migrations/*
|
|
litellm/proxy/migrations/*config.yaml
|
|
litellm/proxy/migrations/*
|
|
litellm/proxy/to_delete_loadtest_work/*
|
|
config.yaml
|
|
tests/litellm/litellm_core_utils/llm_cost_calc/log.txt
|
|
tests/test_custom_dir/*
|
|
test.py
|
|
|
|
litellm_config.yaml
|
|
!.github/observatory/litellm_config.yaml
|
|
.cursor
|
|
litellm/proxy/to_delete_loadtest_work/*
|
|
update_model_cost_map.py
|
|
tests/test_litellm/proxy/_experimental/mcp_server/test_mcp_server_manager.py
|
|
scripts/test_vertex_ai_search.py
|
|
LAZY_LOADING_IMPROVEMENTS.md
|
|
STABILIZATION_TODO.md
|
|
**/test-results
|
|
**/playwright-report
|
|
**/*.storageState.json
|
|
**/coverage
|
|
test-config
|
|
|
|
# Claude Code compatibility-matrix pytest artifact (CI-only output).
|
|
compat-results.json
|
|
# Matrix JSON produced by the daily-cron publisher (pushed to litellm-docs).
|
|
compatibility-matrix.json
|