mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-22 00:31:44 +00:00
bb60422e7d
The previous iteration of this PR ported the populator to a Python
module (`publisher.py`) with 8 unit-tested pure helpers for branch
naming and PR-body rendering. After the docker code came out, the
GitHub App auth came out, and the worktree-vs-tempdir decision was
made, what was left was: 'git fetch + git checkout + uv sync + start
a subprocess + run pytest + clone docs + git commit + gh pr create'.
That's a bash script.
This commit replaces 800 lines of Python (publisher.py + resolver.py +
their unit tests) with a 297-line run_daily.sh and a 48-line
build_matrix.py whose only job is to be importable Python that can
call into the matrix_builder we already have. Net deletion: -822 lines.
Removed
-------
* tests/claude_code/publisher.py — the full Python orchestrator.
Every code path it had is now in run_daily.sh.
* tests/claude_code/resolver.py — the GitHub Releases v*-stable
resolver. Replaced by ~10 lines of jq inside run_daily.sh.
* tests/claude_code/_publisher_unit_tests/ — both test files. The
pure helpers they covered (commit message, file allowlist, branch
name, PR title/body) only existed because publisher.py was Python.
The bash equivalents are short heredoc strings.
Added
-----
* tests/claude_code/cron_vm/run_daily.sh — the actual cron job,
structured as numbered phases (resolve / worktree / proxy /
pytest / build / publish) so journalctl output is readable.
* tests/claude_code/cron_vm/build_matrix.py — a 48-line CLI that
calls the existing matrix_builder.build_from_paths. Kept in
Python because the builder itself is Python and well-tested.
Modified
--------
* tests/claude_code/cron_vm/litellm-compat-matrix.service —
ExecStart now invokes run_daily.sh instead of
'python -m tests.claude_code.publisher'.
* tests/claude_code/cron_vm/README.md — updated layout table,
file roles, and operating commands to match.
Why this is the right shape
---------------------------
* The failure mode at 06:00 UTC is 'read journalctl, see the literal
failing command with its + prefix, copy-paste it into a shell to
reproduce'. Bash makes that immediate; Python's subprocess.run
output looks similar but the surrounding orchestration is harder
to step through interactively.
* Every operation the script does is already a shell command (git,
uv, gh, jq, curl, pytest). The Python wrapper was translating
between argv arrays and back.
* The two pieces that genuinely benefit from being in a typed
language are matrix_builder (already Python) and the resolver's
semver sort (now done in jq, with the version_key tuple sort
inline). 'Already Python' wins, 'tiny jq pipeline' wins.
What's preserved
----------------
* Idempotency: same (litellm, claude, UTC date) -> same branch ->
same PR. force-with-lease push, gh-pr-create no-op-on-exists.
* Byte-identical-JSON early return (git diff --cached --quiet).
* Per-feature status table in the PR body (jq pipeline mirroring
the Python pr_body_for_matrix logic).
* Persistent worktree approach so disk doesn't grow unboundedly.
* Proxy bound to :4100 to avoid colliding with a developer's :4000.
* SKIP_PUBLISH=1 and PYTEST_K=... operator escape hatches.
|
||
|---|---|---|
| .. | ||
| agent_tests | ||
| audio_tests | ||
| basic_proxy_startup_tests | ||
| batches_tests | ||
| benchmarks | ||
| claude_code | ||
| code_coverage_tests | ||
| documentation_tests | ||
| enterprise | ||
| guardrails_tests | ||
| image_gen_tests | ||
| litellm | ||
| litellm-proxy-extras | ||
| litellm_core_utils | ||
| litellm_utils_tests | ||
| llm_responses_api_testing | ||
| llm_translation | ||
| load_tests | ||
| local_testing | ||
| logging_callback_tests | ||
| mcp_tests | ||
| multi_instance_e2e_tests | ||
| ocr_tests | ||
| old_proxy_tests/tests | ||
| openai_endpoints_tests | ||
| otel_tests | ||
| pass_through_tests | ||
| pass_through_unit_tests | ||
| proxy_admin_ui_tests | ||
| proxy_e2e_anthropic_messages_tests | ||
| proxy_security_tests | ||
| proxy_unit_tests | ||
| router_unit_tests | ||
| scim_tests | ||
| search_tests | ||
| spend_tracking_tests | ||
| store_model_in_db_tests | ||
| test_litellm | ||
| unified_google_tests | ||
| vector_store_tests | ||
| windows_tests | ||
| __init__.py | ||
| _flush_vcr_cache.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| eval_swe_bench.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| README.MD | ||
| test_budget_management.py | ||
| test_callbacks_on_proxy.py | ||
| test_config.py | ||
| test_debug_warning.py | ||
| test_default_encoding_non_root.py | ||
| test_end_users.py | ||
| test_entrypoint.py | ||
| test_fallbacks.py | ||
| test_gpt5_azure_temperature_support.py | ||
| test_health.py | ||
| test_keys.py | ||
| test_litellm_proxy_responses_config.py | ||
| test_logging.conf | ||
| test_models.py | ||
| test_new_vector_store_endpoints.py | ||
| test_openai_endpoints.py | ||
| test_organizations.py | ||
| test_otel_thread_leak.py | ||
| test_passthrough_endpoints.py | ||
| test_presidio_latency.py | ||
| test_proxy_server_non_root.py | ||
| test_ratelimit.py | ||
| test_resource_cleanup.py | ||
| test_service_logger_otel.py | ||
| test_spend_logs.py | ||
| test_team.py | ||
| test_team_logging.py | ||
| test_team_members.py | ||
| test_users.py | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/test_litellm
This folder can only run mock tests.