mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-14 23:21:35 +00:00
4e0968ad8d
Address three Greptile/veria-ai concerns on the @agent-shin reconsider flow: 1. **Reconsider had no dry-run path.** The previous reconsider mode ignored `--close` and always posted comments + reopened on a pass. A local operator running `python triage_with_llm.py --reconsider --pr N` would silently take destructive GitHub actions with no way to preview. Reconsider now honors `close=False` the same way regular triage does and returns `would-reopen` / `would-reconsider-still-failing` for step-summary rendering. 2. **Reconsider could reopen maintainer-closed PRs/issues** (Medium security finding from veria-ai). The workflow only checked that the commenter was authorized — it did NOT check that the most recent close was performed by Agent Shin. A contributor could comment `@agent-shin reconsider` on a PR a maintainer closed for non-rubric reasons (duplicate, security report, design rejection) and have the bot reopen it. Add `was_closed_by_agent_shin()` which inspects the issue events API for the most recent `closed` actor and only permits reopen when that actor matches the configured bot login (default `github-actions[bot]`, overridable via env). Fail-closed on missing events. 3. **No rate-limiting on the reconsider trigger.** Every `@agent-shin reconsider` comment burns CI minutes + an OpenAI API call. Add a 10-minute cooldown via `seconds_since_last_reconsider_verdict()` which greps the issue's comment list for the bot's own verdict marker (`<!-- agent-shin:reconsider-verdict -->`). Inside the window the triage returns `skip-rate-limited` and the LLM never runs. Workflow update: - `triage_reconsider.yml` now passes `--close` only when `AGENT_SHIN_ENABLED=true`, matching the pattern of `triage_pr_with_llm.yml`. The script runs in both states so the verdict still appears in the step summary for QA. Tests: - Add 5 reconsider safety tests: dry-run for pass / fail / linked-issue short-circuit, bot-closed-guard refusal on maintainer close, rate-limit refusal inside the cooldown window, and cooldown-elapsed acceptance. - Add unit tests for `was_closed_by_agent_shin` (bot / maintainer / missing actor / env-override) and `seconds_since_last_reconsider_verdict` (no marker / multiple markers / non-bot comment with marker / bot comment without marker). - Pin the `<!-- agent-shin:reconsider-verdict -->` marker in both reopen and still-failing comments — dropping it would silently break the cooldown. Existing reconsider tests updated to pass `close=True` (the production path now) + stub the new guards via `_stub_reconsider_guards`. 112 tests pass (was 93). Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| _test-unit-base.yml | ||
| _test-unit-services-base.yml | ||
| auto_update_price_and_context_window.yml | ||
| auto_update_price_and_context_window_file.py | ||
| check-schema-sync.yml | ||
| check_duplicate_issues.yml | ||
| close_low_quality_prs.yml | ||
| codeql.yml | ||
| codspeed.yml | ||
| create-release-branch.yml | ||
| create-release.yml | ||
| create_daily_staging_branch.yml | ||
| guard-fork-dependencies.yml | ||
| guard-main-branch.yml | ||
| helm_unit_test.yml | ||
| issue-keyword-labeler.yml | ||
| label-component.yml | ||
| mutation-test.yml | ||
| run_llm_translation_tests.py | ||
| scorecard.yml | ||
| stale.yml | ||
| sync-schema.yml | ||
| test-code-quality.yml | ||
| test-linting.yml | ||
| test-litellm-ui-build.yml | ||
| test-mcp.yml | ||
| test-model-map.yaml | ||
| test-semgrep.yml | ||
| test-unit-core-utils.yml | ||
| test-unit-documentation.yml | ||
| test-unit-enterprise-routing.yml | ||
| test-unit-integrations.yml | ||
| test-unit-llm-providers.yml | ||
| test-unit-misc.yml | ||
| test-unit-proxy-auth.yml | ||
| test-unit-proxy-db.yml | ||
| test-unit-proxy-endpoints.yml | ||
| test-unit-proxy-infra.yml | ||
| test-unit-proxy-legacy.yml | ||
| test-unit-responses-caching-types.yml | ||
| test-unit-security.yml | ||
| test_server_root_path.yml | ||
| triage_issue_with_llm.yml | ||
| triage_pr_with_llm.yml | ||
| triage_reconsider.yml | ||
| zizmor.yml | ||