litellm/.github/pull_request_template.md at ff57d5b5466ea8d52621a57fdc4631a469168c6b

mirror of https://github.com/BerriAI/litellm.git synced 2026-09-14 23:21:35 +00:00

feat(triage): close any PR (incl. drafts, any age); add @agent-shin reconsider flow

Follow-up to PR #28117. Three behavior changes + one new workflow,
addressing the team's concerns on the original review:

1) Apply auto-close to ALL open PRs, not just those over a week old.

   - close_low_quality_prs.py: --min-age-days default flipped from 7 to
     0. The flag is preserved as an opt-in safety net for one-off
     backfill runs that want to spare very-young PRs, but the daily
     scheduled sweep now closes external-author PRs as soon as Greptile
     scores them <4/5.
   - close_low_quality_prs.yml: workflow_dispatch input default also
     flipped to 0; doc comments updated.

2) Apply auto-close to draft PRs too.

   - close_low_quality_prs.py: removed the skip-draft branch in
     evaluate_pr. Drafts are NOT a free pass — the team's intent is
     'open PR count == PRs internal collaborators need to action on',
     so a draft Greptile scored 2/5 still belongs in the closed bucket.
     Authors who genuinely need a long-lived draft can attach the 'wip'
     opt-out label, which is unchanged.
   - The 'skip-draft' action is gone; the 'wip' label still skips.

3) Address the 'OSS contributors cannot reopen a bot-closed PR' wrinkle.

   GitHub does NOT let an external (non-write-access) contributor
   reopen a PR that was closed by a bot or maintainer (long-standing
   limitation). The original PR's close-comments told contributors to
   'Reopen the PR — I'll re-evaluate automatically', which is broken
   for the very audience this triage targets. Two changes:

   a) Reword every close-comment (Greptile sweep + Agent Shin PR
      close + Agent Shin issue close + PR template) to recommend:
        - Open a new PR with the updated branch (primary path).
        - Or comment '@agent-shin reconsider' on the closed PR for a
          re-evaluation that, on pass, reopens the PR via the bot's
          GH_TOKEN write access.

   b) Add the @agent-shin reconsider workflow:
        - .github/workflows/triage_reconsider.yml: new
          'issue_comment'-triggered workflow. Authorizes only the
          PR/issue author or an internal collaborator
          (OWNER/MEMBER/COLLABORATOR), gated via a step output so
          unauthorized commenters never reach the destructive steps.
          Globally gated on AGENT_SHIN_ENABLED='true' (positive form,
          matching the test_github_triage_workflows guardrail
          patterns).
        - triage_with_llm.py: --reconsider mode. On a closed PR/issue,
          re-runs the LLM judge (or linked-issue regex short-circuit)
          and:
            - on pass: reopens via reopen_pr/reopen_issue + posts a
              'Re-evaluated and reopened' comment.
            - on fail: leaves closed and posts a 'still missing X'
              comment so the contributor can iterate again.
          Reconsider-on-open is a no-op ('skip-not-closed').
          Internal-author + bot-account skips still take priority over
          reconsider.

4) Greptile-on-closed-PRs question: the team asked whether Greptile can
   re-review a closed PR. Greptile's docs don't address this and we
   shouldn't promise behavior we can't verify, so the new close-comment
   wording does NOT instruct contributors to 're-request greptile on
   the closed PR'. Instead it points them at the new-PR path (which
   Greptile definitely reviews) or the @agent-shin reconsider trigger
   (which re-runs the LiteLLM-side rubric judge, not Greptile).

Tests: 93 passing (was 59).

  - test_github_close_low_quality_prs.py: replaced 'skip drafts' test
    with 'closes drafts when score is low' + 'closes brand-new PR when
    min_age=0' + 'no skip when min_age=0'. The 'skip too young'
    assertion is preserved as opt-in.
  - test_github_triage_with_llm.py: 6 new TestTriageOrchestration cases
    for reconsider mode (skip-not-closed on open, reopen on pass,
    still-failing comment on fail, linked-issue short-circuit reopen,
    skip internal author in reconsider, reopen-issue on pass) + a new
    TestCloseCommentText class that pins the user-facing 'open a new
    PR' + '@agent-shin reconsider' wording.
  - test_github_triage_workflows.py: added triage_reconsider.yml to
    the destructive-gate guardrail table; AGENT_SHIN_ENABLED is its
    own destructive gate (no separate per-run flag needed).

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

2026-05-18 00:15:42 +00:00

4.1 KiB

Raw Blame History

Relevant issues

Linear ticket

Problem description

Expected vs. actual behavior

QA proof

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
My PR passes all unit tests on make test-unit
My PR's scope is as isolated as possible, it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically on open; comment @greptileai to re-trigger after pushing fixes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

50-55 passing tests: main is stable with minor issues.

45-49 passing tests: acceptable but needs attention

<= 40 passing tests: unstable; be careful with your merges and assess the risk.

Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:

Type

🆕 New Feature 🐛 Bug Fix 🧹 Refactoring 📖 Documentation 🚄 Infrastructure ✅ Test

4.1 KiB Raw Blame History