mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-16 23:41:43 +00:00
ff57d5b546
Follow-up to PR #28117. Three behavior changes + one new workflow, addressing the team's concerns on the original review: 1) Apply auto-close to ALL open PRs, not just those over a week old. - close_low_quality_prs.py: --min-age-days default flipped from 7 to 0. The flag is preserved as an opt-in safety net for one-off backfill runs that want to spare very-young PRs, but the daily scheduled sweep now closes external-author PRs as soon as Greptile scores them <4/5. - close_low_quality_prs.yml: workflow_dispatch input default also flipped to 0; doc comments updated. 2) Apply auto-close to draft PRs too. - close_low_quality_prs.py: removed the skip-draft branch in evaluate_pr. Drafts are NOT a free pass — the team's intent is 'open PR count == PRs internal collaborators need to action on', so a draft Greptile scored 2/5 still belongs in the closed bucket. Authors who genuinely need a long-lived draft can attach the 'wip' opt-out label, which is unchanged. - The 'skip-draft' action is gone; the 'wip' label still skips. 3) Address the 'OSS contributors cannot reopen a bot-closed PR' wrinkle. GitHub does NOT let an external (non-write-access) contributor reopen a PR that was closed by a bot or maintainer (long-standing limitation). The original PR's close-comments told contributors to 'Reopen the PR — I'll re-evaluate automatically', which is broken for the very audience this triage targets. Two changes: a) Reword every close-comment (Greptile sweep + Agent Shin PR close + Agent Shin issue close + PR template) to recommend: - Open a new PR with the updated branch (primary path). - Or comment '@agent-shin reconsider' on the closed PR for a re-evaluation that, on pass, reopens the PR via the bot's GH_TOKEN write access. b) Add the @agent-shin reconsider workflow: - .github/workflows/triage_reconsider.yml: new 'issue_comment'-triggered workflow. Authorizes only the PR/issue author or an internal collaborator (OWNER/MEMBER/COLLABORATOR), gated via a step output so unauthorized commenters never reach the destructive steps. Globally gated on AGENT_SHIN_ENABLED='true' (positive form, matching the test_github_triage_workflows guardrail patterns). - triage_with_llm.py: --reconsider mode. On a closed PR/issue, re-runs the LLM judge (or linked-issue regex short-circuit) and: - on pass: reopens via reopen_pr/reopen_issue + posts a 'Re-evaluated and reopened' comment. - on fail: leaves closed and posts a 'still missing X' comment so the contributor can iterate again. Reconsider-on-open is a no-op ('skip-not-closed'). Internal-author + bot-account skips still take priority over reconsider. 4) Greptile-on-closed-PRs question: the team asked whether Greptile can re-review a closed PR. Greptile's docs don't address this and we shouldn't promise behavior we can't verify, so the new close-comment wording does NOT instruct contributors to 're-request greptile on the closed PR'. Instead it points them at the new-PR path (which Greptile definitely reviews) or the @agent-shin reconsider trigger (which re-runs the LiteLLM-side rubric judge, not Greptile). Tests: 93 passing (was 59). - test_github_close_low_quality_prs.py: replaced 'skip drafts' test with 'closes drafts when score is low' + 'closes brand-new PR when min_age=0' + 'no skip when min_age=0'. The 'skip too young' assertion is preserved as opt-in. - test_github_triage_with_llm.py: 6 new TestTriageOrchestration cases for reconsider mode (skip-not-closed on open, reopen on pass, still-failing comment on fail, linked-issue short-circuit reopen, skip internal author in reconsider, reopen-issue on pass) + a new TestCloseCommentText class that pins the user-facing 'open a new PR' + '@agent-shin reconsider' wording. - test_github_triage_workflows.py: added triage_reconsider.yml to the destructive-gate guardrail table; AGENT_SHIN_ENABLED is its own destructive gate (no separate per-run flag needed). Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
107 lines
4.1 KiB
Markdown
107 lines
4.1 KiB
Markdown
<!--
|
|
👋 Hi there — please read before submitting.
|
|
|
|
To keep the review queue healthy for everyone, **every external PR is
|
|
auto-triaged** by an LLM bot ("Agent Shin") on open / reopen, regardless of
|
|
whether the PR is a draft or marked ready for review. PRs that don't meet
|
|
the rubric below are auto-closed with an explanation.
|
|
|
|
To pass triage, your PR must satisfy AT LEAST ONE of:
|
|
|
|
(A) Link a related GitHub issue (e.g. "Fixes #1234" or "Resolves
|
|
https://github.com/BerriAI/litellm/issues/1234"), OR
|
|
|
|
(B) Provide ALL of the following IN THIS PR DESCRIPTION:
|
|
- A clear problem description (what bug or missing feature this addresses)
|
|
- Expected vs. actual behavior
|
|
- Visual QA proof (before/after screenshots, screen recording, or
|
|
terminal output demonstrating that the fix/feature works end-to-end)
|
|
|
|
Every external PR (including drafts, regardless of age) also receives a
|
|
Greptile code review. Any PR with a Greptile Confidence Score below 4/5 is
|
|
auto-closed.
|
|
|
|
If your PR was auto-closed and you've addressed the feedback, you have two
|
|
options to bring it back:
|
|
|
|
- **Open a new PR** with the updated branch (recommended — GitHub does not
|
|
let external contributors reopen a PR that was closed by a bot or
|
|
maintainer).
|
|
- **Or** comment `@agent-shin reconsider` on the closed PR. Agent Shin
|
|
will re-run triage and reopen the PR if it now meets the bar.
|
|
|
|
Internal BerriAI contributors are exempt from this auto-triage — fill in the
|
|
Linear ticket section instead.
|
|
-->
|
|
|
|
## Relevant issues
|
|
|
|
<!-- e.g. "Fixes #000". If you have no related issue, fill in the
|
|
"Problem description / Expected vs. Actual / QA proof" sections below. -->
|
|
|
|
## Linear ticket
|
|
|
|
<!-- INTERNAL CONTRIBUTORS ONLY: add the Linear ticket e.g. "Resolves LIT-1234"
|
|
to magically link the Linear ticket to the GitHub PR. External contributors:
|
|
leave this blank and fill in the problem/expected-actual/QA sections below. -->
|
|
|
|
## Problem description
|
|
|
|
<!-- What bug or missing feature does this PR address? One or two paragraphs.
|
|
External contributors: required unless you linked a GitHub issue above. -->
|
|
|
|
## Expected vs. actual behavior
|
|
|
|
<!-- What did you expect to happen? What is happening today (before this PR)?
|
|
External contributors: required unless you linked a GitHub issue above. -->
|
|
|
|
## QA proof
|
|
|
|
<!-- Required for external contributors: include before/after screenshots,
|
|
a screen recording, or terminal/log output that demonstrates the fix or feature
|
|
works end-to-end. For UI changes, before/after screenshots are mandatory.
|
|
For backend changes, terminal output of a passing test or curl command is fine. -->
|
|
|
|
## Pre-Submission checklist
|
|
|
|
**Please complete all items before asking a LiteLLM maintainer to review your PR**
|
|
|
|
- [ ] I have Added testing in the [`tests/test_litellm/`](https://github.com/BerriAI/litellm/tree/main/tests/test_litellm) directory, **Adding at least 1 test is a hard requirement** - [see details](https://docs.litellm.ai/docs/extras/contributing_code)
|
|
- [ ] My PR passes all unit tests on [`make test-unit`](https://docs.litellm.ai/docs/extras/contributing_code)
|
|
- [ ] My PR's scope is as isolated as possible, it only solves 1 specific problem
|
|
- [ ] I have received a Greptile **Confidence Score of at least 4/5** before requesting a maintainer review (Greptile reviews automatically on open; comment `@greptileai` to re-trigger after pushing fixes)
|
|
|
|
## Delays in PR merge?
|
|
|
|
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on [Slack (#pr-review)](https://join.slack.com/t/litellmossslack/shared_invite/zt-3o7nkuyfr-p_kbNJj8taRfXGgQI1~YyA).
|
|
|
|
## CI (LiteLLM team)
|
|
|
|
> **CI status guideline:**
|
|
>
|
|
> - 50-55 passing tests: main is stable with minor issues.
|
|
> - 45-49 passing tests: acceptable but needs attention
|
|
> - <= 40 passing tests: unstable; be careful with your merges and assess the risk.
|
|
|
|
- [ ] **Branch creation CI run**
|
|
Link:
|
|
|
|
- [ ] **CI run for the last commit**
|
|
Link:
|
|
|
|
- [ ] **Merge / cherry-pick CI run**
|
|
Links:
|
|
|
|
## Type
|
|
|
|
<!-- Select the type of Pull Request -->
|
|
<!-- Keep only the necessary ones -->
|
|
|
|
🆕 New Feature
|
|
🐛 Bug Fix
|
|
🧹 Refactoring
|
|
📖 Documentation
|
|
🚄 Infrastructure
|
|
✅ Test
|
|
|
|
## Changes
|