scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-13 23:11:40 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 101

feat(triage): close any PR (incl. drafts, any age); add @agent-shin reconsider flow

Browse source 
Follow-up to PR #28117. Three behavior changes + one new workflow,
addressing the team's concerns on the original review:

1) Apply auto-close to ALL open PRs, not just those over a week old.

   - close_low_quality_prs.py: --min-age-days default flipped from 7 to
     0. The flag is preserved as an opt-in safety net for one-off
     backfill runs that want to spare very-young PRs, but the daily
     scheduled sweep now closes external-author PRs as soon as Greptile
     scores them <4/5.
   - close_low_quality_prs.yml: workflow_dispatch input default also
     flipped to 0; doc comments updated.

2) Apply auto-close to draft PRs too.

   - close_low_quality_prs.py: removed the skip-draft branch in
     evaluate_pr. Drafts are NOT a free pass — the team's intent is
     'open PR count == PRs internal collaborators need to action on',
     so a draft Greptile scored 2/5 still belongs in the closed bucket.
     Authors who genuinely need a long-lived draft can attach the 'wip'
     opt-out label, which is unchanged.
   - The 'skip-draft' action is gone; the 'wip' label still skips.

3) Address the 'OSS contributors cannot reopen a bot-closed PR' wrinkle.

   GitHub does NOT let an external (non-write-access) contributor
   reopen a PR that was closed by a bot or maintainer (long-standing
   limitation). The original PR's close-comments told contributors to
   'Reopen the PR — I'll re-evaluate automatically', which is broken
   for the very audience this triage targets. Two changes:

   a) Reword every close-comment (Greptile sweep + Agent Shin PR
      close + Agent Shin issue close + PR template) to recommend:
        - Open a new PR with the updated branch (primary path).
        - Or comment '@agent-shin reconsider' on the closed PR for a
          re-evaluation that, on pass, reopens the PR via the bot's
          GH_TOKEN write access.

   b) Add the @agent-shin reconsider workflow:
        - .github/workflows/triage_reconsider.yml: new
          'issue_comment'-triggered workflow. Authorizes only the
          PR/issue author or an internal collaborator
          (OWNER/MEMBER/COLLABORATOR), gated via a step output so
          unauthorized commenters never reach the destructive steps.
          Globally gated on AGENT_SHIN_ENABLED='true' (positive form,
          matching the test_github_triage_workflows guardrail
          patterns).
        - triage_with_llm.py: --reconsider mode. On a closed PR/issue,
          re-runs the LLM judge (or linked-issue regex short-circuit)
          and:
            - on pass: reopens via reopen_pr/reopen_issue + posts a
              'Re-evaluated and reopened' comment.
            - on fail: leaves closed and posts a 'still missing X'
              comment so the contributor can iterate again.
          Reconsider-on-open is a no-op ('skip-not-closed').
          Internal-author + bot-account skips still take priority over
          reconsider.

4) Greptile-on-closed-PRs question: the team asked whether Greptile can
   re-review a closed PR. Greptile's docs don't address this and we
   shouldn't promise behavior we can't verify, so the new close-comment
   wording does NOT instruct contributors to 're-request greptile on
   the closed PR'. Instead it points them at the new-PR path (which
   Greptile definitely reviews) or the @agent-shin reconsider trigger
   (which re-runs the LiteLLM-side rubric judge, not Greptile).

Tests: 93 passing (was 59).

  - test_github_close_low_quality_prs.py: replaced 'skip drafts' test
    with 'closes drafts when score is low' + 'closes brand-new PR when
    min_age=0' + 'no skip when min_age=0'. The 'skip too young'
    assertion is preserved as opt-in.
  - test_github_triage_with_llm.py: 6 new TestTriageOrchestration cases
    for reconsider mode (skip-not-closed on open, reopen on pass,
    still-failing comment on fail, linked-issue short-circuit reopen,
    skip internal author in reconsider, reopen-issue on pass) + a new
    TestCloseCommentText class that pins the user-facing 'open a new
    PR' + '@agent-shin reconsider' wording.
  - test_github_triage_workflows.py: added triage_reconsider.yml to
    the destructive-gate guardrail table; AGENT_SHIN_ENABLED is its
    own destructive gate (no separate per-run flag needed).

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
This commit is contained in:
Cursor Agent 2026-05-18 00:15:42 +00:00
parent 638dba9d14
commit ff57d5b546
No known key found for this signature in database
8 changed files with 658 additions and 61 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

25
.github/pull_request_template.md vendored
View file

@ -1,10 +1,10 @@
<!--
👋 Hi there — please read before submitting.
To keep the review queue healthy for everyone, **external contributions are
auto-triaged** by an LLM bot ("Agent Shin") on open / reopen. Pull requests that
do not meet the rubric below are auto-closed with an explanation, and you can
update + reopen at any time to be re-evaluated.
To keep the review queue healthy for everyone, **every external PR is
auto-triaged** by an LLM bot ("Agent Shin") on open / reopen, regardless of
whether the PR is a draft or marked ready for review. PRs that don't meet
the rubric below are auto-closed with an explanation.
To pass triage, your PR must satisfy AT LEAST ONE of:
@ -17,9 +17,18 @@ To pass triage, your PR must satisfy AT LEAST ONE of:
- Visual QA proof (before/after screenshots, screen recording, or
terminal output demonstrating that the fix/feature works end-to-end)
PRs also receive a Greptile code review. PRs open for ≥7 days with a Greptile
Confidence Score below 4/5 are auto-closed; re-request a review from
@greptileai once you've addressed the feedback and reopen to be re-evaluated.
Every external PR (including drafts, regardless of age) also receives a
Greptile code review. Any PR with a Greptile Confidence Score below 4/5 is
auto-closed.
If your PR was auto-closed and you've addressed the feedback, you have two
options to bring it back:
- **Open a new PR** with the updated branch (recommended — GitHub does not
let external contributors reopen a PR that was closed by a bot or
maintainer).
- **Or** comment `@agent-shin reconsider` on the closed PR. Agent Shin
will re-run triage and reopen the PR if it now meets the bar.
Internal BerriAI contributors are exempt from this auto-triage — fill in the
Linear ticket section instead.
@ -60,7 +69,7 @@ For backend changes, terminal output of a passing test or curl command is fine.
- [ ] I have Added testing in the [`tests/test_litellm/`](https://github.com/BerriAI/litellm/tree/main/tests/test_litellm) directory, **Adding at least 1 test is a hard requirement** - [see details](https://docs.litellm.ai/docs/extras/contributing_code)
- [ ] My PR passes all unit tests on [`make test-unit`](https://docs.litellm.ai/docs/extras/contributing_code)
- [ ] My PR's scope is as isolated as possible, it only solves 1 specific problem
- [ ] I have requested a Greptile review by commenting `@greptileai` and received a **Confidence Score of at least 4/5** before requesting a maintainer review
- [ ] I have received a Greptile **Confidence Score of at least 4/5** before requesting a maintainer review (Greptile reviews automatically on open; comment `@greptileai` to re-trigger after pushing fixes)
## Delays in PR merge?

78
.github/scripts/close_low_quality_prs.py vendored
View file

@ -1,18 +1,24 @@
#!/usr/bin/env python3
"""
Auto-close stale, low-quality pull requests.
Auto-close low-quality pull requests.
Closes open PRs that satisfy ALL of the following:
1. Are at least N days old (default: 7) since creation.
2. Have a Greptile (`greptile-apps`) review comment whose latest
Closes open PRs (including drafts, regardless of age) that satisfy ALL of:
1. Have a Greptile (`greptile-apps`) review comment whose latest
"Confidence Score: X/5" is below the configured threshold (default: 4).
3. Are not drafts.
4. Do not carry an opt-out label (default: "do not close").
2. Are authored by an external OSS contributor (internal BerriAI
contributors are exempt).
3. Do not carry an opt-out label (default: "do not close").
`--min-age-days` is retained as an opt-in safety net for one-off backfill
runs (default: 0). The team's intent is that the count of open PRs equals
the count of PRs internal collaborators need to action on, so neither age
nor draft status acts as a free pass.
For each match, the script posts an explanatory comment and closes the PR.
Contributors are invited to rebase and request a new Greptile review; if
Greptile then scores 4/5 or higher the PR can be reopened by anyone with
push access.
Because OSS contributors *cannot* reopen a PR closed by the bot/maintainer
(GitHub limitation), the close-comment instructs them to push their fixes
and **open a fresh PR**, or to comment `@agent-shin reconsider` on the
closed PR to have the LLM judge re-evaluate (and reopen on pass).
Requires the `gh` CLI to be authenticated.
@ -23,7 +29,7 @@ Usage examples:
# Actually close matching PRs
python3 close_low_quality_prs.py --close
# Tweak thresholds
# Restrict to PRs at least N days old (one-off backfill safety net)
python3 close_low_quality_prs.py --min-age-days 7 --min-score 4 --close
"""
@ -73,7 +79,13 @@ def gh(*args: str) -> str:
def fetch_open_prs(repo: str | None) -> list[dict]:
"""Fetch all open PRs (number, createdAt, isDraft, labels, author)."""
"""Fetch all open PRs (number, createdAt, isDraft, labels, author).
Includes drafts: `gh pr list --state open` returns both ready-for-review
and draft PRs by default. This is the desired behavior — drafts are not
a free pass; the internal-collaborator open-PR queue should reflect every
PR that needs human attention regardless of draft status.
"""
repo_args = ["--repo", repo] if repo else []
fields = "number,title,createdAt,isDraft,labels,author,url"
raw = gh(
@ -206,16 +218,22 @@ def close_pr(
comment_body = (
f"Closing as part of automated PR triage.\n\n"
f"This PR has been open for **{age_days} day(s)** and Greptile's most "
f"recent review scored it **{score}/5**, below our merge bar of "
f"**{threshold}/5**.\n\n"
f"Greptile's most recent review scored this PR **{score}/5**, below "
f"our merge bar of **{threshold}/5**.\n\n"
"We close low-confidence PRs aggressively to keep the review queue "
"manageable for maintainers and contributors alike. **This is not a "
"rejection of the idea** — to bring this back:\n\n"
"1. Rebase on the latest `main` and address the points Greptile raised.\n"
f"2. Re-request a review from `@greptileai` once you've pushed the fixes.\n"
f"3. If Greptile returns a score of **{threshold}/5 or higher**, reopen "
"this PR (or open a new one) — a maintainer will take another look.\n\n"
"1. Push the fixes that address Greptile's feedback (continue using "
"your existing branch is fine).\n"
"2. **Open a new PR** with the updated branch. Greptile will review "
"it again, and if it scores "
f"**{threshold}/5 or higher** a maintainer will take another look.\n\n"
"_Why open a new PR instead of reopening this one?_ GitHub does not "
"let external contributors reopen a PR that was closed by a bot or "
"maintainer, so a fresh PR is the most reliable path forward. If you "
"would prefer this exact PR re-evaluated, comment "
"`@agent-shin reconsider` once you've pushed the fixes — Agent Shin "
"will re-run triage and reopen this PR if it now meets the bar.\n\n"
"Thanks for contributing to LiteLLM. We know auto-closures can sting; "
"the goal is to keep the project healthy, not to dismiss your work."
)
@ -243,18 +261,23 @@ def evaluate_pr(
"""Decide whether to close `pr`.
Returns (action, score_or_none, age_days_or_none) where action is one of:
"skip-draft", "skip-too-young", "skip-optout-label", "skip-internal",
"skip-too-young", "skip-optout-label", "skip-internal",
"skip-no-greptile-score", "skip-score-ok", or "close".
"""
if pr.get("isDraft"):
return ("skip-draft", None, None)
Drafts are NOT skipped — the goal is "open PR count == PRs internal
collaborators need to action on", and a draft that Greptile scored <4/5
is still in that queue. Authors can opt out via the `wip` label (see
`DEFAULT_OPTOUT_LABELS`) if they need to keep a long-lived draft open.
"""
if has_optout_label(pr, optout_labels):
return ("skip-optout-label", None, None)
created = parse_iso8601(pr["createdAt"])
age_days = (now - created).days
if age_days < min_age_days:
# `min_age_days` defaults to 0 (close as soon as Greptile scores low).
# Set a positive value via --min-age-days for one-off backfill runs that
# want to skip very-young PRs.
if min_age_days > 0 and age_days < min_age_days:
return ("skip-too-young", None, age_days)
# Only auto-close external OSS contributors. Internal contributors
@ -285,8 +308,12 @@ def main() -> int:
parser.add_argument(
"--min-age-days",
type=int,
default=7,
help="Minimum age (in days) before a PR is eligible (default: 7).",
default=0,
help=(
"Minimum age (in days) before a PR is eligible. Default 0 = "
"close as soon as Greptile flags it. Set a positive value for "
"one-off backfill runs that want to spare very-young PRs."
),
)
parser.add_argument(
"--min-score",
@ -343,7 +370,6 @@ def main() -> int:
closed = 0
summary = {
"close": 0,
"skip-draft": 0,
"skip-too-young": 0,
"skip-optout-label": 0,
"skip-internal": 0,

164
.github/scripts/triage_with_llm.py vendored
View file

@ -114,6 +114,23 @@ def close_pr(repo: str, number: int) -> None:
)
def reopen_pr(repo: str, number: int) -> None:
"""Reopen a previously-closed pull request (state=open).
Used by the `@agent-shin reconsider` comment-trigger flow: the bot has
write access via GH_TOKEN, so it can reopen on the contributor's behalf
even though GitHub doesn't let the OSS author do it themselves.
"""
gh(
"api",
f"repos/{repo}/pulls/{number}",
"-X",
"PATCH",
"-f",
"state=open",
)
def close_issue(repo: str, number: int, *, not_planned: bool = True) -> None:
"""Close an issue, marking state_reason=not_planned by default."""
args = [
@ -129,6 +146,20 @@ def close_issue(repo: str, number: int, *, not_planned: bool = True) -> None:
gh(*args)
def reopen_issue(repo: str, number: int) -> None:
"""Reopen a previously-closed issue (state=open, state_reason=reopened)."""
gh(
"api",
f"repos/{repo}/issues/{number}",
"-X",
"PATCH",
"-f",
"state=open",
"-f",
"state_reason=reopened",
)
# ---------------------------------------------------------------------------
# Author classification
@ -356,12 +387,17 @@ def format_pr_close_comment(verdict: dict) -> str:
" - Link a related GitHub issue (e.g. `Fixes #1234`), OR\n"
" - Add a clear **problem description**, **expected vs. actual behavior**, and **visual QA proof** "
"(before/after screenshots, a short screen recording, or terminal/log output).\n"
"2. **Reopen** the PR (or open a fresh one) — I'll re-evaluate automatically.\n"
"2. Either:\n"
" - **Open a new PR** with the same fixes — recommended path. GitHub does not let external "
"contributors reopen a PR that was closed by a bot/maintainer, so a fresh PR is the most reliable way "
"to get back into the review queue.\n"
" - **Or** comment `@agent-shin reconsider` on this closed PR after updating the description. "
"I'll re-run the triage; if it now passes, I'll reopen this PR automatically.\n"
"\n"
"Internal BerriAI contributors: this rubric doesn't apply to you — ping a maintainer.\n"
"\n"
"_(I'm an LLM, so I'm not infallible. If you think I got this wrong, reopen and ping a maintainer — "
"they'll override me.)_"
"_(I'm an LLM, so I'm not infallible. If you think I got this wrong, comment "
"`@agent-shin reconsider` or ping a maintainer — they'll override me.)_"
)
@ -385,12 +421,15 @@ def format_issue_close_comment(verdict: dict) -> str:
"and a screenshot / traceback / log showing the bug.\n"
" - For **feature requests**: a concrete description of what should change, plus a use case and example "
"(config / API call / UI flow).\n"
"2. **Reopen** the issue — I'll re-evaluate automatically.\n"
"2. Comment `@agent-shin reconsider` on this issue once you've updated it. "
"I'll re-run triage and reopen the issue if it now meets the bar. "
"(GitHub doesn't always let the original reporter reopen a bot-closed issue, "
"so the comment-based reconsider is the reliable path.)\n"
"\n"
"Internal BerriAI contributors: this rubric doesn't apply to you — ping a maintainer.\n"
"\n"
"_(I'm an LLM, so I'm not infallible. If you think I got this wrong, reopen and ping a maintainer — "
"they'll override me.)_"
"_(I'm an LLM, so I'm not infallible. If you think I got this wrong, comment "
"`@agent-shin reconsider` or ping a maintainer — they'll override me.)_"
)
@ -416,6 +455,45 @@ def write_step_summary(content: str) -> None:
# Core orchestration
def format_reopen_comment(kind: str) -> str:
"""Comment posted when Agent Shin reopens after a successful reconsider."""
noun = "PR" if kind == "pr" else "issue"
return (
f"♻️ **Re-evaluated and reopened.** Thanks for updating the {noun}!\n"
"\n"
"Agent Shin re-ran triage on the latest description and it now meets "
"the bar. A maintainer will take another look soon — please don't "
f"close this {noun} again unless asked to.\n"
"\n"
"_(If a maintainer ends up closing this for non-rubric reasons, that "
"decision stands; comment `@agent-shin reconsider` again only if you "
"have substantively new information.)_"
)
def format_reconsider_still_failing_comment(kind: str, verdict: dict) -> str:
"""Comment posted when reconsider re-runs triage but the verdict is still fail."""
missing_lines = _format_missing(verdict.get("missing") or [])
explanation = verdict.get("explanation") or ""
noun = "PR" if kind == "pr" else "issue"
return (
f"⏸️ **Re-evaluated; this {noun} still doesn't meet the rubric.**\n"
"\n"
"Agent Shin re-ran triage on the current description but is still "
"missing:\n"
"\n"
f"{missing_lines}\n"
"\n"
f"> {explanation}\n"
"\n"
"Update the description with the missing pieces and comment "
"`@agent-shin reconsider` again, or ping a maintainer if you think "
"I got this wrong.\n"
"\n"
"_(I'm an LLM and I'm not infallible.)_"
)
def triage(
*,
repo: str,
@ -425,11 +503,21 @@ def triage(
model: str,
judge: Any = None,
print_prompt: bool = False,
reconsider: bool = False,
) -> dict:
"""Triage a single PR or issue. Returns a result dict for logging/tests.
`judge` is an optional callable `(prompt) -> str` for tests / dry-run with
a stub. In production, leave it None and the script uses `call_llm_judge`.
When `reconsider=True`, the closed-state guard is skipped and a
fail-but-no-comment is replaced with a "still failing" comment + leave
closed; a pass triggers `reopen_pr`/`reopen_issue` plus a reopen comment.
Reconsider mode is intended for the `@agent-shin reconsider` comment
trigger. `close` is forced True implicitly when `reconsider` is set
because the bot has already decided this is a real (non-dry-run)
invocation; it's the caller's responsibility to gate on
AGENT_SHIN_ENABLED before calling reconsider mode.
"""
fetcher = {"pr": fetch_pr, "issue": fetch_issue}[kind]
item = fetcher(repo, number)
@ -447,10 +535,18 @@ def triage(
"author": login,
"author_association": association,
"state": state,
"reconsider": reconsider,
}
if state != "open":
return {**base_result, "action": "skip-not-open"}
# Reconsider only makes sense on a closed PR/issue. A "reconsider on an
# open PR" is a no-op (the regular triage flow already evaluates open
# PRs); return a clear skip so the workflow can short-circuit.
if reconsider:
if state != "closed":
return {**base_result, "action": "skip-not-closed"}
else:
if state != "open":
return {**base_result, "action": "skip-not-open"}
if is_internal_contributor(item):
return {**base_result, "action": "skip-internal-author"}
@ -459,7 +555,7 @@ def triage(
prompt = build_pr_prompt(title=title, body=body)
# Short-circuit: if body very clearly links a related issue, just pass.
if has_linked_issue(body):
return {
base = {
**base_result,
"action": "pass-linked-issue",
"verdict": {
@ -468,6 +564,17 @@ def triage(
"explanation": "Linked-issue regex matched; LLM was not called.",
},
}
if reconsider:
# Pass-on-reconsider -> reopen the PR with a friendly comment.
reopen_body = format_reopen_comment(kind)
post_comment(repo, number, reopen_body)
reopen_pr(repo, number)
return {
**base,
"action": "reopened",
"comment": reopen_body,
}
return base
else:
prompt = build_issue_prompt(title=title, body=body)
@ -495,6 +602,33 @@ def triage(
return {**base_result, "action": "skip-llm-error", "error": str(exc)}
decision = (verdict.get("verdict") or "").lower()
if reconsider:
# Reconsider: pass -> reopen + post reopen comment;
# fail -> leave closed + post a "still failing" comment so the
# contributor can iterate again.
if decision != "fail":
reopen_body = format_reopen_comment(kind)
post_comment(repo, number, reopen_body)
if kind == "pr":
reopen_pr(repo, number)
else:
reopen_issue(repo, number)
return {
**base_result,
"action": "reopened",
"verdict": verdict,
"comment": reopen_body,
}
still_failing = format_reconsider_still_failing_comment(kind, verdict)
post_comment(repo, number, still_failing)
return {
**base_result,
"action": "reconsider-still-failing",
"verdict": verdict,
"comment": still_failing,
}
if decision != "fail":
return {**base_result, "action": "pass-llm", "verdict": verdict}
@ -579,6 +713,17 @@ def main() -> int:
action="store_true",
help="Print the prompt that would be sent to the judge and exit.",
)
parser.add_argument(
"--reconsider",
action="store_true",
help=(
"Re-run triage on a CLOSED PR/issue and reopen it on pass. "
"Used by the `@agent-shin reconsider` comment-trigger workflow. "
"Only invoke this from a workflow that has already gated on "
"AGENT_SHIN_ENABLED=true and verified the commenter is the "
"PR/issue author or an internal collaborator."
),
)
args = parser.parse_args()
kind = "pr" if args.pr is not None else "issue"
@ -591,6 +736,7 @@ def main() -> int:
close=args.close,
model=args.model,
print_prompt=args.print_prompt,
reconsider=args.reconsider,
)
if result.get("action") == "print-prompt":

22
.github/workflows/close_low_quality_prs.yml vendored
View file

@ -1,15 +1,17 @@
name: Close Low-Quality Stale PRs
name: Close Low-Quality PRs
# Auto-close PRs that have been open for a week or more and that Greptile has
# reviewed with a confidence score below 4/5. Closures are explained in a
# comment so contributors know how to bring the PR back (rebase, request a new
# Greptile review, reopen on a 4+/5).
# Auto-close any open PR (including drafts, regardless of age) authored by an
# external OSS contributor that Greptile reviewed with a confidence score
# below 4/5. Closures are explained in a comment that tells the contributor
# to push fixes and open a fresh PR (since OSS authors cannot reopen a PR
# closed by a bot/maintainer) or comment `@agent-shin reconsider` to have
# Agent Shin re-evaluate.
#
# Manual one-off run:
# gh workflow run "Close Low-Quality Stale PRs" -f close=true
# gh workflow run "Close Low-Quality PRs" -f close=true
#
# Dry-run preview (no PRs are touched):
# gh workflow run "Close Low-Quality Stale PRs" -f close=false
# gh workflow run "Close Low-Quality PRs" -f close=false
on:
schedule:
@ -26,9 +28,9 @@ on:
- "true"
- "false"
min_age_days:
description: "Minimum PR age in days before it is eligible."
description: "Minimum PR age in days (default 0 = no age filter)."
required: false
default: "7"
default: "0"
min_score:
description: "Greptile score below which a PR is closed (1-5)."
required: false
@ -68,7 +70,7 @@ jobs:
# on manual workflow_dispatch with close=true (and the variable set).
CLOSE_FLAG: ${{ github.event.inputs.close || 'false' }}
AGENT_SHIN_ENABLED: ${{ vars.AGENT_SHIN_ENABLED }}
MIN_AGE_DAYS: ${{ github.event.inputs.min_age_days || '7' }}
MIN_AGE_DAYS: ${{ github.event.inputs.min_age_days || '0' }}
MIN_SCORE: ${{ github.event.inputs.min_score || '4' }}
LIMIT: ${{ github.event.inputs.limit || '25' }}
run: |

126
.github/workflows/triage_reconsider.yml vendored Normal file
View file

@ -0,0 +1,126 @@
name: Agent Shin — reconsider
# Comment-trigger workflow: when the PR/issue author (or an internal
# collaborator) comments `@agent-shin reconsider` on a CLOSED PR/issue,
# Agent Shin re-runs LLM-judge triage on the current title+body and:
#
# - on PASS: posts a "re-evaluated and reopened" comment + reopens.
# - on FAIL: posts a "still missing X" comment and leaves it closed,
# so the contributor can iterate again.
#
# This exists because GitHub does NOT let an external (non-write-access)
# OSS contributor reopen a PR/issue closed by a bot or maintainer. Without
# this comment trigger, a contributor whose PR Agent Shin auto-closed
# would have no path back into the review queue except opening a fresh PR
# (which loses the original PR's history). The bot, on the other hand,
# has write access via GH_TOKEN and can reopen on their behalf.
#
# DRY-RUN BY DEFAULT — gated on `vars.AGENT_SHIN_ENABLED == 'true'` just
# like the other Agent Shin workflows. The workflow also gates on the
# commenter being either the PR/issue author or an internal collaborator
# (OWNER/MEMBER/COLLABORATOR) so random commenters cannot DOS the LLM
# judge or force a reopen.
on:
issue_comment:
types: [created]
permissions:
contents: read
issues: write
pull-requests: write
jobs:
reconsider:
if: |
github.repository == 'BerriAI/litellm'
&& contains(github.event.comment.body, '@agent-shin reconsider')
runs-on: ubuntu-latest
steps:
- name: Authorize commenter
# Only the PR/issue author OR an internal collaborator may trigger
# a reconsider. Outside random commenters could otherwise spam the
# phrase to burn LLM budget or, if a fail-open bug were ever
# introduced, force a reopen on someone else's behalf.
#
# We expose the authorization decision as a step output and gate
# every subsequent (potentially destructive) step on it. A `run:`
# step with `exit 0` would NOT stop the job — only `if:` gating
# on a known-true output is safe here.
id: auth
env:
COMMENTER: ${{ github.event.comment.user.login }}
AUTHOR: ${{ github.event.issue.user.login }}
ASSOCIATION: ${{ github.event.comment.author_association }}
run: |
set -euo pipefail
if [ "${COMMENTER}" = "${AUTHOR}" ]; then
echo "::notice::Authorized: commenter is the PR/issue author."
echo "authorized=true" >> "$GITHUB_OUTPUT"
exit 0
fi
case "${ASSOCIATION}" in
OWNER|MEMBER|COLLABORATOR)
echo "::notice::Authorized: commenter is an internal collaborator (${ASSOCIATION})."
echo "authorized=true" >> "$GITHUB_OUTPUT"
;;
*)
echo "::notice::Commenter '${COMMENTER}' (${ASSOCIATION}) is not authorized to trigger reconsider; skipping subsequent steps."
echo "authorized=false" >> "$GITHUB_OUTPUT"
;;
esac
- name: Checkout triage script
if: steps.auth.outputs.authorized == 'true'
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
sparse-checkout: .github/scripts
persist-credentials: false
- name: Set up Python
if: steps.auth.outputs.authorized == 'true'
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"
- name: Install LLM client
if: steps.auth.outputs.authorized == 'true'
run: pip install --no-cache-dir "openai>=1.40.0"
- name: Run Agent Shin reconsider
if: steps.auth.outputs.authorized == 'true'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENAI_BASE_URL: ${{ vars.OPENAI_BASE_URL }}
TRIAGE_MODEL: ${{ vars.TRIAGE_MODEL }}
AGENT_SHIN_ENABLED: ${{ vars.AGENT_SHIN_ENABLED }}
# `issue_comment` events fire for both issues and PR comments.
# `issue.pull_request` is set iff this is a PR comment, so we use
# its presence to decide whether to invoke `--pr N` or `--issue N`.
IS_PR: ${{ github.event.issue.pull_request != null }}
NUMBER: ${{ github.event.issue.number }}
run: |
set -euo pipefail
if [ "${IS_PR}" = "true" ]; then
ARGS=(--repo "${{ github.repository }}" --pr "${NUMBER}" --reconsider)
else
ARGS=(--repo "${{ github.repository }}" --issue "${NUMBER}" --reconsider)
fi
# Reconsider IS the destructive path here (it can post comments
# and reopen) — there's no separate `--close` flag because the
# script's reconsider mode handles both pass (reopen) and fail
# (still-failing comment) outcomes itself.
#
# Use the positive `= "true"` gate (instead of `!= "true" -> exit`)
# so the workflow guardrails in
# tests/test_litellm/test_github_triage_workflows.py see the
# canonical fail-safe enable pattern. Unknown values like "True",
# "yes", "1", or typos will fall through to the dry-run else
# branch, which is the safe default.
if [ "${AGENT_SHIN_ENABLED:-false}" = "true" ]; then
echo "::notice::Agent Shin reconsider ENABLED — running real triage."
python3 .github/scripts/triage_with_llm.py "${ARGS[@]}"
else
echo "::notice::AGENT_SHIN_ENABLED is not 'true' -> reconsider stays in dry-run (no comment, no reopen)."
fi

68
tests/test_litellm/test_github_close_low_quality_prs.py
View file

@ -146,22 +146,47 @@ class TestEvaluatePr:
closer_module, "is_external_pr_author", lambda pr, repo: True
)
def test_should_skip_drafts(self, closer_module, _now, monkeypatch):
def test_should_close_drafts_when_score_low(self, closer_module, _now, monkeypatch):
# Drafts are NOT a free pass — the open-PR queue should reflect any
# PR that needs human attention regardless of draft status. Authors
# who need a long-lived draft can use the `wip` opt-out label.
monkeypatch.setattr(
closer_module,
"fetch_pr_comments",
lambda *a, **kw: pytest.fail("should not fetch comments for drafts"),
lambda *a, **kw: [_greptile_comment("Confidence Score: 2/5")],
)
action, score, age = closer_module.evaluate_pr(
self._make_pr(is_draft=True),
self._make_pr(is_draft=True, created_days_ago=0),
now=_now,
min_age_days=7,
min_age_days=0,
min_score=4,
repo=None,
optout_labels=set(),
)
assert action == "skip-draft"
assert score is None and age is None
assert action == "close"
assert score == 2 and age == 0
def test_should_close_brand_new_pr_when_min_age_zero(
self, closer_module, _now, monkeypatch
):
# `min_age_days=0` means no age filter — a freshly-opened PR is
# eligible the moment Greptile scores it below threshold. This is
# the new default behavior.
monkeypatch.setattr(
closer_module,
"fetch_pr_comments",
lambda *a, **kw: [_greptile_comment("Confidence Score: 1/5")],
)
action, score, age = closer_module.evaluate_pr(
self._make_pr(created_days_ago=0),
now=_now,
min_age_days=0,
min_score=4,
repo=None,
optout_labels=set(),
)
assert action == "close"
assert score == 1 and age == 0
def test_should_skip_optout_label_case_insensitive(
self, closer_module, _now, monkeypatch
@ -181,7 +206,12 @@ class TestEvaluatePr:
)
assert action == "skip-optout-label"
def test_should_skip_too_young(self, closer_module, _now, monkeypatch):
def test_should_skip_too_young_when_min_age_set(
self, closer_module, _now, monkeypatch
):
# The min-age-days flag is now opt-in (default 0). When a maintainer
# explicitly passes a positive value (e.g. for a backfill run that
# wants to spare brand-new PRs), the skip-too-young path still works.
monkeypatch.setattr(
closer_module,
"fetch_pr_comments",
@ -198,6 +228,28 @@ class TestEvaluatePr:
assert action == "skip-too-young"
assert age == 2
def test_should_not_skip_when_min_age_is_zero(
self, closer_module, _now, monkeypatch
):
# With the new default min_age_days=0, even a 0-day-old PR is
# evaluated. This test pins that behavior so future refactors don't
# silently restore an age filter.
monkeypatch.setattr(
closer_module,
"fetch_pr_comments",
lambda *a, **kw: [_greptile_comment("Confidence Score: 5/5")],
)
action, score, age = closer_module.evaluate_pr(
self._make_pr(created_days_ago=0),
now=_now,
min_age_days=0,
min_score=4,
repo=None,
optout_labels=set(),
)
assert action == "skip-score-ok"
assert score == 5 and age == 0
def test_should_skip_when_greptile_has_not_reviewed(
self, closer_module, _now, monkeypatch
):
@ -303,7 +355,7 @@ class TestMainOptoutLabelDefault:
def fake_evaluate(pr, now, min_age_days, min_score, repo, optout_labels):
captured["optout_labels"] = set(optout_labels)
return ("skip-draft", None, None)
return ("skip-internal", None, None)
monkeypatch.setattr(closer_module, "evaluate_pr", fake_evaluate)
return captured

232
tests/test_litellm/test_github_triage_with_llm.py
View file

@ -123,6 +123,41 @@ class TestStripHtmlComments:
assert triage_module.strip_html_comments(None) == ""
class TestCloseCommentText:
"""Pin the user-facing language in close comments so changes are intentional."""
def test_pr_close_comment_should_recommend_new_pr_primarily(self, triage_module):
body = triage_module.format_pr_close_comment(
{"verdict": "fail", "missing": ["QA proof"], "explanation": "thin"}
)
# Primary path: open a new PR (because OSS authors can't reopen a
# bot-closed PR). Secondary path: `@agent-shin reconsider`.
assert "Open a new PR" in body
assert "@agent-shin reconsider" in body
# Old advice that no longer works for OSS contributors must NOT
# appear (they can't reopen a PR closed by a bot/maintainer).
assert "Reopen the PR" not in body
def test_pr_close_comment_should_not_promise_automatic_reopen_on_open(
self, triage_module
):
# The previous comment said "I'll re-evaluate automatically" — that
# only worked because the author could reopen, which they often
# can't. The new wording must point them at the comment trigger or
# a new PR instead.
body = triage_module.format_pr_close_comment(
{"verdict": "fail", "missing": [], "explanation": ""}
)
assert "I'll re-evaluate automatically" not in body
def test_issue_close_comment_should_use_reconsider_trigger(self, triage_module):
body = triage_module.format_issue_close_comment(
{"verdict": "fail", "missing": ["repro"], "explanation": "thin"}
)
assert "@agent-shin reconsider" in body
assert "Reopen the issue" not in body
class TestParseVerdict:
def test_should_parse_plain_json(self, triage_module):
raw = '{"verdict": "pass", "missing": []}'
@ -497,6 +532,203 @@ class TestTriageOrchestration:
assert result["action"] == "skip-llm-error"
assert "upstream 500" in result["error"]
def test_should_skip_open_pr_in_reconsider_mode(self, triage_module, monkeypatch):
# Reconsider only makes sense on a CLOSED PR — running it on an open
# one is a no-op (the regular triage flow already evaluated it).
pr = self._make_pr(state="open")
monkeypatch.setattr(triage_module, "fetch_pr", lambda repo, n: pr)
result = triage_module.triage(
repo="o/r",
kind="pr",
number=1,
close=False,
model="m",
judge=lambda p: pytest.fail("should not run on open PR in reconsider"),
reconsider=True,
)
assert result["action"] == "skip-not-closed"
def test_should_reopen_on_reconsider_pass(self, triage_module, monkeypatch):
# Reconsider on a closed PR with a passing verdict -> reopen + post a
# friendly "re-evaluated" comment.
pr = self._make_pr(
state="closed", body="Updated body with QA proof + screenshots."
)
monkeypatch.setattr(triage_module, "fetch_pr", lambda repo, n: pr)
posted = {}
reopened = {}
monkeypatch.setattr(
triage_module,
"post_comment",
lambda repo, n, body: posted.update({"n": n, "body": body}),
)
monkeypatch.setattr(
triage_module,
"reopen_pr",
lambda repo, n: reopened.update({"n": n}),
)
# close_pr / close_issue MUST NOT fire in reconsider mode.
monkeypatch.setattr(
triage_module,
"close_pr",
lambda *a, **kw: pytest.fail("must not close on reconsider pass"),
)
result = triage_module.triage(
repo="o/r",
kind="pr",
number=42,
close=False,
model="m",
judge=lambda p: json.dumps(
{"verdict": "pass", "missing": [], "explanation": "ok now"}
),
reconsider=True,
)
assert result["action"] == "reopened"
assert reopened["n"] == 42
assert posted["n"] == 42
assert "reopened" in posted["body"].lower()
def test_should_post_still_failing_on_reconsider_fail(
self, triage_module, monkeypatch
):
pr = self._make_pr(state="closed", body="still empty")
monkeypatch.setattr(triage_module, "fetch_pr", lambda repo, n: pr)
posted = {}
monkeypatch.setattr(
triage_module,
"post_comment",
lambda repo, n, body: posted.update({"n": n, "body": body}),
)
# Neither reopen nor close should fire when reconsider verdict is fail.
monkeypatch.setattr(
triage_module,
"reopen_pr",
lambda *a, **kw: pytest.fail("must not reopen on fail"),
)
monkeypatch.setattr(
triage_module,
"close_pr",
lambda *a, **kw: pytest.fail("must not close again on reconsider fail"),
)
verdict = {
"verdict": "fail",
"missing": ["QA proof"],
"explanation": "Still no QA proof.",
}
result = triage_module.triage(
repo="o/r",
kind="pr",
number=42,
close=False,
model="m",
judge=lambda p: json.dumps(verdict),
reconsider=True,
)
assert result["action"] == "reconsider-still-failing"
assert posted["n"] == 42
assert "QA proof" in posted["body"]
def test_should_reopen_on_reconsider_with_linked_issue_short_circuit(
self, triage_module, monkeypatch
):
# The linked-issue short-circuit also has to honor reconsider mode:
# if the contributor edited the body to add `Fixes #1234`, the regex
# path should reopen the PR without calling the LLM.
pr = self._make_pr(state="closed", body="Fixes #1234\n\nAddresses the bug.")
monkeypatch.setattr(triage_module, "fetch_pr", lambda repo, n: pr)
posted = {}
reopened = {}
monkeypatch.setattr(
triage_module,
"post_comment",
lambda repo, n, body: posted.update({"body": body}),
)
monkeypatch.setattr(
triage_module,
"reopen_pr",
lambda repo, n: reopened.update({"n": n}),
)
result = triage_module.triage(
repo="o/r",
kind="pr",
number=55,
close=False,
model="m",
judge=lambda p: pytest.fail("LLM must not run when linked-issue matches"),
reconsider=True,
)
assert result["action"] == "reopened"
assert reopened["n"] == 55
assert "reopened" in posted["body"].lower()
def test_should_skip_internal_in_reconsider_mode(self, triage_module, monkeypatch):
# Internal authors are exempt from triage in both regular and
# reconsider mode — Agent Shin should never reopen one of their PRs
# automatically, in case a maintainer closed it intentionally.
pr = self._make_pr(
state="closed",
author_association="MEMBER",
user={"login": "krrishdholakia"},
)
monkeypatch.setattr(triage_module, "fetch_pr", lambda repo, n: pr)
monkeypatch.setattr(
triage_module,
"reopen_pr",
lambda *a, **kw: pytest.fail("must not reopen for internal author"),
)
result = triage_module.triage(
repo="o/r",
kind="pr",
number=1,
close=False,
model="m",
judge=lambda p: pytest.fail("LLM must not run for internal author"),
reconsider=True,
)
assert result["action"] == "skip-internal-author"
def test_should_reopen_issue_on_reconsider_pass(self, triage_module, monkeypatch):
issue = {
"number": 7,
"title": "Bug: now with repro",
"body": "## Repro\n```bash\ncurl ...\n```\n\nExpected X, got Y.",
"state": "closed",
"author_association": "NONE",
"user": {"login": "outside"},
}
monkeypatch.setattr(triage_module, "fetch_issue", lambda repo, n: issue)
posted = {}
reopened = {}
monkeypatch.setattr(
triage_module,
"post_comment",
lambda repo, n, body: posted.update({"body": body}),
)
monkeypatch.setattr(
triage_module,
"reopen_issue",
lambda repo, n: reopened.update({"n": n}),
)
result = triage_module.triage(
repo="o/r",
kind="issue",
number=7,
close=False,
model="m",
judge=lambda p: json.dumps(
{"verdict": "pass", "missing": [], "explanation": "now reproducible"}
),
reconsider=True,
)
assert result["action"] == "reopened"
assert reopened["n"] == 7
assert "reopened" in posted["body"].lower()
def test_should_triage_issues_kind(self, triage_module, monkeypatch):
issue = {
"number": 7,

4
tests/test_litellm/test_github_triage_workflows.py
View file

@ -43,6 +43,10 @@ DESTRUCTIVE_GATE_ENV: dict[str, str] = {
"triage_pr_with_llm.yml": "DISPATCH_CLOSE",
"triage_issue_with_llm.yml": "DISPATCH_CLOSE",
"close_low_quality_prs.yml": "CLOSE_FLAG",
# The reconsider workflow has no per-run "really do it?" knob — its
# only kill switch is `AGENT_SHIN_ENABLED`, which already serves as
# both the destructive gate and the global enablement gate.
"triage_reconsider.yml": "AGENT_SHIN_ENABLED",
}