scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-14 23:21:35 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 102

Merge PR #28117 + #28147 (Agent Shin auto-close + reconsider safety)

Browse source
This commit is contained in:
Mateo 2026-05-24 17:41:17 -07:00
commit 7fb1ec6b9b
12 changed files with 4811 additions and 33 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

51
.github/ISSUE_TEMPLATE/bug_report.yml vendored
View file

@ -6,8 +6,11 @@ body:
- type: markdown
attributes:
value: |
Thanks for taking the time to fill out this bug report!
Thanks for taking the time to file a bug report!
> ⚠️ **Auto-triage notice for external contributors:**
> Bug reports without **clear reproduction steps, expected vs. actual behavior, and a screenshot or terminal/log output** are auto-closed by our LLM triage bot with an explanation of what was missing. You can fill in the missing details and reopen at any time — the bot will re-evaluate. Internal BerriAI contributors are exempt.
**💡 Tip:** See our [Troubleshooting Guide](https://docs.litellm.ai/docs/troubleshoot) for what information to include.
- type: checkboxes
id: duplicate-check
@ -20,21 +23,33 @@ body:
- type: textarea
id: what-happened
attributes:
label: What happened?
description: Also tell us, what did you expect to happen?
placeholder: Tell us what you see!
value: "A bug happened!"
label: What happened? (Actual behavior)
description: A clear description of what is happening today, with the bug.
placeholder: e.g. "Calling completion() with model=gpt-4o-mini returns an empty string."
validations:
required: true
- type: textarea
id: expected-behavior
attributes:
label: What did you expect to happen? (Expected behavior)
description: A clear description of what you expected to happen. **Required.**
placeholder: e.g. "I expected completion() to return the model's response text."
validations:
required: true
- type: textarea
id: steps-to-reproduce
attributes:
label: Steps to Reproduce
description: Please provide detailed steps to reproduce this bug(A curl/python code to reproduce the bug)
label: Steps to reproduce
description: |
Provide a minimal reproduction. Include a runnable Python snippet or a
`curl` command, your config.yaml if relevant, and the exact LiteLLM
version + Python version. Reports without a runnable reproduction are
auto-closed.
placeholder: |
1. config.yaml file/ .env file/ etc.
2. Run the following code...
3. Observe the error...
1. Create `config.yaml` with: ...
2. Start the proxy with: `litellm --config config.yaml --port 4000`
3. Run this Python / curl: ...
4. Observe: ...
value: |
1.
2.
@ -44,9 +59,15 @@ body:
- type: textarea
id: logs
attributes:
label: Relevant log output
description: Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks.
label: Relevant log output / screenshot
description: |
**Required.** Paste the full traceback, stderr, proxy logs, or attach a
screenshot showing the bug. For UI bugs a screenshot or screen
recording is mandatory. Without proof of the bug, the issue is
auto-closed.
render: shell
validations:
required: true
- type: dropdown
id: component
attributes:
@ -63,14 +84,14 @@ body:
- type: input
id: version
attributes:
label: What LiteLLM version are you on ?
label: What LiteLLM version are you on ?
placeholder: v1.53.1
validations:
required: true
- type: input
id: contact
attributes:
label: Twitter / LinkedIn details
label: Twitter / LinkedIn details
description: We announce new features on Twitter + LinkedIn. If this issue leads to an announcement, and you'd like a mention, we'll gladly shout you out!
placeholder: ex. @krrish_dh / https://www.linkedin.com/in/krish-d/
validations:

28
.github/ISSUE_TEMPLATE/feature_request.yml vendored
View file

@ -1,4 +1,4 @@
name: 🚀 Feature Request
name: 🚀 Feature Request
description: Submit a proposal/request for a new LiteLLM feature.
title: "[Feature]: "
labels: ["enhancement"]
@ -6,7 +6,10 @@ body:
- type: markdown
attributes:
value: |
Thanks for making LiteLLM better!
Thanks for making LiteLLM better!
> ⚠️ **Auto-triage notice for external contributors:**
> Feature requests need (1) a clear description of the proposed feature, (2) the motivation / use case with a concrete example, and (3) what success looks like. Vague requests are auto-closed by our LLM triage bot with an explanation. Fill in the missing details and reopen at any time — the bot will re-evaluate. Internal BerriAI contributors are exempt.
- type: checkboxes
id: duplicate-check
attributes:
@ -18,16 +21,25 @@ body:
- type: textarea
id: the-feature
attributes:
label: The Feature
description: A clear and concise description of the feature proposal
placeholder: Tell us what you want!
label: The feature
description: A clear and concise description of the feature proposal. What should LiteLLM do that it doesn't today?
placeholder: e.g. "Support per-team max_input_tokens overrides on the proxy."
validations:
required: true
- type: textarea
id: motivation
attributes:
label: Motivation, pitch
description: Please outline the motivation for the proposal. Is your feature request related to a specific problem? e.g., "I'm working on X and would like Y to be possible". If this is related to another GitHub issue, please link here too.
label: Motivation, pitch, and concrete example
description: |
**Required.** Why is this needed? Include a concrete use case — what
you're trying to accomplish, what's blocked today, and what success
would look like (ideally with an example config / API call / UI flow).
If this is related to another GitHub issue, link it here too.
placeholder: |
I'm running a multi-tenant proxy where team A processes long docs and
team B only does short chats. Today I have to spin up two proxies.
With this feature I could set max_input_tokens per team and route in one
proxy. Example config: ...
validations:
required: true
- type: dropdown
@ -56,7 +68,7 @@ body:
- type: input
id: contact
attributes:
label: Twitter / LinkedIn details
label: Twitter / LinkedIn details
description: We announce new features on Twitter + LinkedIn. When this is announced, and you'd like a mention, we'll gladly shout you out!
placeholder: ex. @krrish_dh / https://www.linkedin.com/in/krish-d/
validations:

69
.github/pull_request_template.md vendored
View file

@ -1,10 +1,66 @@
<!--
👋 Hi there — please read before submitting.
To keep the review queue healthy for everyone, **every external PR is
auto-triaged** by an LLM bot ("Agent Shin") on open / reopen, regardless of
whether the PR is a draft or marked ready for review. PRs that don't meet
the rubric below are auto-closed with an explanation.
To pass triage, your PR must satisfy AT LEAST ONE of:
(A) Link a related GitHub issue (e.g. "Fixes #1234" or "Resolves
https://github.com/BerriAI/litellm/issues/1234"), OR
(B) Provide ALL of the following IN THIS PR DESCRIPTION:
- A clear problem description (what bug or missing feature this addresses)
- Expected vs. actual behavior
- Visual QA proof (before/after screenshots, screen recording, or
terminal output demonstrating that the fix/feature works end-to-end)
Every external PR (including drafts, regardless of age) also receives a
Greptile code review. Any PR with a Greptile Confidence Score below 4/5 is
auto-closed.
If your PR was auto-closed and you've addressed the feedback, you have two
options to bring it back:
- **Open a new PR** with the updated branch (recommended — GitHub does not
let external contributors reopen a PR that was closed by a bot or
maintainer).
- **Or** comment `@agent-shin reconsider` on the closed PR. Agent Shin
will re-run triage and reopen the PR if it now meets the bar.
Internal BerriAI contributors are exempt from this auto-triage — fill in the
Linear ticket section instead.
-->
## Relevant issues
<!-- e.g. "Fixes #000" -->
<!-- e.g. "Fixes #000". If you have no related issue, fill in the
"Problem description / Expected vs. Actual / QA proof" sections below. -->
## Linear ticket
<!-- if you are an internal contributor, add the Linear ticket e.g. "Resolves LIT-1234" to magically link the Linear ticket to the GitHub PR -->
<!-- INTERNAL CONTRIBUTORS ONLY: add the Linear ticket e.g. "Resolves LIT-1234"
to magically link the Linear ticket to the GitHub PR. External contributors:
leave this blank and fill in the problem/expected-actual/QA sections below. -->
## Problem description
<!-- What bug or missing feature does this PR address? One or two paragraphs.
External contributors: required unless you linked a GitHub issue above. -->
## Expected vs. actual behavior
<!-- What did you expect to happen? What is happening today (before this PR)?
External contributors: required unless you linked a GitHub issue above. -->
## QA proof
<!-- Required for external contributors: include before/after screenshots,
a screen recording, or terminal/log output that demonstrates the fix or feature
works end-to-end. For UI changes, before/after screenshots are mandatory.
For backend changes, terminal output of a passing test or curl command is fine. -->
## Pre-Submission checklist
@ -13,7 +69,7 @@
- [ ] I have Added testing in the [`tests/test_litellm/`](https://github.com/BerriAI/litellm/tree/main/tests/test_litellm) directory, **Adding at least 1 test is a hard requirement** - [see details](https://docs.litellm.ai/docs/extras/contributing_code)
- [ ] My PR passes all unit tests on [`make test-unit`](https://docs.litellm.ai/docs/extras/contributing_code)
- [ ] My PR's scope is as isolated as possible, it only solves 1 specific problem
- [ ] I have requested a Greptile review by commenting `@greptileai` and received a **Confidence Score of at least 4/5** before requesting a maintainer review
- [ ] I have received a Greptile **Confidence Score of at least 4/5** before requesting a maintainer review (Greptile reviews automatically on open; comment `@greptileai` to re-trigger after pushing fixes)
## Delays in PR merge?
@ -36,13 +92,6 @@ If you're seeing a delay in your PR being merged, ping the LiteLLM Team on [Slac
- [ ] **Merge / cherry-pick CI run**
Links:
## Screenshots / Proof of Fix
<!-- Include screenshots, screen recordings, or log output demonstrating that your changes work as expected.
For bug fixes: show reproduction before the fix and passing behavior after.
For new features: show the feature working end-to-end.
For UI changes: include before/after screenshots. -->
## Type
<!-- Select the type of Pull Request -->

645
.github/scripts/close_low_quality_prs.py vendored Normal file
View file

@ -0,0 +1,645 @@
#!/usr/bin/env python3
"""
Auto-close low-quality pull requests.
Closes open PRs (including drafts, regardless of age) that satisfy ALL of:
1. Have a Greptile (`greptile-apps`) review comment whose latest
"Confidence Score: X/5" is below the configured threshold (default: 4).
2. Are authored by an external OSS contributor (internal BerriAI
contributors are exempt).
3. Do not carry an opt-out label (default: "do not close").
`--min-age-days` is retained as an opt-in safety net for one-off backfill
runs (default: 0). The team's intent is that the count of open PRs equals
the count of PRs internal collaborators need to action on, so neither age
nor draft status acts as a free pass.
For each match, the script posts an explanatory comment and closes the PR.
Because OSS contributors *cannot* reopen a PR closed by the bot/maintainer
(GitHub limitation), the close-comment instructs them to push their fixes
and **open a fresh PR**, or to comment `@agent-shin reconsider` on the
closed PR to have the LLM judge re-evaluate (and reopen on pass).
Requires the `gh` CLI to be authenticated.
Usage examples:
# Dry run (default) - prints what would be closed
python3 close_low_quality_prs.py
# Actually close matching PRs
python3 close_low_quality_prs.py --close
# Restrict to PRs at least N days old (one-off backfill safety net)
python3 close_low_quality_prs.py --min-age-days 7 --min-score 4 --close
"""
from __future__ import annotations
import argparse
import datetime as dt
import json
import os
import re
import subprocess
import sys
from typing import Iterable
# Greptile's GitHub App appears as `greptile-apps[bot]` in REST API comments
# and `greptile-apps` in `gh pr view --json` output. Accept either form.
GREPTILE_BOT_LOGINS = frozenset({"greptile-apps", "greptile-apps[bot]"})
# Matches lines like:
# <h3>Confidence Score: 3/5</h3>
# **Confidence Score: 4/5**
# Confidence Score: 5 / 5
SCORE_PATTERN = re.compile(
r"confidence\s*score\s*[:\-]?\s*(\d+)\s*/\s*5",
re.IGNORECASE,
)
# `author_association` values for internal BerriAI contributors who should be
# exempt from auto-triage.
INTERNAL_AUTHOR_ASSOCIATIONS = frozenset({"OWNER", "MEMBER", "COLLABORATOR"})
# Default labels that exempt a PR from auto-close. Defined at module scope (not
# as a mutable argparse default) so that `--optout-label foo` REPLACES the
# defaults instead of appending to them — the argparse `action="append"` +
# `default=[...]` combination silently mutates the shared default list.
DEFAULT_OPTOUT_LABELS = ("do not close", "keep open", "wip")
# HTML marker appended to grace-period warning comments. Shared with the
# Agent Shin LLM-judge script (`triage_with_llm.py`) so a warning posted
# by either path is recognized by both: the LLM judge can see "Greptile
# already warned this contributor 12 hours ago" and skip re-warning, and
# the Greptile closer can see "Agent Shin already warned" and close on
# the next run if Greptile still has a low score.
GRACE_COMMENT_MARKER = "<!-- agent-shin:grace-warning -->"
# Length of the grace period between the warning comment and the actual
# auto-close. Set to 24 hours so the contributor has at least one full
# working day across any time zone to push fixes or comment
# `@agent-shin reconsider`. Mirrors the constant of the same name in
# `triage_with_llm.py` — keep them in sync if either changes.
GRACE_PERIOD_SECONDS = 86400
# Default login of the GitHub identity that performs Agent Shin's writes;
# used for matching the author of a grace-warning comment so we don't
# count somebody quoting the marker. The env override
# `AGENT_SHIN_BOT_LOGIN` mirrors `triage_with_llm.py`.
AGENT_SHIN_DEFAULT_BOT_LOGIN = "github-actions[bot]"
# Logins (case-insensitive) that bypass BOTH the 1-day grace period AND
# the dry-run gating. Mirrors `IMMEDIATE_CLOSE_LOGINS` in
# `triage_with_llm.py`. Used for dogfooding the bot from external test
# accounts that have no push permissions to the repo.
IMMEDIATE_CLOSE_LOGINS = frozenset({"swiftwinds"})
def gh(*args: str) -> str:
"""Run a `gh` CLI command and return stdout. Raises on non-zero exit."""
result = subprocess.run(
["gh", *args],
capture_output=True,
text=True,
check=True,
)
return result.stdout
def fetch_open_prs(repo: str | None) -> list[dict]:
"""Fetch all open PRs (number, createdAt, isDraft, labels, author).
Includes drafts: `gh pr list --state open` returns both ready-for-review
and draft PRs by default. This is the desired behavior — drafts are not
a free pass; the internal-collaborator open-PR queue should reflect every
PR that needs human attention regardless of draft status.
"""
repo_args = ["--repo", repo] if repo else []
fields = "number,title,createdAt,isDraft,labels,author,url"
raw = gh(
"pr",
"list",
"--state",
"open",
"--limit",
"1000",
"--json",
fields,
*repo_args,
)
return json.loads(raw)
def fetch_pr_author_association(pr_number: int, repo: str | None) -> str:
"""Return the GitHub `author_association` for a PR, uppercase.
Values: OWNER, MEMBER, COLLABORATOR, CONTRIBUTOR, FIRST_TIME_CONTRIBUTOR,
FIRST_TIMER, MANNEQUIN, NONE. Returns "" on lookup failure.
"""
endpoint = (
f"repos/{repo}/pulls/{pr_number}"
if repo
else f"repos/{{owner}}/{{repo}}/pulls/{pr_number}"
)
try:
data = json.loads(gh("api", endpoint))
except subprocess.CalledProcessError:
return ""
return (data.get("author_association") or "").upper()
def is_external_pr_author(pr: dict, repo: str | None) -> bool:
"""Return True if the PR author is an external OSS contributor.
Internal = `OWNER` / `MEMBER` / `COLLABORATOR` association, or a bot login.
"""
login = ((pr.get("author") or {}).get("login") or "").lower()
if login.endswith("[bot]") or login in {"dependabot", "github-actions"}:
return False
association = fetch_pr_author_association(pr["number"], repo)
# Fail-safe: if the API lookup failed (empty string), treat the author as
# internal so we don't auto-close their PR. Auto-close is destructive, so
# an unknown association should never make a PR eligible for closing.
if not association or association in INTERNAL_AUTHOR_ASSOCIATIONS:
return False
return True
def fetch_pr_comments(pr_number: int, repo: str | None) -> list[dict]:
"""Fetch issue-level comments on a PR (where Greptile posts its summary)."""
endpoint = (
f"repos/{repo}/issues/{pr_number}/comments?per_page=100"
if repo
else f"repos/{{owner}}/{{repo}}/issues/{pr_number}/comments?per_page=100"
)
raw = gh("api", "--paginate", endpoint)
comments: list[dict] = []
for line in raw.strip().splitlines():
line = line.strip()
if not line:
continue
parsed = json.loads(line)
if isinstance(parsed, list):
comments.extend(parsed)
else:
comments.append(parsed)
return comments
def extract_greptile_score(comments: Iterable[dict]) -> tuple[int, dict] | None:
"""Return (score, comment) for the most recent Greptile-authored comment
that contains a "Confidence Score: X/5". Returns None if no such comment.
"Most recent" is determined by the comment's `updated_at` (falling back to
`created_at`), so re-reviews override earlier passes.
"""
candidates: list[tuple[str, int, dict]] = []
for comment in comments:
user = (comment.get("user") or {}).get("login", "")
if user not in GREPTILE_BOT_LOGINS:
continue
body = comment.get("body") or ""
match = SCORE_PATTERN.search(body)
if not match:
continue
score = int(match.group(1))
timestamp = comment.get("updated_at") or comment.get("created_at") or ""
candidates.append((timestamp, score, comment))
if not candidates:
return None
candidates.sort(key=lambda triple: triple[0])
_, score, comment = candidates[-1]
return score, comment
def parse_iso8601(value: str) -> dt.datetime:
"""Parse a GitHub ISO-8601 timestamp into a timezone-aware datetime."""
return dt.datetime.fromisoformat(value.replace("Z", "+00:00"))
def has_optout_label(pr: dict, optout_labels: set[str]) -> bool:
labels = {label.get("name", "").lower() for label in pr.get("labels", [])}
return bool(labels & {lbl.lower() for lbl in optout_labels})
def seconds_since_last_grace_warning(
comments: Iterable[dict],
*,
bot_login: str | None = None,
now: dt.datetime | None = None,
) -> float | None:
"""Return seconds since the bot's most recent grace-period warning, or
None if no such warning has ever been posted on this PR.
Detects warnings by matching `GRACE_COMMENT_MARKER` in comments
authored by the bot identity. Operates on an already-fetched
comments list (avoids a second `gh api` call when the caller has
already pulled the page for Greptile-score extraction).
`now` is injectable so callers (and tests) can pin the reference
time. The closer runs everything against a single `now` snapshot
captured at the top of `main()` so age calculations stay consistent
across many PRs in a single run.
"""
expected_login = (
bot_login
or os.environ.get("AGENT_SHIN_BOT_LOGIN")
or AGENT_SHIN_DEFAULT_BOT_LOGIN
).lower()
latest: dt.datetime | None = None
for comment in comments:
author = ((comment.get("user") or {}).get("login") or "").lower()
if author != expected_login:
continue
body = comment.get("body") or ""
if GRACE_COMMENT_MARKER not in body:
continue
created = comment.get("created_at")
if not created:
continue
try:
ts = parse_iso8601(created)
except ValueError:
continue
if latest is None or ts > latest:
latest = ts
if latest is None:
return None
reference = now if now is not None else dt.datetime.now(dt.timezone.utc)
return (reference - latest).total_seconds()
def format_grace_warning_comment(score: int, threshold: int) -> str:
"""Comment posted on the FIRST low-Greptile-score detection — gives
the contributor a 1-day grace window before the auto-close fires on
the next daily cron run.
Mirrors `format_grace_warning_pr_comment` in
`triage_with_llm.py` in spirit (1-day grace + escape hatches), but
framed around Greptile's confidence score instead of the LLM judge's
rubric since the close trigger here is the Greptile signal.
"""
return (
"👋 Hi, thanks for the PR! I'm **Agent Shin**, the automated triage bot for this repository.\n"
"\n"
"Heads up — Greptile's most recent review scored this PR "
f"**{score}/5**, below our merge bar of **{threshold}/5**.\n"
"\n"
"⏳ **You have 1 day to address Greptile's feedback before this PR is auto-closed.** "
"We close low-confidence PRs aggressively to keep the review queue manageable for "
"maintainers and contributors alike. **This isn't a rejection of the idea.**\n"
"\n"
"During the grace period:\n"
"\n"
"1. Push fixes that address Greptile's feedback (continue using your existing branch is fine).\n"
"2. Either:\n"
" - Comment `@greptileai` to request a fresh Greptile review. If the new score is "
f"**{threshold}/5 or higher**, the PR stays open.\n"
" - Or comment `@agent-shin reconsider` to have Agent Shin re-evaluate the PR description.\n"
"\n"
"If this PR is auto-closed in 24 hours, you'll still have options:\n"
"\n"
"- Comment `@agent-shin reconsider` after pushing fixes — Agent Shin will re-run triage "
"and reopen the PR if it now meets the bar.\n"
"- Comment `@greptileai` to request a re-review — that works **even after the PR is closed**.\n"
"\n"
"Thanks for contributing to LiteLLM. We know auto-closures can sting; the goal is to keep "
"the project healthy, not to dismiss your work.\n"
"\n"
f"{GRACE_COMMENT_MARKER}"
)
def post_grace_warning(
pr: dict,
score: int,
threshold: int,
repo: str | None,
dry_run: bool,
) -> None:
"""Post the 1-day grace-period warning comment on `pr`.
The warning carries `GRACE_COMMENT_MARKER` so subsequent runs can
detect that the contributor has already been told about the
pending close. Does NOT close the PR — the close happens on the
next eligible run after `GRACE_PERIOD_SECONDS` elapses (handled
by `close_pr`).
"""
pr_number = pr["number"]
repo_args = ["--repo", repo] if repo else []
if dry_run:
print(
f" [DRY RUN] Would post grace warning to PR #{pr_number} "
f"(greptile={score}/5): {pr['title']}"
)
return
comment_body = format_grace_warning_comment(score, threshold)
gh("pr", "comment", str(pr_number), "--body", comment_body, *repo_args)
print(f" Posted grace warning on PR #{pr_number} (greptile={score}/5)")
def close_pr(
pr: dict,
score: int,
threshold: int,
age_days: int,
repo: str | None,
dry_run: bool,
label: str | None,
grace_period_elapsed: bool = True,
) -> None:
"""Post the explanatory comment and close the PR."""
pr_number = pr["number"]
repo_args = ["--repo", repo] if repo else []
if dry_run:
print(
f" [DRY RUN] Would close PR #{pr_number} "
f"(age={age_days}d, greptile={score}/5): {pr['title']}"
)
return
score_sentence = (
f"Greptile's most recent review scored this PR **{score}/5**, below "
f"our merge bar of **{threshold}/5**, and the 1-day grace period since "
"the warning has elapsed.\n\n"
if grace_period_elapsed
else (
f"Greptile's most recent review scored this PR **{score}/5**, "
f"below our merge bar of **{threshold}/5**.\n\n"
)
)
comment_body = (
f"Closing as part of automated PR triage.\n\n"
f"{score_sentence}"
"We close low-confidence PRs aggressively to keep the review queue "
"manageable for maintainers and contributors alike. **This is not a "
"rejection of the idea** — to bring this back:\n\n"
"1. Push the fixes that address Greptile's feedback (continue using "
"your existing branch is fine).\n"
"2. **Open a new PR** with the updated branch. Greptile will review "
"it again, and if it scores "
f"**{threshold}/5 or higher** a maintainer will take another look.\n\n"
"_Why open a new PR instead of reopening this one?_ GitHub does not "
"let external contributors reopen a PR that was closed by a bot or "
"maintainer, so a fresh PR is the most reliable path forward. If you "
"would prefer this exact PR re-evaluated, comment "
"`@agent-shin reconsider` once you've pushed the fixes — Agent Shin "
"will re-run triage and reopen this PR if it now meets the bar. "
"You can also comment `@greptileai` to request a fresh Greptile "
"review — that works **even after the PR is closed**.\n\n"
"Thanks for contributing to LiteLLM. We know auto-closures can sting; "
"the goal is to keep the project healthy, not to dismiss your work."
)
gh("pr", "comment", str(pr_number), "--body", comment_body, *repo_args)
if label:
try:
gh("pr", "edit", str(pr_number), "--add-label", label, *repo_args)
except subprocess.CalledProcessError as exc:
stderr = (exc.stderr or "").strip()
print(f" warn: failed to add label '{label}' to #{pr_number}: {stderr}")
gh("pr", "close", str(pr_number), *repo_args)
print(f" Closed PR #{pr_number} (greptile={score}/5, age={age_days}d)")
def evaluate_pr(
pr: dict,
now: dt.datetime,
min_age_days: int,
min_score: int,
repo: str | None,
optout_labels: set[str],
) -> tuple[str, int | None, int | None]:
"""Decide what to do with `pr` on this triage run.
Returns (action, score_or_none, age_days_or_none) where action is one of:
"skip-too-young", "skip-optout-label", "skip-internal",
"skip-no-greptile-score", "skip-score-ok",
"warn-grace", "skip-in-grace-period", or "close".
Drafts are NOT skipped — the goal is "open PR count == PRs internal
collaborators need to action on", and a draft that Greptile scored <4/5
is still in that queue. Authors can opt out via the `wip` label (see
`DEFAULT_OPTOUT_LABELS`) if they need to keep a long-lived draft open.
Grace-period semantics: the first time a PR fails the rubric, the
action is `warn-grace` — the caller should post a warning comment but
NOT close the PR. On a subsequent run, if the warning is still less
than `GRACE_PERIOD_SECONDS` old AND the PR still fails, the action is
`skip-in-grace-period`. Once the warning ages out and the rubric is
still failing, the action is `close`.
Grace is bypassed for `IMMEDIATE_CLOSE_LOGINS` (test/dogfood
accounts), which always go straight to `close` on the first failing
run so the bot is dogfoodable end-to-end without a 24h delay.
"""
if has_optout_label(pr, optout_labels):
return ("skip-optout-label", None, None)
created = parse_iso8601(pr["createdAt"])
age_days = (now - created).days
# `min_age_days` defaults to 0 (close as soon as Greptile scores low).
# Set a positive value via --min-age-days for one-off backfill runs that
# want to skip very-young PRs.
if min_age_days > 0 and age_days < min_age_days:
return ("skip-too-young", None, age_days)
# Only auto-close external OSS contributors. Internal contributors
# (BerriAI org members) handle their own backlog.
if not is_external_pr_author(pr, repo):
return ("skip-internal", None, age_days)
comments = fetch_pr_comments(pr["number"], repo)
extraction = extract_greptile_score(comments)
if extraction is None:
return ("skip-no-greptile-score", None, age_days)
score, _ = extraction
if score >= min_score:
return ("skip-score-ok", score, age_days)
login = ((pr.get("author") or {}).get("login") or "").lower()
if login in IMMEDIATE_CLOSE_LOGINS:
return ("close", score, age_days)
grace_age = seconds_since_last_grace_warning(comments, now=now)
if grace_age is None:
return ("warn-grace", score, age_days)
if grace_age < GRACE_PERIOD_SECONDS:
return ("skip-in-grace-period", score, age_days)
return ("close", score, age_days)
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--repo",
type=str,
default=None,
help="Repository (owner/repo). Auto-detected if omitted.",
)
parser.add_argument(
"--min-age-days",
type=int,
default=0,
help=(
"Minimum age (in days) before a PR is eligible. Default 0 = "
"close as soon as Greptile flags it. Set a positive value for "
"one-off backfill runs that want to spare very-young PRs."
),
)
parser.add_argument(
"--min-score",
type=int,
default=4,
choices=range(1, 6),
help="Greptile score below which a PR is closed (default: 4 -> closes <4/5).",
)
parser.add_argument(
"--optout-label",
action="append",
default=None,
help=(
"Label(s) that exempt a PR from auto-close. Repeat to add more. "
"Case-insensitive. When omitted, defaults to "
f"{list(DEFAULT_OPTOUT_LABELS)!r}; passing this flag REPLACES the "
"defaults (argparse `append` with a mutable default would append "
"instead, which we explicitly avoid)."
),
)
parser.add_argument(
"--close-label",
type=str,
default=None,
help=(
"Optional label to add to PRs that get auto-closed "
"(e.g. 'auto-closed-low-quality'). Must already exist on the repo."
),
)
parser.add_argument(
"--close",
action="store_true",
help="Actually close matching PRs (default is dry-run).",
)
parser.add_argument(
"--limit",
type=int,
default=None,
help="Maximum number of PRs to close in one run (safety net).",
)
args = parser.parse_args()
dry_run = not args.close
if dry_run:
print("=== DRY RUN MODE (pass --close to actually close PRs) ===\n")
print("Fetching open PRs...")
prs = fetch_open_prs(args.repo)
print(f"Found {len(prs)} open PRs.\n")
now = dt.datetime.now(dt.timezone.utc)
optout_labels = set(args.optout_label or DEFAULT_OPTOUT_LABELS)
closed = 0
summary = {
"close": 0,
"warn-grace": 0,
"skip-in-grace-period": 0,
"skip-too-young": 0,
"skip-optout-label": 0,
"skip-internal": 0,
"skip-no-greptile-score": 0,
"skip-score-ok": 0,
}
for pr in sorted(prs, key=lambda p: p["createdAt"]):
action, score, age_days = evaluate_pr(
pr,
now,
args.min_age_days,
args.min_score,
args.repo,
optout_labels,
)
summary[action] = summary.get(action, 0) + 1
# Per-PR dry-run override: `IMMEDIATE_CLOSE_LOGINS` accounts (e.g.
# SwiftWinds) always run in real-close mode regardless of the
# global `--close` flag. Lets a maintainer dogfood the bot from
# an external account while the rest of the open-PR queue stays
# on the safe dry-run default.
author_login = ((pr.get("author") or {}).get("login") or "").lower()
is_immediate = author_login in IMMEDIATE_CLOSE_LOGINS
pr_dry_run = dry_run and not is_immediate
if action == "warn-grace":
assert score is not None
print(
f"#{pr['number']}: \"{pr['title']}\" "
f"(age={age_days}d, greptile={score}/5) -> warn-grace"
)
post_grace_warning(
pr,
score=score,
threshold=args.min_score,
repo=args.repo,
dry_run=pr_dry_run,
)
continue
if action != "close":
continue
assert score is not None and age_days is not None
print(
f"#{pr['number']}: \"{pr['title']}\" "
f"(age={age_days}d, greptile={score}/5) -> close"
+ (" [immediate-close login]" if is_immediate else "")
)
close_pr(
pr,
score=score,
threshold=args.min_score,
age_days=age_days,
repo=args.repo,
dry_run=pr_dry_run,
label=args.close_label,
grace_period_elapsed=not is_immediate,
)
if not pr_dry_run:
closed += 1
if args.limit is not None and closed >= args.limit:
print(f"\nReached --limit={args.limit}; stopping.")
break
print("\n=== Summary ===")
for key, value in summary.items():
print(f" {key:28s} {value}")
# `IMMEDIATE_CLOSE_LOGINS` PRs are closed even in global dry-run mode, so
# report actual closures alongside the dry-run "would close" count to avoid
# misleading operators into thinking no writes occurred.
would_close = summary["close"] - closed
if dry_run:
if closed:
print(f"\nTotal closed: {closed}; would close: {would_close}")
else:
print(f"\nTotal would close: {would_close}")
else:
print(f"\nTotal closed: {closed}")
print(
f"Total {'would warn (grace)' if dry_run else 'warned (grace)'}: "
f"{summary['warn-grace']}"
)
return 0
if __name__ == "__main__":
sys.exit(main())

1126
.github/scripts/triage_with_llm.py vendored Normal file
View file

File diff suppressed because it is too large Load diff

92
.github/workflows/close_low_quality_prs.yml vendored Normal file
View file

@ -0,0 +1,92 @@
name: Close Low-Quality PRs
# Auto-close any open PR (including drafts, regardless of age) authored by an
# external OSS contributor that Greptile reviewed with a confidence score
# below 4/5. Closures are explained in a comment that tells the contributor
# to push fixes and open a fresh PR (since OSS authors cannot reopen a PR
# closed by a bot/maintainer) or comment `@agent-shin reconsider` to have
# Agent Shin re-evaluate.
#
# Manual one-off run:
# gh workflow run "Close Low-Quality PRs" -f close=true
#
# Dry-run preview (no PRs are touched):
# gh workflow run "Close Low-Quality PRs" -f close=false
on:
schedule:
# Daily at 09:00 UTC. Pairs well with the stale-issue workflow at midnight.
- cron: "0 9 * * *"
workflow_dispatch:
inputs:
close:
description: "Actually close matching PRs (false = dry run)."
required: false
default: "false"
type: choice
options:
- "true"
- "false"
min_age_days:
description: "Minimum PR age in days (default 0 = no age filter)."
required: false
default: "0"
min_score:
description: "Greptile score below which a PR is closed (1-5)."
required: false
default: "4"
limit:
description: "Maximum number of PRs to close in a single run."
required: false
default: "25"
permissions:
contents: read
pull-requests: write
issues: write
jobs:
close-low-quality-prs:
if: github.repository == 'BerriAI/litellm'
runs-on: ubuntu-latest
steps:
- name: Checkout triage script
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
sparse-checkout: .github/scripts
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"
- name: Run low-quality PR closer
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# Scheduled runs are ALWAYS dry-run, even when AGENT_SHIN_ENABLED is
# "true", so the team can QA the closer's verdicts in step summaries
# before any contributor sees a PR closed. Real closures only happen
# on manual workflow_dispatch with close=true (and the variable set).
CLOSE_FLAG: ${{ github.event.inputs.close || 'false' }}
AGENT_SHIN_ENABLED: ${{ vars.AGENT_SHIN_ENABLED }}
MIN_AGE_DAYS: ${{ github.event.inputs.min_age_days || '0' }}
MIN_SCORE: ${{ github.event.inputs.min_score || '4' }}
LIMIT: ${{ github.event.inputs.limit || '25' }}
run: |
set -euo pipefail
ARGS=(
--repo "${{ github.repository }}"
--min-age-days "${MIN_AGE_DAYS}"
--min-score "${MIN_SCORE}"
--limit "${LIMIT}"
)
if [ "${AGENT_SHIN_ENABLED:-false}" != "true" ]; then
echo "::notice::AGENT_SHIN_ENABLED is not 'true' -> forcing dry-run regardless of close input."
elif [ "${GITHUB_EVENT_NAME:-}" = "workflow_dispatch" ] && [ "${CLOSE_FLAG}" = "true" ]; then
ARGS+=(--close)
echo "::notice::Running in close-on-fail mode."
else
echo "::notice::AGENT_SHIN_ENABLED is true but this trigger is dry-run (scheduled event or close=false)."
fi
python3 .github/scripts/close_low_quality_prs.py "${ARGS[@]}"

90
.github/workflows/triage_issue_with_llm.yml vendored Normal file
View file

@ -0,0 +1,90 @@
name: Agent Shin — Issue triage
# LLM-as-judge triage for external GitHub issues.
#
# DRY-RUN BY DEFAULT. See .github/workflows/triage_pr_with_llm.yml for the
# enablement procedure — same repo variable (`AGENT_SHIN_ENABLED=true`)
# unlocks the PR and issue triage flows together.
on:
issues:
types: [opened, reopened]
workflow_dispatch:
inputs:
issue_number:
description: "Issue number to triage manually."
required: true
close:
description: "If true and AGENT_SHIN_ENABLED=true, actually close on fail."
required: false
default: "false"
type: choice
options:
- "true"
- "false"
permissions:
contents: read
issues: write
jobs:
triage:
if: github.repository == 'BerriAI/litellm'
runs-on: ubuntu-latest
steps:
- name: Checkout triage script
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
sparse-checkout: .github/scripts
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"
- name: Install LLM client
run: pip install --no-cache-dir "openai>=1.40.0"
- name: Run Agent Shin
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENAI_BASE_URL: ${{ vars.OPENAI_BASE_URL }}
TRIAGE_MODEL: ${{ vars.TRIAGE_MODEL }}
AGENT_SHIN_ENABLED: ${{ vars.AGENT_SHIN_ENABLED }}
DISPATCH_CLOSE: ${{ github.event.inputs.close }}
ISSUE_NUMBER: ${{ github.event.issue.number || github.event.inputs.issue_number }}
run: |
set -euo pipefail
ARGS=(--repo "${{ github.repository }}" --issue "${ISSUE_NUMBER}")
# Fail-safe gating: only the EXACT string "true" enables the
# destructive --close path. The workflow_dispatch input is a
# `choice` dropdown of "true"/"false" so the UI is constrained,
# but the API (`gh workflow run -f close=...`) accepts any
# string, and a `!= "false"` check would treat "True", "yes",
# "1", "TRUE", typos, and accidental whitespace as enabling
# closure. Mirror the Greptile closer's `= "true"` pattern.
if [ "${AGENT_SHIN_ENABLED:-false}" = "true" ] && [ "${DISPATCH_CLOSE:-false}" = "true" ]; then
ARGS+=(--close)
echo "::notice::Agent Shin is ENABLED and running in close-on-fail mode."
elif [ "${AGENT_SHIN_ENABLED:-false}" = "true" ]; then
echo "::notice::Agent Shin is ENABLED but this trigger is dry-run (workflow_dispatch close != 'true')."
else
echo "::notice::Agent Shin is in DRY-RUN mode (AGENT_SHIN_ENABLED is not 'true'). No comments will be posted; no issues will be closed."
fi
# Automatic `issues` events stay dry-run regardless until the team
# explicitly invokes workflow_dispatch with close=true.
if [ "${GITHUB_EVENT_NAME:-}" = "issues" ]; then
# filter out --close rather than substituting to "" (which would
# leave an empty positional arg that argparse rejects)
FILTERED=()
for arg in "${ARGS[@]}"; do
if [ "${arg}" != "--close" ]; then
FILTERED+=("${arg}")
fi
done
ARGS=("${FILTERED[@]}")
echo "::notice::issues trigger -> forcing dry-run."
fi
python3 .github/scripts/triage_with_llm.py "${ARGS[@]}"

104
.github/workflows/triage_pr_with_llm.yml vendored Normal file
View file

@ -0,0 +1,104 @@
name: Agent Shin — PR triage
# LLM-as-judge triage for external pull requests.
#
# DRY-RUN BY DEFAULT. Closures and public comments are gated on the repo
# variable `AGENT_SHIN_ENABLED` being set to the string `"true"`. Until then,
# every run only writes its verdict to the workflow step summary so the team
# can QA the judge's decisions before flipping it on.
#
# To enable for real:
# 1. Add a repo secret `OPENAI_API_KEY` (or compatible).
# 2. Set repo variable `AGENT_SHIN_ENABLED` to `true`
# (Settings > Secrets and variables > Actions > Variables).
#
# We use `pull_request_target` so the workflow has access to repo secrets
# and runs against PRs from forks. We never check out fork code — only read
# PR metadata via `gh api`, so this is safe.
on:
pull_request_target:
types: [opened, reopened]
workflow_dispatch:
inputs:
pr_number:
description: "PR number to triage manually."
required: true
close:
description: "If true and AGENT_SHIN_ENABLED=true, actually close on fail."
required: false
default: "false"
type: choice
options:
- "true"
- "false"
permissions:
contents: read
issues: write
pull-requests: write
jobs:
triage:
if: github.repository == 'BerriAI/litellm'
runs-on: ubuntu-latest
steps:
- name: Checkout triage script
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
sparse-checkout: .github/scripts
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"
- name: Install LLM client
run: pip install --no-cache-dir "openai>=1.40.0"
- name: Run Agent Shin
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENAI_BASE_URL: ${{ vars.OPENAI_BASE_URL }}
TRIAGE_MODEL: ${{ vars.TRIAGE_MODEL }}
AGENT_SHIN_ENABLED: ${{ vars.AGENT_SHIN_ENABLED }}
DISPATCH_CLOSE: ${{ github.event.inputs.close }}
PR_NUMBER: ${{ github.event.pull_request.number || github.event.inputs.pr_number }}
run: |
set -euo pipefail
ARGS=(--repo "${{ github.repository }}" --pr "${PR_NUMBER}")
# Fail-safe gating: only the EXACT string "true" enables the
# destructive --close path. The workflow_dispatch input is a
# `choice` dropdown of "true"/"false" so the UI is constrained,
# but the API (`gh workflow run -f close=...`) accepts any
# string, and a `!= "false"` check would treat "True", "yes",
# "1", "TRUE", typos, and accidental whitespace as enabling
# closure. Mirror the Greptile closer's `= "true"` pattern.
if [ "${AGENT_SHIN_ENABLED:-false}" = "true" ] && [ "${DISPATCH_CLOSE:-false}" = "true" ]; then
ARGS+=(--close)
echo "::notice::Agent Shin is ENABLED and running in close-on-fail mode."
elif [ "${AGENT_SHIN_ENABLED:-false}" = "true" ]; then
echo "::notice::Agent Shin is ENABLED but this trigger is dry-run (workflow_dispatch close != 'true' or scheduled event)."
else
echo "::notice::Agent Shin is in DRY-RUN mode (AGENT_SHIN_ENABLED is not 'true'). No comments will be posted; no PRs will be closed."
fi
# On the scheduled/automatic pull_request_target trigger we default to
# dry-run regardless, so the team can review verdicts in the step
# summary before any contributor sees a comment. Only the manual
# workflow_dispatch path (with close=true) closes PRs.
if [ "${GITHUB_EVENT_NAME:-}" = "pull_request_target" ]; then
# strip any --close added above (filter out, don't substitute
# to empty string — that would leave a stray "" positional arg
# that argparse rejects)
FILTERED=()
for arg in "${ARGS[@]}"; do
if [ "${arg}" != "--close" ]; then
FILTERED+=("${arg}")
fi
done
ARGS=("${FILTERED[@]}")
echo "::notice::pull_request_target trigger -> forcing dry-run."
fi
python3 .github/scripts/triage_with_llm.py "${ARGS[@]}"

130
.github/workflows/triage_reconsider.yml vendored Normal file
View file

@ -0,0 +1,130 @@
name: Agent Shin — reconsider
# Comment-trigger workflow: when the PR/issue author (or an internal
# collaborator) comments `@agent-shin reconsider` on a CLOSED PR/issue,
# Agent Shin re-runs LLM-judge triage on the current title+body and:
#
# - on PASS: posts a "re-evaluated and reopened" comment + reopens.
# - on FAIL: posts a "still missing X" comment and leaves it closed,
# so the contributor can iterate again.
#
# This exists because GitHub does NOT let an external (non-write-access)
# OSS contributor reopen a PR/issue closed by a bot or maintainer. Without
# this comment trigger, a contributor whose PR Agent Shin auto-closed
# would have no path back into the review queue except opening a fresh PR
# (which loses the original PR's history). The bot, on the other hand,
# has write access via GH_TOKEN and can reopen on their behalf.
#
# DRY-RUN BY DEFAULT — gated on `vars.AGENT_SHIN_ENABLED == 'true'` just
# like the other Agent Shin workflows. The workflow also gates on the
# commenter being either the PR/issue author or an internal collaborator
# (OWNER/MEMBER/COLLABORATOR) so random commenters cannot DOS the LLM
# judge or force a reopen.
on:
issue_comment:
types: [created]
permissions:
contents: read
issues: write
pull-requests: write
jobs:
reconsider:
if: |
github.repository == 'BerriAI/litellm'
&& contains(github.event.comment.body, '@agent-shin reconsider')
runs-on: ubuntu-latest
steps:
- name: Authorize commenter
# Only the PR/issue author OR an internal collaborator may trigger
# a reconsider. Outside random commenters could otherwise spam the
# phrase to burn LLM budget or, if a fail-open bug were ever
# introduced, force a reopen on someone else's behalf.
#
# We expose the authorization decision as a step output and gate
# every subsequent (potentially destructive) step on it. A `run:`
# step with `exit 0` would NOT stop the job — only `if:` gating
# on a known-true output is safe here.
id: auth
env:
COMMENTER: ${{ github.event.comment.user.login }}
AUTHOR: ${{ github.event.issue.user.login }}
ASSOCIATION: ${{ github.event.comment.author_association }}
run: |
set -euo pipefail
if [ "${COMMENTER}" = "${AUTHOR}" ]; then
echo "::notice::Authorized: commenter is the PR/issue author."
echo "authorized=true" >> "$GITHUB_OUTPUT"
exit 0
fi
case "${ASSOCIATION}" in
OWNER|MEMBER|COLLABORATOR)
echo "::notice::Authorized: commenter is an internal collaborator (${ASSOCIATION})."
echo "authorized=true" >> "$GITHUB_OUTPUT"
;;
*)
echo "::notice::Commenter '${COMMENTER}' (${ASSOCIATION}) is not authorized to trigger reconsider; skipping subsequent steps."
echo "authorized=false" >> "$GITHUB_OUTPUT"
;;
esac
- name: Checkout triage script
if: steps.auth.outputs.authorized == 'true'
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
with:
sparse-checkout: .github/scripts
persist-credentials: false
- name: Set up Python
if: steps.auth.outputs.authorized == 'true'
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: "3.12"
- name: Install LLM client
if: steps.auth.outputs.authorized == 'true'
run: pip install --no-cache-dir "openai>=1.40.0"
- name: Run Agent Shin reconsider
if: steps.auth.outputs.authorized == 'true'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENAI_BASE_URL: ${{ vars.OPENAI_BASE_URL }}
TRIAGE_MODEL: ${{ vars.TRIAGE_MODEL }}
AGENT_SHIN_ENABLED: ${{ vars.AGENT_SHIN_ENABLED }}
# `issue_comment` events fire for both issues and PR comments.
# `issue.pull_request` is set iff this is a PR comment, so we use
# its presence to decide whether to invoke `--pr N` or `--issue N`.
IS_PR: ${{ github.event.issue.pull_request != null }}
NUMBER: ${{ github.event.issue.number }}
run: |
set -euo pipefail
if [ "${IS_PR}" = "true" ]; then
ARGS=(--repo "${{ github.repository }}" --pr "${NUMBER}" --reconsider)
else
ARGS=(--repo "${{ github.repository }}" --issue "${NUMBER}" --reconsider)
fi
# Reconsider's destructive actions (post comment + reopen) are
# gated on `--close`, mirroring the regular triage workflows.
# When AGENT_SHIN_ENABLED is not the EXACT string "true", we
# still run the script so its verdict + would-X action lands in
# the step summary for QA — but without `--close`, the script
# returns `would-reopen` / `would-reconsider-still-failing`
# instead of touching GitHub state.
#
# Use the positive `= "true"` gate (not `!= "true" -> exit`) so
# the workflow guardrails in
# tests/test_litellm/test_github_triage_workflows.py see the
# canonical fail-safe enable pattern. Unknown values like
# "True", "yes", "1", or typos fall through to the dry-run
# branch, which is the safe default.
if [ "${AGENT_SHIN_ENABLED:-false}" = "true" ]; then
ARGS+=(--close)
echo "::notice::Agent Shin reconsider ENABLED — running real triage (close=true)."
else
echo "::notice::AGENT_SHIN_ENABLED is not 'true' -> reconsider stays in dry-run (no comment, no reopen)."
fi
python3 .github/scripts/triage_with_llm.py "${ARGS[@]}"

655
tests/test_litellm/test_github_close_low_quality_prs.py Normal file
View file

@ -0,0 +1,655 @@
"""Unit tests for `.github/scripts/close_low_quality_prs.py`.
These exercise the pure logic (score extraction and per-PR evaluation) without
hitting GitHub. Network/CLI calls are stubbed via monkeypatch.
"""
from __future__ import annotations
import datetime as dt
import importlib.util
import sys
from pathlib import Path
import pytest
SCRIPT_PATH = (
Path(__file__).resolve().parents[2]
/ ".github"
/ "scripts"
/ "close_low_quality_prs.py"
)
@pytest.fixture(scope="module")
def closer_module():
"""Load the script as a module via its file path (it lives outside the package)."""
spec = importlib.util.spec_from_file_location("close_low_quality_prs", SCRIPT_PATH)
assert spec and spec.loader, f"Could not load spec for {SCRIPT_PATH}"
module = importlib.util.module_from_spec(spec)
sys.modules["close_low_quality_prs"] = module
spec.loader.exec_module(module)
return module
def _greptile_comment(
body: str,
updated_at: str = "2026-05-10T00:00:00Z",
login: str = "greptile-apps[bot]",
) -> dict:
return {
"user": {"login": login},
"body": body,
"created_at": updated_at,
"updated_at": updated_at,
}
class TestExtractGreptileScore:
def test_should_extract_score_from_html_header(self, closer_module):
comments = [
_greptile_comment("<h3>Confidence Score: 3/5</h3>\nSome body text.")
]
result = closer_module.extract_greptile_score(comments)
assert result is not None
score, _ = result
assert score == 3
def test_should_accept_both_greptile_login_variants(self, closer_module):
# REST API form ("greptile-apps[bot]") and GraphQL form ("greptile-apps")
for login in ("greptile-apps", "greptile-apps[bot]"):
comments = [
_greptile_comment("<h3>Confidence Score: 2/5</h3>", login=login)
]
result = closer_module.extract_greptile_score(comments)
assert result is not None, f"failed to detect score for login={login}"
score, _ = result
assert score == 2
def test_should_extract_score_from_plain_text(self, closer_module):
comments = [_greptile_comment("Confidence Score: 5/5 — looks good!")]
result = closer_module.extract_greptile_score(comments)
assert result is not None
score, _ = result
assert score == 5
def test_should_tolerate_whitespace_and_case(self, closer_module):
comments = [_greptile_comment("**confidence score : 2 / 5**")]
result = closer_module.extract_greptile_score(comments)
assert result is not None
score, _ = result
assert score == 2
def test_should_pick_most_recent_comment_when_rereview_happens(self, closer_module):
comments = [
_greptile_comment(
"Confidence Score: 2/5", updated_at="2026-05-01T00:00:00Z"
),
_greptile_comment(
"Confidence Score: 5/5", updated_at="2026-05-12T00:00:00Z"
),
]
result = closer_module.extract_greptile_score(comments)
assert result is not None
score, _ = result
assert score == 5
def test_should_ignore_non_greptile_authors(self, closer_module):
comments = [
{
"user": {"login": "some-human"},
"body": "Confidence Score: 1/5",
"created_at": "2026-05-12T00:00:00Z",
"updated_at": "2026-05-12T00:00:00Z",
}
]
assert closer_module.extract_greptile_score(comments) is None
def test_should_return_none_when_no_score_present(self, closer_module):
comments = [_greptile_comment("Greptile summary without a score.")]
assert closer_module.extract_greptile_score(comments) is None
def test_should_return_none_for_empty_comments(self, closer_module):
assert closer_module.extract_greptile_score([]) is None
class TestEvaluatePr:
@pytest.fixture(autouse=True)
def _now(self):
return dt.datetime(2026, 5, 17, tzinfo=dt.timezone.utc)
def _make_pr(
self,
*,
number: int = 1,
created_days_ago: int = 10,
is_draft: bool = False,
labels: list[str] | None = None,
author_login: str = "someone",
) -> dict:
created = dt.datetime(2026, 5, 17, tzinfo=dt.timezone.utc) - dt.timedelta(
days=created_days_ago
)
return {
"number": number,
"title": f"PR #{number}",
"createdAt": created.isoformat().replace("+00:00", "Z"),
"isDraft": is_draft,
"labels": [{"name": lbl} for lbl in (labels or [])],
"author": {"login": author_login},
"url": f"https://example.com/pr/{number}",
}
@pytest.fixture(autouse=True)
def _external_author(self, closer_module, monkeypatch):
"""Treat every test PR as external unless overridden."""
monkeypatch.setattr(
closer_module, "is_external_pr_author", lambda pr, repo: True
)
def test_should_warn_drafts_when_score_low_first_time(
self, closer_module, _now, monkeypatch
):
# Drafts are NOT a free pass — the open-PR queue should reflect any
# PR that needs human attention regardless of draft status. Authors
# who need a long-lived draft can use the `wip` opt-out label.
# First run: warn the contributor (1-day grace), don't close yet.
monkeypatch.setattr(
closer_module,
"fetch_pr_comments",
lambda *a, **kw: [_greptile_comment("Confidence Score: 2/5")],
)
action, score, age = closer_module.evaluate_pr(
self._make_pr(is_draft=True, created_days_ago=0),
now=_now,
min_age_days=0,
min_score=4,
repo=None,
optout_labels=set(),
)
assert action == "warn-grace"
assert score == 2 and age == 0
def test_should_warn_brand_new_pr_when_min_age_zero(
self, closer_module, _now, monkeypatch
):
# `min_age_days=0` means no age filter — a freshly-opened PR is
# eligible the moment Greptile scores it below threshold. The
# first detection still goes through the warn-grace step rather
# than closing immediately, giving the contributor 24 hours to
# respond before the next run actually closes the PR.
monkeypatch.setattr(
closer_module,
"fetch_pr_comments",
lambda *a, **kw: [_greptile_comment("Confidence Score: 1/5")],
)
action, score, age = closer_module.evaluate_pr(
self._make_pr(created_days_ago=0),
now=_now,
min_age_days=0,
min_score=4,
repo=None,
optout_labels=set(),
)
assert action == "warn-grace"
assert score == 1 and age == 0
def test_should_skip_optout_label_case_insensitive(
self, closer_module, _now, monkeypatch
):
monkeypatch.setattr(
closer_module,
"fetch_pr_comments",
lambda *a, **kw: pytest.fail("should not fetch comments for opt-outs"),
)
action, _, _ = closer_module.evaluate_pr(
self._make_pr(labels=["WIP"]),
now=_now,
min_age_days=7,
min_score=4,
repo=None,
optout_labels={"wip"},
)
assert action == "skip-optout-label"
def test_should_skip_too_young_when_min_age_set(
self, closer_module, _now, monkeypatch
):
# The min-age-days flag is now opt-in (default 0). When a maintainer
# explicitly passes a positive value (e.g. for a backfill run that
# wants to spare brand-new PRs), the skip-too-young path still works.
monkeypatch.setattr(
closer_module,
"fetch_pr_comments",
lambda *a, **kw: pytest.fail("should not fetch comments for young PRs"),
)
action, _, age = closer_module.evaluate_pr(
self._make_pr(created_days_ago=2),
now=_now,
min_age_days=7,
min_score=4,
repo=None,
optout_labels=set(),
)
assert action == "skip-too-young"
assert age == 2
def test_should_not_skip_when_min_age_is_zero(
self, closer_module, _now, monkeypatch
):
# With the new default min_age_days=0, even a 0-day-old PR is
# evaluated. This test pins that behavior so future refactors don't
# silently restore an age filter.
monkeypatch.setattr(
closer_module,
"fetch_pr_comments",
lambda *a, **kw: [_greptile_comment("Confidence Score: 5/5")],
)
action, score, age = closer_module.evaluate_pr(
self._make_pr(created_days_ago=0),
now=_now,
min_age_days=0,
min_score=4,
repo=None,
optout_labels=set(),
)
assert action == "skip-score-ok"
assert score == 5 and age == 0
def test_should_skip_when_greptile_has_not_reviewed(
self, closer_module, _now, monkeypatch
):
monkeypatch.setattr(closer_module, "fetch_pr_comments", lambda *a, **kw: [])
action, score, age = closer_module.evaluate_pr(
self._make_pr(created_days_ago=10),
now=_now,
min_age_days=7,
min_score=4,
repo=None,
optout_labels=set(),
)
assert action == "skip-no-greptile-score"
assert score is None and age == 10
def test_should_skip_when_score_meets_threshold(
self, closer_module, _now, monkeypatch
):
monkeypatch.setattr(
closer_module,
"fetch_pr_comments",
lambda *a, **kw: [_greptile_comment("Confidence Score: 4/5")],
)
action, score, age = closer_module.evaluate_pr(
self._make_pr(created_days_ago=10),
now=_now,
min_age_days=7,
min_score=4,
repo=None,
optout_labels=set(),
)
assert action == "skip-score-ok"
assert score == 4 and age == 10
def test_should_warn_when_old_and_low_score_no_prior_warning(
self, closer_module, _now, monkeypatch
):
# Even an old PR that still has no grace warning gets one on the
# first eligible run — the daily cron is the natural cadence, so
# an existing-but-never-warned PR enters the grace flow normally.
monkeypatch.setattr(
closer_module,
"fetch_pr_comments",
lambda *a, **kw: [_greptile_comment("Confidence Score: 3/5")],
)
action, score, age = closer_module.evaluate_pr(
self._make_pr(created_days_ago=10),
now=_now,
min_age_days=7,
min_score=4,
repo=None,
optout_labels=set(),
)
assert action == "warn-grace"
assert score == 3 and age == 10
def test_should_close_when_grace_warning_aged_out_and_score_still_low(
self, closer_module, _now, monkeypatch
):
# Day-1 the closer posted a warning. Day-2 the PR still scores <4
# AND the warning is older than `GRACE_PERIOD_SECONDS`, so the
# action flips to `close`. This is the "grace expired" path.
old_warning = {
"user": {"login": "github-actions[bot]"},
"body": (
"you have 1 day to fix this\n\n" + closer_module.GRACE_COMMENT_MARKER
),
"created_at": (
_now - dt.timedelta(seconds=closer_module.GRACE_PERIOD_SECONDS + 60)
)
.isoformat()
.replace("+00:00", "Z"),
"updated_at": "2026-05-15T00:00:00Z",
}
monkeypatch.setattr(
closer_module,
"fetch_pr_comments",
lambda *a, **kw: [
_greptile_comment(
"<h3>Confidence Score: 1/5</h3>",
updated_at="2026-05-15T00:00:00Z",
),
old_warning,
],
)
action, score, _ = closer_module.evaluate_pr(
self._make_pr(created_days_ago=14),
now=_now,
min_age_days=7,
min_score=4,
repo=None,
optout_labels=set(),
)
assert action == "close"
assert score == 1
def test_should_skip_when_grace_warning_within_window(
self, closer_module, _now, monkeypatch
):
# Within the 24-hour grace window the closer must NOT close the
# PR even if the score is still low. The warning is only an hour
# old; give the contributor time to push fixes before destruction.
recent_warning = {
"user": {"login": "github-actions[bot]"},
"body": "warning text\n\n" + closer_module.GRACE_COMMENT_MARKER,
"created_at": (_now - dt.timedelta(hours=1))
.isoformat()
.replace("+00:00", "Z"),
}
monkeypatch.setattr(
closer_module,
"fetch_pr_comments",
lambda *a, **kw: [
_greptile_comment("Confidence Score: 2/5"),
recent_warning,
],
)
action, score, _ = closer_module.evaluate_pr(
self._make_pr(created_days_ago=10),
now=_now,
min_age_days=0,
min_score=4,
repo=None,
optout_labels=set(),
)
assert action == "skip-in-grace-period"
assert score == 2
def test_should_close_immediately_for_swiftwinds_login(
self, closer_module, _now, monkeypatch
):
# SwiftWinds is in `IMMEDIATE_CLOSE_LOGINS` for dogfooding the bot
# from an external account. Skip grace; close on first detection
# so the iteration loop is fast.
monkeypatch.setattr(
closer_module,
"fetch_pr_comments",
lambda *a, **kw: [_greptile_comment("Confidence Score: 1/5")],
)
action, score, _ = closer_module.evaluate_pr(
self._make_pr(created_days_ago=0, author_login="SwiftWinds"),
now=_now,
min_age_days=0,
min_score=4,
repo=None,
optout_labels=set(),
)
assert action == "close"
assert score == 1
def test_should_close_immediately_for_swiftwinds_login_case_insensitive(
self, closer_module, _now, monkeypatch
):
# GitHub login matching is case-insensitive on GitHub's side; the
# API returns the original casing. Make sure the bypass fires
# regardless of how the login was registered.
monkeypatch.setattr(
closer_module,
"fetch_pr_comments",
lambda *a, **kw: [_greptile_comment("Confidence Score: 1/5")],
)
for login in ("SwiftWinds", "swiftwinds", "SWIFTWINDS"):
action, _, _ = closer_module.evaluate_pr(
self._make_pr(created_days_ago=0, author_login=login),
now=_now,
min_age_days=0,
min_score=4,
repo=None,
optout_labels=set(),
)
assert action == "close", login
def test_should_skip_internal_authors(self, closer_module, _now, monkeypatch):
# Override the fixture for this one test.
monkeypatch.setattr(
closer_module, "is_external_pr_author", lambda pr, repo: False
)
monkeypatch.setattr(
closer_module,
"fetch_pr_comments",
lambda *a, **kw: pytest.fail("should not fetch comments for internal"),
)
action, score, _ = closer_module.evaluate_pr(
self._make_pr(created_days_ago=14),
now=_now,
min_age_days=7,
min_score=4,
repo=None,
optout_labels=set(),
)
assert action == "skip-internal"
assert score is None
class TestMainOptoutLabelDefault:
"""`--optout-label` must REPLACE the canonical defaults, not append."""
def _patch_no_op(self, closer_module, monkeypatch):
monkeypatch.setattr(closer_module, "fetch_open_prs", lambda repo: [])
# `optout_labels` is captured indirectly via evaluate_pr; sniff the
# set passed in by stubbing evaluate_pr.
captured: dict = {}
def fake_evaluate(pr, now, min_age_days, min_score, repo, optout_labels):
captured["optout_labels"] = set(optout_labels)
return ("skip-internal", None, None)
monkeypatch.setattr(closer_module, "evaluate_pr", fake_evaluate)
return captured
def test_should_use_canonical_defaults_when_flag_omitted(
self, closer_module, monkeypatch
):
captured = self._patch_no_op(closer_module, monkeypatch)
# No PRs -> capture won't fire; instead inject one synthetic PR via
# fetch_open_prs so evaluate_pr is invoked at least once.
monkeypatch.setattr(
closer_module,
"fetch_open_prs",
lambda repo: [
{
"number": 1,
"title": "p",
"createdAt": "2026-05-10T00:00:00Z",
"isDraft": True,
"labels": [],
"author": {"login": "x"},
}
],
)
monkeypatch.setattr(sys, "argv", ["close_low_quality_prs.py"])
rc = closer_module.main()
assert rc == 0
assert captured["optout_labels"] == set(closer_module.DEFAULT_OPTOUT_LABELS)
def test_should_replace_defaults_when_flag_provided(
self, closer_module, monkeypatch
):
captured = self._patch_no_op(closer_module, monkeypatch)
monkeypatch.setattr(
closer_module,
"fetch_open_prs",
lambda repo: [
{
"number": 1,
"title": "p",
"createdAt": "2026-05-10T00:00:00Z",
"isDraft": True,
"labels": [],
"author": {"login": "x"},
}
],
)
monkeypatch.setattr(
sys,
"argv",
[
"close_low_quality_prs.py",
"--optout-label",
"hold",
"--optout-label",
"needs-discussion",
],
)
rc = closer_module.main()
assert rc == 0
# Crucially, none of the canonical defaults leak in.
assert captured["optout_labels"] == {"hold", "needs-discussion"}
for default in closer_module.DEFAULT_OPTOUT_LABELS:
assert default not in captured["optout_labels"], default
class TestSecondsSinceLastGraceWarning:
"""Grace-period detection: only counts comments by the bot identity
that contain the shared `GRACE_COMMENT_MARKER`."""
def _make_marker_comment(
self,
closer_module,
*,
login: str = "github-actions[bot]",
created_at: str = "2026-05-16T00:00:00Z",
include_marker: bool = True,
) -> dict:
body = "warning text"
if include_marker:
body += "\n\n" + closer_module.GRACE_COMMENT_MARKER
return {
"user": {"login": login},
"body": body,
"created_at": created_at,
}
def test_should_return_none_when_no_marker_comment(self, closer_module):
comments = [
{
"user": {"login": "github-actions[bot]"},
"body": "Some other bot comment",
"created_at": "2026-05-16T00:00:00Z",
}
]
assert closer_module.seconds_since_last_grace_warning(comments) is None
def test_should_return_none_for_empty(self, closer_module):
assert closer_module.seconds_since_last_grace_warning([]) is None
def test_should_ignore_non_bot_comments_with_marker(self, closer_module):
# If a curious user quotes the marker in a comment, we must NOT
# treat it as a bot warning. The grace timer would then never fire.
comments = [
self._make_marker_comment(closer_module, login="random-user"),
]
assert closer_module.seconds_since_last_grace_warning(comments) is None
def test_should_pick_latest_marker_comment(self, closer_module):
# When multiple grace warnings exist (e.g. a re-open cycle), use
# the most recent one to compute the age.
comments = [
self._make_marker_comment(closer_module, created_at="2026-05-15T00:00:00Z"),
self._make_marker_comment(closer_module, created_at="2026-05-16T23:00:00Z"),
]
now = dt.datetime(2026, 5, 17, 0, 0, 0, tzinfo=dt.timezone.utc)
age = closer_module.seconds_since_last_grace_warning(comments, now=now)
# 1h = 3600s
assert age == 3600.0
class TestImmediateCloseLoginsConstant:
"""SwiftWinds is the dogfood account the user explicitly named — pin
its presence so a future cleanup that removes the constant or
forgets to keep the entry doesn't silently break the test path."""
def test_should_include_swiftwinds(self, closer_module):
assert "swiftwinds" in closer_module.IMMEDIATE_CLOSE_LOGINS
def test_should_be_lowercase_for_case_insensitive_match(self, closer_module):
for login in closer_module.IMMEDIATE_CLOSE_LOGINS:
assert login == login.lower(), login
class TestGraceWarningCommentText:
"""Pin the user-facing language in the grace warning comment so the
`1 day grace` and `@greptileai still works after close` promises
don't get accidentally dropped in a future refactor.
"""
def test_should_state_one_day_grace_period(self, closer_module):
body = closer_module.format_grace_warning_comment(score=2, threshold=4)
# The user's PR explicitly said "specify in the comment" — pin
# that the literal "1 day" appears in the comment.
assert "1 day" in body
def test_should_mention_agent_shin_reconsider(self, closer_module):
body = closer_module.format_grace_warning_comment(score=2, threshold=4)
assert "@agent-shin reconsider" in body
def test_should_promise_greptileai_works_after_close(self, closer_module):
body = closer_module.format_grace_warning_comment(score=2, threshold=4)
assert "@greptileai" in body
assert "even after the PR is closed" in body
def test_should_carry_grace_marker(self, closer_module):
# The marker is what `seconds_since_last_grace_warning` greps for
# to detect a prior warning — dropping it would silently break
# the cooldown.
body = closer_module.format_grace_warning_comment(score=2, threshold=4)
assert closer_module.GRACE_COMMENT_MARKER in body
def test_close_comment_should_mention_greptileai_post_close(self, closer_module):
# The actual close comment should ALSO point at the @greptileai
# post-close re-review path so contributors see the same options
# whether they read the warning or only catch the close comment.
# `close_pr` writes the close comment via `gh pr comment` — we
# don't easily call it directly here, but the comment body is
# constructed inline. Re-creating it via a no-op `gh` stub is
# awkward, so we assert against the same string template by
# asserting that the close path's text constant is updated.
# `close_pr` source must contain the marker text — guarded by
# this whole-module read-and-assert.
from pathlib import Path
source = Path(closer_module.__file__).read_text()
assert "even after the PR is closed" in source
assert "@greptileai" in source
class TestHasOptoutLabel:
def test_should_match_label_case_insensitively(self, closer_module):
pr = {"labels": [{"name": "Do Not Close"}, {"name": "bug"}]}
assert closer_module.has_optout_label(pr, {"do not close"}) is True
def test_should_return_false_when_no_match(self, closer_module):
pr = {"labels": [{"name": "bug"}, {"name": "enhancement"}]}
assert closer_module.has_optout_label(pr, {"wip", "keep open"}) is False
def test_should_handle_missing_labels(self, closer_module):
assert closer_module.has_optout_label({}, {"wip"}) is False

1715
tests/test_litellm/test_github_triage_with_llm.py Normal file
View file

File diff suppressed because it is too large Load diff

139
tests/test_litellm/test_github_triage_workflows.py Normal file
View file

@ -0,0 +1,139 @@
"""Static guardrails for the Agent Shin + Greptile workflow YAML files.
These workflows can post comments and close PRs/issues on
BerriAI/litellm, so the gating logic that decides "is this a real
close-on-fail run?" must fail-safe on any unexpected input. The risk
is mostly maintenance: someone edits the bash gate, drops a quote,
inverts a comparison, or uses `!= "false"` (which treats "True",
"yes", "1", and typos as enabling closure) and the regression isn't
caught until a real OSS contributor's PR gets auto-closed.
The tests below pin two invariants across every workflow that gates a
destructive `--close`:
1. The gate uses the fail-safe `= "true"` comparison — not `!= "false"`,
not `!= ""`. Only the literal string "true" should ever enable
closure.
2. The gate also requires `AGENT_SHIN_ENABLED = "true"` (or the
scheduled-job equivalent) — disabling the variable must always
force dry-run.
Static parsing of the YAML + bash text is the right level of test here:
the gating logic lives in a `run:` block, not in a Python module we can
import, and end-to-end testing a GitHub Actions workflow from CI is
infeasible. A YAML-level guardrail is exactly what would have caught
the original `!= "false"` regression at PR time.
"""
from __future__ import annotations
from pathlib import Path
import pytest
import yaml
REPO_ROOT = Path(__file__).resolve().parents[2]
WORKFLOWS_DIR = REPO_ROOT / ".github" / "workflows"
# Map of workflow file -> the env var name that drives the destructive
# gate inside that workflow's `run:` block. Keeping this table explicit
# (rather than scraping every workflow file) means a new workflow file
# that bypasses the dry-run gating doesn't silently slip past this test.
DESTRUCTIVE_GATE_ENV: dict[str, str] = {
"triage_pr_with_llm.yml": "DISPATCH_CLOSE",
"triage_issue_with_llm.yml": "DISPATCH_CLOSE",
"close_low_quality_prs.yml": "CLOSE_FLAG",
# The reconsider workflow has no per-run "really do it?" knob — its
# only kill switch is `AGENT_SHIN_ENABLED`, which already serves as
# both the destructive gate and the global enablement gate.
"triage_reconsider.yml": "AGENT_SHIN_ENABLED",
}
def _load_workflow(name: str) -> dict:
return yaml.safe_load((WORKFLOWS_DIR / name).read_text())
def _all_run_blocks(workflow: dict) -> list[str]:
"""Return every `run:` step's command text, joined."""
commands: list[str] = []
jobs = workflow.get("jobs") or {}
for job in jobs.values():
for step in job.get("steps", []) or []:
if not isinstance(step, dict):
continue
run = step.get("run")
if isinstance(run, str):
commands.append(run)
return commands
@pytest.mark.parametrize("workflow_file,env_var", sorted(DESTRUCTIVE_GATE_ENV.items()))
def test_should_use_failsafe_equals_true_comparison(
workflow_file: str, env_var: str
) -> None:
"""The destructive `--close` gate must use `= "true"` (fail-safe), not
`!= "false"` (which would treat "True", "yes", "1", or any typo as
enabling closure).
Both bare `${ENV_VAR}` and `${ENV_VAR:-false}` (with a default) are
accepted forms — what matters is the comparison operator. The
Greptile closer relies on an outer `AGENT_SHIN_ENABLED` gate so it
can use the bare form; the Agent Shin workflows include `:-false`
for defense in depth. Either is fine.
"""
workflow = _load_workflow(workflow_file)
text = "\n".join(_all_run_blocks(workflow))
assert env_var in text, (
f"{workflow_file} no longer references {env_var}; was the "
"gating env var renamed without updating this test?"
)
accepted_patterns = (
f'"${{{env_var}}}" = "true"',
f'"${{{env_var}:-false}}" = "true"',
)
assert any(p in text for p in accepted_patterns), (
f"{workflow_file} must gate the destructive --close flag on the "
f'EXACT string "true" (one of: {accepted_patterns!r}). Mirror '
'the Greptile closer pattern; do NOT use `!= "false"` which '
'fail-opens on unknown values like "True", "yes", "1", or typos.'
)
forbidden_patterns = (
f'"${{{env_var}}}" != "false"',
f'"${{{env_var}:-false}}" != "false"',
f'"${{{env_var}:-true}}" != "false"',
)
for forbidden in forbidden_patterns:
assert forbidden not in text, (
f"{workflow_file} uses the fail-open pattern {forbidden!r}. "
'Switch to `= "true"` so unknown values stay dry-run.'
)
@pytest.mark.parametrize("workflow_file", sorted(DESTRUCTIVE_GATE_ENV))
def test_should_require_agent_shin_enabled_for_close(workflow_file: str) -> None:
"""Every destructive gate must also gate on the global enablement
variable, so flipping `AGENT_SHIN_ENABLED` off is a kill switch
regardless of any per-run input.
Two patterns are equally fine:
- Positive: `[ "${AGENT_SHIN_ENABLED:-false}" = "true" ]` to enter
the close branch (Agent Shin workflows).
- Negative: `[ "${AGENT_SHIN_ENABLED:-false}" != "true" ]` then
bail out / force dry-run (Greptile closer).
What matters is that the comparison value is the literal "true";
`!= "false"` or `= "1"` etc. would not be a true kill switch.
"""
workflow = _load_workflow(workflow_file)
text = "\n".join(_all_run_blocks(workflow))
accepted_patterns = (
'"${AGENT_SHIN_ENABLED:-false}" = "true"',
'"${AGENT_SHIN_ENABLED:-false}" != "true"',
)
assert any(p in text for p in accepted_patterns), (
f"{workflow_file} must gate destructive actions on "
'`AGENT_SHIN_ENABLED = "true"` (or the inverted `!= "true"` '
"guard that forces dry-run). Without this, an unset repo "
"variable would not be treated as a kill switch."
)