fix(ci): make Check Duplicate Issues robust to Unicode separators in issue bodies

The 'Check Duplicate Issues' workflow has been failing with
  json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 2792467
whenever an open issue body contains a Unicode line/paragraph separator
(U+2028 / U+2029).

Root cause: fetch_open_issues() called 'gh api --paginate' and then
str.splitlines() on the raw output. gh concatenates JSON pages without
any separator (...][...), so the entire output is one logical line --
unless a U+2028/U+2029 character inside a string body trips splitlines(),
which splits mid-JSON-string and yields an unterminated array fragment.

Switch to 'gh issue list --json number,title --state open --limit 100000'
which returns a single well-formed JSON array and already excludes pull
requests, removing both pitfalls in one go.

Verified locally: the script now fetches 1355 open issues and runs the
single-issue and --scan modes without errors.

Co-authored-by: Krrish Dholakia <krrish-berri-2@users.noreply.github.com>
This commit is contained in:
Cursor Agent 2026-05-21 03:10:40 +00:00
parent 37ef8d9059
commit c9adb3f72c
No known key found for this signature in database

View file

@ -40,30 +40,36 @@ def gh(*args: str) -> str:
def fetch_open_issues(repo: str | None) -> list[dict]:
"""Fetch all open issues (excluding PRs) via gh api --paginate."""
"""Fetch all open issues (excluding PRs) via `gh issue list --json`.
Using `gh issue list` instead of `gh api --paginate` because:
* it returns a single, well-formed JSON array (the `--paginate` output
concatenates pages as `...][...` with no separator, which is not valid
JSON on its own), and
* it already excludes pull requests, so we don't need a post-filter.
A previous implementation used `gh api --paginate` and tried to parse each
line with `json.loads`, but Python's ``str.splitlines`` also splits on
Unicode line/paragraph separators (``\\u2028``/``\\u2029``) that are valid
inside JSON strings. Issues whose body contains those characters then
caused ``Unterminated string`` errors mid-page (see the failure in
``Check Duplicate Issues`` CI runs).
"""
cmd = [
"issue",
"list",
"--state",
"open",
"--limit",
"100000",
"--json",
"number,title",
]
if repo:
endpoint = (
f"repos/{repo}/issues?state=open&per_page=100&sort=created&direction=asc"
)
else:
endpoint = "repos/{owner}/{repo}/issues?state=open&per_page=100&sort=created&direction=asc"
cmd = ["api", "--paginate", endpoint]
cmd.extend(["--repo", repo])
raw = gh(*cmd)
# gh --paginate concatenates JSON arrays, so we may get multiple arrays
issues = []
for line in raw.strip().splitlines():
line = line.strip()
if not line:
continue
parsed = json.loads(line)
if isinstance(parsed, list):
issues.extend(parsed)
else:
issues.append(parsed)
# Filter out pull requests (they also appear in the issues endpoint)
return [i for i in issues if "pull_request" not in i]
return json.loads(raw)
def close_as_duplicate(