scriptoriumadmin/GitNexus
1
0
Fork 0
mirror of https://github.com/abhigyanpatwari/GitNexus.git synced 2026-09-14 23:22:54 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 7

Merge branch 'main' into dependabot/github_actions/astral-sh/setup-uv-9.0.0

Browse source
This commit is contained in:
Gergő Magyar 2026-09-12 15:30:53 +01:00 • committed by GitHub
commit 47f60352c8
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1625 changed files with 1159356 additions and 15838 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.agents/plugins/marketplace.json
View file

@ -6,7 +6,7 @@
"plugins": [
{
"name": "gitnexus",
"version": "1.6.9",
"version": "1.6.11",
"source": {
"source": "local",
"path": "./gitnexus-claude-plugin"

2
.claude-plugin/marketplace.json
View file

@ -11,7 +11,7 @@
"plugins": [
{
"name": "gitnexus",
"version": "1.6.9",
"version": "1.6.11",
"source": "./gitnexus-claude-plugin",
"description": "Code intelligence powered by a knowledge graph. Provides execution flow tracing, blast radius analysis, and augmented search across your codebase."
}

24
.claude/skills/gitnexus-cli/SKILL.md
View file

@ -5,9 +5,9 @@ description: "Use when the user needs to run GitNexus CLI commands like analyze/
# GitNexus CLI Commands
Commands below use `node .gitnexus/run.cjs <command>` — the project-local runner `gitnexus analyze` drops next to the index. It auto-selects an available runner at call time (global `gitnexus`, else `pnpm dlx`, else `npx`), so no package-manager assumption and no global install is required.
Commands below use `node .gitnexus/run.cjs <command>` — the project-local runner `gitnexus analyze` drops next to the index. It auto-selects an available runner at call time (global `gitnexus`, else `pnpm dlx`, else `bunx`, else `npx`), so no package-manager assumption and no global install is required — including on a bun-only machine, which has no npm, npx or pnpm at all.
> **Not analyzed yet, or `node .gitnexus/run.cjs` reports `Cannot find module`** (the gitignored runner is absent — e.g. a fresh clone or `git clean`)? (Re)generate it with `npx gitnexus analyze` from the project root. On **npm 11.x**, if `npx` crashes during install (`node.target is null`), install once with `npm i -g gitnexus` (then `gitnexus analyze`) or use `pnpm --allow-build=@ladybugdb/core --allow-build=gitnexus --allow-build=tree-sitter dlx gitnexus@latest analyze`. See [#1939](https://github.com/abhigyanpatwari/GitNexus/issues/1939).
> **Not analyzed yet, or `node .gitnexus/run.cjs` reports `Cannot find module`** (the gitignored runner is absent — e.g. a fresh clone or `git clean`)? (Re)generate it with `npx gitnexus analyze` from the project root, or `bunx gitnexus@latest analyze` on a bun-only machine. On **npm 11.x**, if `npx` crashes during install (`node.target is null`), install once with `npm i -g gitnexus` (then `gitnexus analyze`), or use `bunx gitnexus@latest analyze`, or `pnpm --allow-build=@ladybugdb/core --allow-build=gitnexus --allow-build=tree-sitter dlx gitnexus@latest analyze`. See [#1939](https://github.com/abhigyanpatwari/GitNexus/issues/1939).
## Commands
@ -21,13 +21,21 @@ Run from the project root. This parses all source files, builds the knowledge gr
| Flag | Effect |
| -------------- | ---------------------------------------------------------------- |
| `--watch` | Keep a Git repository index current with serialized refreshes |
| `--debounce <ms>` | Watch quiet period before refresh (default: 300 ms) |
| `--force` | Force full re-index even if up to date |
| `--embeddings` | Enable embedding generation for semantic search (off by default) |
| `--drop-embeddings` | Drop existing embeddings on rebuild. By default, an `analyze` without `--embeddings` preserves them. |
| `--pdg` | Build the program-dependence layers used by `explain` and `pdg_query` (taint, CDG, and REACHING_DEF). |
| `--spring-actuator <path>` | Import opt-in Spring Boot Actuator mappings, beans, conditions, configprops, and env snapshots. Forces a full rebuild; unsupported with `--watch`. |
| `--asyncapi-spec <path>` | Read opt-in AsyncAPI 3.x documents (directory or single file) and mint `Destination` nodes from their operations. 2.x is refused, not mapped. Unsupported with `--watch`. |
**When to run:** First time in a project, after major code changes, or when `gitnexus://repo/{name}/context` reports the index is stale. In Claude Code, a PostToolUse hook detects staleness after `git commit` and `git merge` and notifies the agent to run `analyze` — the hook does not run analyze itself, to avoid blocking the agent for up to 120s and risking KuzuDB corruption on timeout.
For Spring runtime enrichment, pass a JSON bundle, one endpoint JSON file, or a directory containing endpoint files. Route evidence is authoritative only when `runtimeConfirmed === true`; `runtimeSource` records provenance and may also accompany `handler-conflict`. Env/configprops values are never persisted.
Use `node .gitnexus/run.cjs analyze --watch` for a long-lived local Git repository. It performs an initial analysis, queues scanner-admitted file changes, and retries intact failed batches with bounded backoff. Watch refreshes update only the graph: they skip AGENTS.md / CLAUDE.md injection and standard skill installation, so run a one-shot `analyze` when those generated files need updating. Watch rejects one-shot or context-output flags including `--force`, embedding flags, `--skills`, `--default-branch`, `--skip-agents-md`, `--skip-skills`, `--no-stats`, `--self-commit`, `--index-only`, and `--skip-git`. It never pulls remotes. Scheduled remote clone/pull is a different command: `gitnexus auto-sync`. Bare `gitnexus watch` is reserved and does not start either job. Running MCP and `serve` processes periodically check for a published replacement and reopen it without a restart. MCP checks are throttled to once every five seconds, so a tool call before the next check can briefly use the previous index.
### status — Check index freshness
```bash
@ -55,15 +63,19 @@ Deletes the `.gitnexus/` directory and unregisters the repo from the global regi
node .gitnexus/run.cjs wiki
```
Generates repository documentation from the knowledge graph using an LLM. Requires an API key (saved to `~/.gitnexus/config.json` on first use).
Generates repository documentation from the knowledge graph using an LLM. HTTP providers require an API key (saved to `~/.gitnexus/config.json` on first use). Local CLI providers (`--provider cursor|claude|codex|opencode|grok`) use your existing CLI login.
| Flag | Effect |
| ------------------- | ----------------------------------------- |
| `--force` | Force full regeneration |
| `--model <model>` | LLM model (default: minimax/minimax-m2.5) |
| `--force` | Force full regeneration, also required to re-generate an existing wiki in a different language |
| `--provider <name>` | LLM provider: minimax, openai, openrouter, azure, custom, cursor, claude, codex, opencode, or grok (default: minimax). Local CLIs (`cursor`, `claude`, `codex`, `opencode`, `grok`) use your existing CLI login and skip `--api-key`. |
| `--model <model>` | LLM model (default: MiniMax-M3) |
| `--base-url <url>` | LLM API base URL |
| `--api-key <key>` | LLM API key |
| `--concurrency <n>` | Parallel LLM calls (default: 3) |
| `--timeout <seconds>` | LLM request timeout in seconds (default: disabled) |
| `--retries <n>` | Max LLM retry attempts per request (default: 3) |
| `--lang <lang>` | Output language for generated documentation (e.g. english, chinese, spanish, japanese) |
| `--gist` | Publish wiki as a public GitHub Gist |
### list — Show all indexed repos
@ -82,5 +94,5 @@ Lists all repositories registered in `~/.gitnexus/registry.json`. The MCP `list_
## Troubleshooting
- **"Not inside a git repository"**: Run from a directory inside a git repo
- **Index is stale after re-analyzing**: Restart Claude Code to reload the MCP server
- **Index is stale after re-analyzing**: Wait for the next MCP tool call to reopen the published index; this normally takes no more than five seconds
- **Embeddings slow**: Omit `--embeddings` (it's off by default) or set `OPENAI_API_KEY` for faster API-based embedding

44
.claude/skills/gitnexus-debugging/SKILL.md
View file

@ -13,9 +13,28 @@ description: "Use when the user is debugging a bug, tracing an error, or asking
- "This endpoint returns 500"
- Investigating bugs, errors, or unexpected behavior
## Bind the repository first
A root cause traced in the wrong repository is a wrong root cause.
Call `list_repos {}` before the first tool call. With one indexed repository,
use the examples below as written. With more than one, pass `repo` on every
call: an omitted `repo` normally errors, but under an MCP policy with a
configured default it resolves to that default silently. If you cannot tell
which repository is meant, stop and ask. This matters most for `cypher`, whose
statement carries no in-band hint of which database it ran against.
`list_repos` is paginated, so page with `offset: pagination.nextOffset` until
`hasMore` is false before concluding a repository is absent.
A stale index describes the code from before your bug, so refresh before
trusting a trace, and state the repository and index freshness with the
diagnosis.
## Workflow
```
0. list_repos {} → Bind repo
1. query({search_query: "<error or symptom>"}) → Find related execution flows
2. context({name: "<suspect>"}) → See callers/callees/processes
3. READ gitnexus://repo/{name}/process/{name} → Trace execution flow
@ -27,6 +46,7 @@ description: "Use when the user is debugging a bug, tracing an error, or asking
## Checklist
```
- [ ] list_repos {} — bind repo; explicit repo when >1 indexed, ask if ambiguous
- [ ] Understand the symptom (error message, unexpected behavior)
- [ ] query for error text or related code
- [ ] Identify the suspect function from returned processes
@ -34,6 +54,7 @@ description: "Use when the user is debugging a bug, tracing an error, or asking
- [ ] Trace execution flow via process resource if applicable
- [ ] cypher for custom call chain traces if needed
- [ ] Read source files to confirm root cause
- [ ] State the repository and index freshness with the diagnosis
```
## Debugging Patterns
@ -44,7 +65,7 @@ description: "Use when the user is debugging a bug, tracing an error, or asking
| Wrong return value | `context` on the function → trace callees for data flow |
| Intermittent failure | `context` → look for external calls, async deps |
| Performance issue | `context` → find symbols with many callers (hot paths) |
| Recent regression | `detect_changes` to see what your changes affect |
| Recent regression | `detect_changes` to see what your changes affect — pass `worktree` for a linked worktree |
| "How does A reach B?" | `trace` between the two symbols — shortest call chain in one call |
## Tools
@ -52,7 +73,7 @@ description: "Use when the user is debugging a bug, tracing an error, or asking
**query** — find code related to error:
```
query({search_query: "payment validation error"})
query({search_query: "payment validation error", repo: "my-app"})
→ Processes: CheckoutFlow, ErrorHandling
→ Symbols: validatePayment, handlePaymentError, PaymentException
```
@ -60,13 +81,15 @@ query({search_query: "payment validation error"})
**context** — full context for a suspect:
```
context({name: "validatePayment"})
context({name: "validatePayment", repo: "my-app"})
→ Incoming calls: processCheckout, webhookHandler
→ Outgoing calls: verifyCard, fetchRates (external API!)
→ Processes: CheckoutFlow (step 3/7)
```
**cypher** — custom call chain traces:
**cypher** — custom call chain traces. Pass `repo` alongside the statement; the
Cypher text itself names no repository, so the result is unattributable without
it:
```cypher
MATCH path = (a)-[:CodeRelation {type: 'CALLS'}*1..2]->(b:Function {name: "validatePayment"})
@ -76,7 +99,7 @@ RETURN [n IN nodes(path) | n.name] AS chain
**trace** — shortest call chain between two symbols ("how does A reach B?"), one call instead of chaining `context` hops:
```
trace({ from: "processCheckout", to: "fetchRates" })
trace({ from: "processCheckout", to: "fetchRates", repo: "my-app" })
→ status: ok, hopCount: 3
→ hops: processCheckout → validatePayment → verifyCard → fetchRates
→ edges: CALLS (1.0), CALLS (0.95), CALLS (1.0)
@ -87,15 +110,22 @@ When no path exists, `trace` reports the furthest reachable node — exactly whe
## Example: "Payment endpoint returns 500 intermittently"
```
1. query({search_query: "payment error handling"})
0. list_repos {}
→ total: 2 (my-app, billing-api) — bind my-app explicitly on every call
1. query({search_query: "payment error handling", repo: "my-app"})
→ Processes: CheckoutFlow, ErrorHandling
→ Symbols: validatePayment, handlePaymentError
2. context({name: "validatePayment"})
2. context({name: "validatePayment", repo: "my-app"})
→ Outgoing calls: verifyCard, fetchRates (external API!)
3. READ gitnexus://repo/my-app/process/CheckoutFlow
→ Step 3: validatePayment → calls fetchRates (external)
4. Root cause: fetchRates calls external API without proper timeout
Repository: my-app Index: current
```
With a single indexed repository, step 0 returns `total: 1` and the `repo`
argument drops out of every call above.

30
.claude/skills/gitnexus-exploring/SKILL.md
View file

@ -13,10 +13,22 @@ description: "Use when the user asks how code works, wants to understand archite
- "Where is the database logic?"
- Understanding code you haven't seen before
## Bind the repository first
Step 1 discovers what is indexed; every call after it must say which of those
it means. With one indexed repository, use the examples below as written. With
more than one, pass `repo` on every call: an omitted `repo` normally errors,
but under an MCP policy with a configured default it resolves to that default
silently. If you cannot tell which repository is meant, stop and ask. Report
the bound repository and index freshness alongside your explanation.
`list_repos` is paginated, so page with `offset: pagination.nextOffset` until
`hasMore` is false before concluding a repository is absent.
## Workflow
```
1. READ gitnexus://repos → Discover indexed repos
1. list_repos {} or READ gitnexus://repos → Discover indexed repos
2. READ gitnexus://repo/{name}/context → Codebase overview, check staleness
3. query({search_query: "<what you want to understand>"}) → Find related execution flows
4. context({name: "<symbol>"}) → Deep dive on specific symbol
@ -28,12 +40,14 @@ description: "Use when the user asks how code works, wants to understand archite
## Checklist
```
- [ ] list_repos {} — bind repo; explicit repo when >1 indexed, ask if ambiguous
- [ ] READ gitnexus://repo/{name}/context
- [ ] query for the concept you want to understand
- [ ] Review returned processes (execution flows)
- [ ] context on key symbols for callers/callees
- [ ] READ process resource for full execution traces
- [ ] Read source files for implementation details
- [ ] State the repository and index freshness with the explanation
```
## Resources
@ -50,7 +64,7 @@ description: "Use when the user asks how code works, wants to understand archite
**query** — find execution flows related to a concept:
```
query({search_query: "payment processing"})
query({search_query: "payment processing", repo: "my-app"})
→ Processes: CheckoutFlow, RefundFlow, WebhookHandler
→ Symbols grouped by flow with file locations
```
@ -58,16 +72,20 @@ query({search_query: "payment processing"})
**context** — 360-degree view of a symbol:
```
context({name: "validateUser"})
context({name: "validateUser", repo: "my-app"})
→ Incoming calls: loginHandler, apiMiddleware
→ Outgoing calls: checkToken, getUserById
→ Processes: LoginFlow (step 2/5), TokenRefresh (step 1/3)
```
`repo` is required once more than one repository is indexed, and may be omitted
with a single one.
## Example: "How does payment processing work?"
```
1. READ gitnexus://repo/my-app/context → 918 symbols, 45 processes
1. list_repos {} → total: 1 (my-app) — bind it
READ gitnexus://repo/my-app/context → 918 symbols, 45 processes
2. query({search_query: "payment processing"})
→ CheckoutFlow: processPayment → validateCard → chargeStripe
→ RefundFlow: initiateRefund → calculateRefund → processRefund
@ -75,4 +93,8 @@ context({name: "validateUser"})
→ Incoming: checkoutHandler, webhookHandler
→ Outgoing: validateCard, chargeStripe, saveTransaction
4. Read src/payments/processor.ts for implementation details
5. Answer, noting: Repository my-app, index current
```
Had step 1 returned two repositories, every call above would carry
`repo: "my-app"`.

14
.claude/skills/gitnexus-guide/SKILL.md
View file

@ -83,15 +83,23 @@ Notes: `offset` ≥ `total` returns an empty page (with `total` still reported).
### Inline staleness signal (`query` / `context` / `impact` / `cypher`)
These four hot read tools attach a non-blocking `staleness` field to their response when the index is behind the checkout's current HEAD — the same `{ commitsBehind, hint }` shape `list_repos` already reports — so a direct tool call surfaces a behind-HEAD index without a separate `list_repos` call:
These four hot read tools attach a non-blocking `staleness` field to their response when the index is not at the checkout's current HEAD — the same `{ status, commitsBehind?, hint? }` shape `list_repos` already reports — so a direct tool call surfaces a stale index without a separate `list_repos` call:
```jsonc
{ /* …the tool's normal result… */
"staleness": { "commitsBehind": 3, "hint": "⚠️ Index is 3 commits behind HEAD. Run analyze tool to update." }
"staleness": { "status": "behind", "commitsBehind": 3, "hint": "⚠️ Index is 3 commits behind HEAD. Run analyze tool to update." }
}
```
The field is **absent when the index is current** (or when the freshness check can't run), so its presence is the signal. It is only ever added to object results — raw-array `cypher` output and error envelopes are returned unchanged. `@group`-targeted calls do not carry it (multi-repo staleness is ill-defined). When you see it, the graph may be behind the working tree — re-run `analyze` before trusting blast-radius or dependence answers.
`commitsBehind` is present only when git counted the gap. When git could not count it but HEAD still resolves to a commit other than the indexed one — usually because the indexed commit is no longer in the clone's history — the index is provably not at HEAD with no countable gap, so no number is reported:
```jsonc
{ /* …the tool's normal result… */
"staleness": { "status": "diverged", "hint": "⚠️ Index is not at HEAD and the commit gap could not be counted — the recorded commit may no longer be in this clone's history. Run analyze tool to update." }
}
```
The field is **absent when the index is current**, and these four tools also omit it when the freshness check could not run at all — that case is `status: "unknown"`, which only the `list_repos` listing reports. So its presence means the status is not `current`: read `status` before using `commitsBehind`. It is only ever added to object results — raw-array `cypher` output and error envelopes are returned unchanged. `@group`-targeted calls do not carry it (multi-repo staleness is ill-defined). When you see it, the graph may be behind the working tree — re-run `analyze` before trusting blast-radius or dependence answers.
### Taint findings (`explain`)

90
.claude/skills/gitnexus-impact-analysis/SKILL.md
View file

@ -14,26 +14,58 @@ description: "Use when the user wants to know what will break if they change som
- Before making non-trivial code changes
- Before committing — to understand what your changes affect
## Bind the repository first
Impact analysis is the gate that authorizes an edit, so it must answer for the
repository you are about to edit.
Call `list_repos {}` before the first tool call. With one indexed repository,
use the examples below as written. With more than one, pass `repo` on every
call: an omitted `repo` normally errors, but under an MCP policy with a
configured default it resolves to that default silently. If you cannot tell
which repository is meant, stop and ask — every result below an ambiguous
identity inherits the ambiguity. `list_repos` is paginated, so page with
`offset: pagination.nextOffset` until `hasMore` is false before concluding a
repository is absent.
`detect_changes` takes `worktree` when your changes are in a linked worktree
the MCP server was not launched from. The server auto-detects a worktree only
when it was launched from inside one; otherwise `git diff` runs in the wrong
checkout and reports zero changed symbols — a false clean check that carries
none of the degradation flags described below. In the CLI fallbacks, `--repo .`
means the current checkout; pass the intended repository path instead when you
are not standing in it.
State the bound identity with your risk report:
```
Repository: <name> (<path>) Worktree: <path> Index: <commit>, <n> behind HEAD
```
## Workflow
```
1. impact({target: "X", direction: "upstream"}) → What depends on this
0. list_repos {} → Bind repo (and worktree)
1. impact({target: "X", direction: "upstream"}) or `node .gitnexus/run.cjs impact "X" --direction upstream --repo .`
2. READ gitnexus://repo/{name}/processes → Check affected execution flows
3. detect_changes() → Map current git changes to affected flows
4. Assess risk and report to user
3. detect_changes({scope: "all"}) or `node .gitnexus/run.cjs detect-changes --scope all --repo .`
4. Assess risk and report to user, echoing repo/worktree/index identity
```
> If "Index is stale" → run `node .gitnexus/run.cjs analyze` in terminal.
> If `.gitnexus/run.cjs` is missing, replace `node .gitnexus/run.cjs` with `npx gitnexus` in the fallback commands.
## Checklist
```
- [ ] impact({target, direction: "upstream"}) to find dependents
- [ ] list_repos {} — bind repo; explicit repo when >1 indexed, ask if ambiguous
- [ ] impact({target, direction: "upstream"}) or CLI fallback to find dependents
- [ ] Review d=1 items first (these WILL BREAK)
- [ ] Check high-confidence (>0.8) dependencies
- [ ] READ processes to check affected execution flows
- [ ] detect_changes() for pre-commit check
- [ ] Assess risk level and report to user
- [ ] detect_changes({scope: "all"}) or CLI fallback for pre-commit check
- [ ] Confirm the checkout you edited is the checkout that was diffed
- [ ] Assess risk level and report, stating repo/worktree/index identity
```
## Understanding Output
@ -52,14 +84,32 @@ description: "Use when the user wants to know what will break if they change som
| 5-15 symbols, 2-5 processes | MEDIUM |
| >15 symbols or many processes | HIGH |
| Critical path (auth, payments) | CRITICAL |
| **Zero callers found** | **UNKNOWN** |
`UNKNOWN` is not a low rung on this scale — it means the walk could not answer.
An empty caller set is equally consistent with "genuinely unused" and "the
callers are not resolvable by the index" (plain-object property access, dynamic
dispatch, cross-language calls), so few-callers ⇒ LOW does **not** apply. The
result carries a `riskNote` saying so. Confirm with a text search before
treating the symbol as safe to change or delete.
`risk` is the edit gate: warn on HIGH/CRITICAL and stop on UNKNOWN until the
uncertainty is resolved. Within single-repo mode, compare File and symbol
targets with local `riskSharedAxes` (direct/total only). Within group mode,
compare only group results: their `riskSharedAxes` overlays resolved
cross-repo crossings on that local value. Never use either field to waive the
edit gate. Check `riskScale.unusedAxes` before comparing kinds: MCP File walks
omit process/module axes, while web Graph-RAG expands File targets to in-file
symbols before enrichment.
## Tools
**impact** — the primary tool for symbol blast radius:
**impact** — the primary tool for symbol blast radius. If MCP is unavailable, use `node .gitnexus/run.cjs impact <symbol> --direction upstream --repo .` instead:
```
impact({
target: "validateUser",
repo: "my-app", // required once >1 repository is indexed
direction: "upstream",
minConfidence: 0.8,
maxDepth: 3
@ -73,20 +123,36 @@ impact({
- authRouter (src/routes/auth.ts:22) [CALLS, 95%]
```
**detect_changes** — git-diff based impact analysis:
**detect_changes** — git-diff based impact analysis. If MCP is unavailable, use `node .gitnexus/run.cjs detect-changes --scope all --repo .` instead:
```
detect_changes({scope: "staged"})
detect_changes({scope: "all"})
→ Changed: 5 symbols in 3 files
→ Affected: LoginFlow, TokenRefresh, APIMiddlewarePipeline
→ Risk: MEDIUM
```
Add `repo` once more than one repository is indexed, and `worktree: "<abs
path>"` when your changes are in a linked worktree the server was not launched
from.
`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol
listing was capped) means the result is short of the truth, and reads like
`UNKNOWN` above: a zero there means unseen, not unaffected. Re-run it rather
than tick the pre-commit check.
A wrong-worktree zero carries neither flag and is shape-identical to a genuine
clean result, so confirm the checkout you edited is the one that was diffed
before treating an empty change set as a passed check.
## Example: "What breaks if I change validateUser?"
```
1. impact({target: "validateUser", direction: "upstream"})
0. list_repos {}
→ total: 2 (my-app, billing-api) — both define validateUser, so bind explicitly
1. impact({target: "validateUser", repo: "my-app", direction: "upstream"}) or `node .gitnexus/run.cjs impact "validateUser" --direction upstream --repo .`
→ d=1: loginHandler, apiMiddleware (WILL BREAK)
→ d=2: authRouter, sessionManager (LIKELY AFFECTED)
@ -94,4 +160,8 @@ detect_changes({scope: "staged"})
→ LoginFlow and TokenRefresh touch validateUser
3. Risk: 2 direct callers, 2 processes = MEDIUM
Repository: my-app (/abs/path/my-app) Worktree: same Index: current
```
With a single indexed repository, step 0 returns `total: 1` and the `repo`
argument drops out of every call above.

17
.claude/skills/gitnexus-plan/README.md
View file

@ -124,12 +124,17 @@ phase that needs them.
statement-level claims (never reconstructs fake edges).
- No GitNexus at all → fallback mode: targeted grep/read exploration, findings
labelled **source-derived**, with a recommendation to index.
- Reading or publishing a plan requires Linux `/proc/self/fd`, `O_DIRECTORY`,
and `O_NOFOLLOW`; publication also requires a validated absolute Python 3
PATH candidate with libc `renameat2(RENAME_NOREPLACE)` support, a
writable target repository, and a shared filesystem for the plan and
Git-admin vault. The writer fails closed when those guarantees are
unavailable; it never redirects the plan elsewhere.
- Reading or publishing a plan requires `O_DIRECTORY` and `O_NOFOLLOW`, plus
`/proc/self/fd` on Linux; every other platform is refused. No interpreter is
spawned and no native code is loaded. Publication is `link(2)`, which fails
rather than replaces when the destination name is taken. Linux resolves every
name against a held descriptor, so a parent swapped mid-write cannot redirect
the operation; macOS has no equivalent path and instead pins each directory
with an open descriptor and re-proves the chain either side of every step,
which detects such a swap and aborts. Publishing also needs a writable target
repository and a shared filesystem for the plan and Git-admin vault. The
writer fails closed when those guarantees are unavailable; it never redirects
the plan elsewhere.
## Limitations

61
.claude/skills/gitnexus-plan/references/evidence-provenance.md
View file

@ -98,8 +98,11 @@ excluded.
## Safe existing-plan read contract
`read-plan` fails closed unless Linux `/proc/self/fd`, `O_DIRECTORY`, and
`O_NOFOLLOW` are available. It resolves the exact Git top-level, opens the
`read-plan` fails closed unless the host platform can resolve names against a
held directory descriptor: Linux `/proc/self/fd` with `O_DIRECTORY` and
`O_NOFOLLOW`, or macOS `O_DIRECTORY`/`O_NOFOLLOW`. Every other platform is
refused outright — an unverified read is not a degraded read, it is a different,
racy operation. It resolves the exact Git top-level, opens the
repository root and every plan parent as held no-follow directory descriptors,
rejects missing, symlink, non-directory, and escaping parents, and opens the
leaf with `O_NOFOLLOW`. It reads at most 16 MiB from that held file descriptor,
@ -109,13 +112,17 @@ Neither Deepen nor work may parse bytes obtained before or outside this receipt.
## Safe generated-plan write contract
The writer fails closed unless Linux `/proc/self/fd`, `O_DIRECTORY`,
`O_NOFOLLOW`, and Python 3 with libc `renameat2(RENAME_NOREPLACE)` support are
available. Python may live in `/usr/local`, a Nix profile, or another absolute
PATH directory, but the helper accepts only a resolved executable and
containing directory owned by root or the current user and not writable by
group/other. The resolved executable is opened without following links and
invoked through that held descriptor. Relative PATH entries are ignored. The plan parent and the
The writer fails closed unless the host platform offers `O_DIRECTORY` and
`O_NOFOLLOW`, plus `/proc/self/fd` on Linux. It spawns no interpreter and loads
no native code: publication is `link(2)`, which is atomic, fails `EEXIST` when
the destination name is taken, and refuses a symlinked destination without
following it — the same no-replace guarantee `renameat2(RENAME_NOREPLACE)` and
`renameatx_np(RENAME_EXCL)` provide, available through `fs.linkSync` on every
supported platform. The temporary name is unlinked once the link succeeds; the
published file is the same inode the writer created and verified, so every
identity check downstream holds by construction. A link that succeeds followed
by an unlink that fails leaves the plan published and is reported as success,
because it is one. The plan parent and the
repository's Git-admin directory must also share a filesystem. It resolves
the target repository's exact Git top-level, opens that root and every
destination parent as held no-follow directory descriptors, creates missing
@ -128,15 +135,45 @@ The writer creates a random exclusive temporary file relative to the held final
parent descriptor and keeps its no-follow descriptor open. It writes and
flushes the bytes, binds the temporary name to the opened inode, and hashes the
open file before publication. Immediately before publication it revalidates
the parent and the temporary path, inode, size, and digest. Publication uses an
atomic no-replace move relative to the held directory descriptor. Initial mode
therefore cannot overwrite a destination that appears after the absent check.
the parent and the temporary path, inode, size, and digest. Publication links
the temporary name to the destination relative to the held directory
descriptor, which fails rather than replaces if the destination is taken.
Initial mode therefore cannot overwrite a destination that appears after the
absent check.
The writer then flushes the directory and revalidates the committed path by
opening it with `O_NOFOLLOW`, hashing both the original temporary fd and the
path-bound fd, and performing a second descriptor-anchored path identity check
after hashing. A detected mutation or replacement aborts instead of accepting
mixed-era output.
### Linux anchors, macOS verifies
The two platforms reach the same destination by different proofs, and the
difference is real enough to state rather than smooth over.
On Linux every name resolves through `/proc/self/fd/<fd>/<child>`, a magic link
the kernel resolves against the inode the descriptor already holds. The names
above it are never re-walked, so an attacker who renames a parent between the
check and the use cannot redirect the operation. The race is impossible, not
merely detected.
macOS has no such path. `/dev/fd/<fd>` is a devfs node, not a magic link: it can
be opened, but nothing can be resolved through it. `open("/dev/fd/<fd>/child")`
returns `ENOENT`, and `realpath` of it returns `/dev/fd/<fd>` rather than the
directory's path — measured on macOS 26, not inferred. Node exposes no `openat`,
no `dir_fd` parameter, and no FFI, so on macOS the writer resolves names
lexically with `O_NOFOLLOW` at every component, holds an open descriptor on
every directory in the chain for the whole operation, and proves before *and*
after each step that the chain still names exactly the inodes it is holding.
Holding the descriptors is what makes the recorded inode numbers trustworthy:
an open descriptor pins its inode, so a freed number cannot be recycled beneath
the walk.
What that buys is detection rather than prevention. A parent swapped inside the
window between a check and its use is caught by the check that follows, and the
operation aborts having written nothing — but on Linux it could not have
happened at all. No published byte escapes verification on either platform.
`--replace` accepts only a pre-existing regular file and is reserved for
Deepen; without it, accidental overwrite is rejected. It also requires the
exact canonical `generated_plan_path` and `plan_digest` from the same session's

1048
.claude/skills/gitnexus-plan/scripts/evidence-provenance.mjs
View file

File diff suppressed because it is too large Load diff

53
.claude/skills/gitnexus-refactoring/SKILL.md
View file

@ -13,9 +13,32 @@ description: "Use when the user wants to rename, extract, split, move, or restru
- "Move this to a new file"
- Any task involving renaming, extracting, splitting, or restructuring code
## Bind the repository first
Refactoring writes to disk. `rename` with `dry_run: false` edits files in
whichever repository was resolved, so binding identity here is a safety gate,
not bookkeeping.
Call `list_repos {}` before the first tool call. With one indexed repository,
use the examples below as written. With more than one, pass `repo` on every
call: an omitted `repo` normally errors, but under an MCP policy with a
configured default it resolves to that default silently. If you cannot tell
which repository is meant, stop and ask. Never run `rename` with
`dry_run: false` until the preview in the same bound repository has been
reviewed — its returned `file_path` values show which checkout is about to be
written, so read them as a confirmation of identity.
`list_repos` is paginated, so page with `offset: pagination.nextOffset` until
`hasMore` is false before concluding a repository is absent.
`detect_changes` takes `worktree` when you are editing a linked worktree the
MCP server was not launched from; otherwise `git diff` runs in the wrong
checkout and reports nothing changed, which reads as a verified refactor.
## Workflow
```
0. list_repos {} → Bind repo (and worktree)
1. impact({target: "X", direction: "upstream"}) → Map all dependents
2. query({search_query: "X"}) → Find execution flows involving X
3. context({name: "X"}) → See all incoming/outgoing refs
@ -29,7 +52,9 @@ description: "Use when the user wants to rename, extract, split, move, or restru
### Rename Symbol
```
- [ ] list_repos {} — bind repo; explicit repo when >1 indexed, ask if ambiguous
- [ ] rename({symbol_name: "oldName", new_name: "newName", dry_run: true}) — preview all edits
- [ ] Confirm the previewed file paths are in the bound repository/worktree
- [ ] Review graph edits (high confidence) and text_search edits (review carefully)
- [ ] If satisfied: rename({..., dry_run: false}) — apply edits
- [ ] detect_changes() — verify only expected files changed
@ -39,6 +64,7 @@ description: "Use when the user wants to rename, extract, split, move, or restru
### Extract Module
```
- [ ] list_repos {} — bind repo; explicit repo when >1 indexed, ask if ambiguous
- [ ] context({name: target}) — see all incoming/outgoing refs
- [ ] impact({target, direction: "upstream"}) — find all external callers
- [ ] Define new module interface
@ -50,6 +76,7 @@ description: "Use when the user wants to rename, extract, split, move, or restru
### Split Function/Service
```
- [ ] list_repos {} — bind repo; explicit repo when >1 indexed, ask if ambiguous
- [ ] context({name: target}) — understand all callees
- [ ] Group callees by responsibility
- [ ] impact({target, direction: "upstream"}) — map callers to update
@ -64,7 +91,7 @@ description: "Use when the user wants to rename, extract, split, move, or restru
**rename** — automated multi-file rename:
```
rename({symbol_name: "validateUser", new_name: "authenticateUser", dry_run: true})
rename({symbol_name: "validateUser", new_name: "authenticateUser", repo: "my-app", dry_run: true})
→ 12 edits across 8 files
→ 10 graph edits (high confidence), 2 text_search edits (review)
→ Changes: [{file_path, edits: [{line, old_text, new_text, confidence}]}]
@ -73,7 +100,7 @@ rename({symbol_name: "validateUser", new_name: "authenticateUser", dry_run: true
**impact** — map all dependents first:
```
impact({target: "validateUser", direction: "upstream"})
impact({target: "validateUser", repo: "my-app", direction: "upstream"})
→ d=1: loginHandler, apiMiddleware, testUtils
→ Affected Processes: LoginFlow, TokenRefresh
```
@ -87,6 +114,14 @@ detect_changes({scope: "all"})
→ Risk: MEDIUM
```
`partial: true` (a graph query failed) or `truncated: true` (the changed-symbol
listing was capped) means the result is short of the truth: a short or empty
list is not proof that only the expected files changed. Re-run it rather than
treat the refactor as verified.
A wrong-worktree zero carries neither flag and is indistinguishable from a
clean verification, so confirm the diffed checkout is the one you edited.
**cypher** — custom reference queries:
```cypher
@ -102,20 +137,28 @@ RETURN caller.name, caller.filePath ORDER BY caller.filePath
| Cross-area refs | Use detect_changes after to verify scope |
| String/dynamic refs | query to find them |
| External/public API | Version and deprecate properly |
| Same name in another indexed repo | Bind `repo`; verify previewed paths before applying |
## Example: Rename `validateUser` to `authenticateUser`
```
1. rename({symbol_name: "validateUser", new_name: "authenticateUser", dry_run: true})
0. list_repos {}
→ total: 2 (my-app, billing-api) — both define validateUser, so bind explicitly
1. rename({symbol_name: "validateUser", new_name: "authenticateUser", repo: "my-app", dry_run: true})
→ 12 edits: 10 graph (safe), 2 text_search (review)
→ Files: validator.ts, login.ts, middleware.ts, config.json...
2. Review text_search edits (config.json: dynamic reference!)
3. rename({symbol_name: "validateUser", new_name: "authenticateUser", dry_run: false})
3. rename({symbol_name: "validateUser", new_name: "authenticateUser", repo: "my-app", dry_run: false})
→ Applied 12 edits across 8 files
4. detect_changes({scope: "all"})
4. detect_changes({scope: "all", repo: "my-app"})
→ Affected: LoginFlow, TokenRefresh
→ Risk: MEDIUM — run tests for these flows
Repository: my-app (/abs/path/my-app) Worktree: same Index: current
```
With a single indexed repository, step 0 returns `total: 1` and the `repo`
argument drops out of every call above.

15
.claude/skills/gitnexus-review/SKILL.md
View file

@ -120,10 +120,17 @@ and do not claim a complete graph-backed review.
review surface: when the diff changes what gets emitted or persisted,
verify every schema/version constant gating caches, incremental
writebacks, and fingerprint baselines was bumped or regenerated — in
GitNexus itself, for example: `INCREMENTAL_SCHEMA_VERSION` (the
incremental write set covers only changed files, so new cross-file edges
never reach an existing index without the bump), the parse-store
`SCHEMA_BUMP`, and both bench fingerprint sets.
GitNexus itself, for example: graph DDL needs no manual bump, because
`SCHEMA_FINGERPRINT` (`gitnexus/src/core/lbug/schema.ts`) is derived
from `NODE_SCHEMA_QUERIES` + `REL_SCHEMA_QUERIES` and moves on its own;
the check there is whether the diff changed any string in those arrays,
and, if it added a new DDL array, whether that array was folded into the
fingerprint. The hand-maintained ritual still applies where no
declarative artifact describes the invalidated set: the parse-store
`SCHEMA_BUMP` and both bench fingerprint sets still need an explicit
bump, re-checked against the base branch right before merge. Semantic
changes that leave the DDL untouched are outside the fingerprint; they
rely on the analyzer runner-identity receipt in the index metadata.
## Expert lenses

5
.claude/skills/gitnexus-work/SKILL.md
View file

@ -216,7 +216,10 @@ Work through plan §7 step by step, in order. For each step:
`detect_changes` → commit as one unbroken sequence from the repository
root — interleaving other work between the gate and the commit is how
the gate gets skipped. Unexpected
affected flows → investigate before committing, not after.
affected flows → investigate before committing, not after. A result
flagged `partial` (a graph query failed) or `truncated` (the symbol
listing was capped) blocks the commit the same way: the gate did not
see every changed symbol, so re-run it rather than read it as clean.
A relationship-affecting implementation edit or commit invalidates the
procedure's prior proof. The next step must perform the required inter-step

61
.claude/skills/gitnexus-work/references/evidence-provenance.md
View file

@ -98,8 +98,11 @@ excluded.
## Safe existing-plan read contract
`read-plan` fails closed unless Linux `/proc/self/fd`, `O_DIRECTORY`, and
`O_NOFOLLOW` are available. It resolves the exact Git top-level, opens the
`read-plan` fails closed unless the host platform can resolve names against a
held directory descriptor: Linux `/proc/self/fd` with `O_DIRECTORY` and
`O_NOFOLLOW`, or macOS `O_DIRECTORY`/`O_NOFOLLOW`. Every other platform is
refused outright — an unverified read is not a degraded read, it is a different,
racy operation. It resolves the exact Git top-level, opens the
repository root and every plan parent as held no-follow directory descriptors,
rejects missing, symlink, non-directory, and escaping parents, and opens the
leaf with `O_NOFOLLOW`. It reads at most 16 MiB from that held file descriptor,
@ -109,13 +112,17 @@ Neither Deepen nor work may parse bytes obtained before or outside this receipt.
## Safe generated-plan write contract
The writer fails closed unless Linux `/proc/self/fd`, `O_DIRECTORY`,
`O_NOFOLLOW`, and Python 3 with libc `renameat2(RENAME_NOREPLACE)` support are
available. Python may live in `/usr/local`, a Nix profile, or another absolute
PATH directory, but the helper accepts only a resolved executable and
containing directory owned by root or the current user and not writable by
group/other. The resolved executable is opened without following links and
invoked through that held descriptor. Relative PATH entries are ignored. The plan parent and the
The writer fails closed unless the host platform offers `O_DIRECTORY` and
`O_NOFOLLOW`, plus `/proc/self/fd` on Linux. It spawns no interpreter and loads
no native code: publication is `link(2)`, which is atomic, fails `EEXIST` when
the destination name is taken, and refuses a symlinked destination without
following it — the same no-replace guarantee `renameat2(RENAME_NOREPLACE)` and
`renameatx_np(RENAME_EXCL)` provide, available through `fs.linkSync` on every
supported platform. The temporary name is unlinked once the link succeeds; the
published file is the same inode the writer created and verified, so every
identity check downstream holds by construction. A link that succeeds followed
by an unlink that fails leaves the plan published and is reported as success,
because it is one. The plan parent and the
repository's Git-admin directory must also share a filesystem. It resolves
the target repository's exact Git top-level, opens that root and every
destination parent as held no-follow directory descriptors, creates missing
@ -128,15 +135,45 @@ The writer creates a random exclusive temporary file relative to the held final
parent descriptor and keeps its no-follow descriptor open. It writes and
flushes the bytes, binds the temporary name to the opened inode, and hashes the
open file before publication. Immediately before publication it revalidates
the parent and the temporary path, inode, size, and digest. Publication uses an
atomic no-replace move relative to the held directory descriptor. Initial mode
therefore cannot overwrite a destination that appears after the absent check.
the parent and the temporary path, inode, size, and digest. Publication links
the temporary name to the destination relative to the held directory
descriptor, which fails rather than replaces if the destination is taken.
Initial mode therefore cannot overwrite a destination that appears after the
absent check.
The writer then flushes the directory and revalidates the committed path by
opening it with `O_NOFOLLOW`, hashing both the original temporary fd and the
path-bound fd, and performing a second descriptor-anchored path identity check
after hashing. A detected mutation or replacement aborts instead of accepting
mixed-era output.
### Linux anchors, macOS verifies
The two platforms reach the same destination by different proofs, and the
difference is real enough to state rather than smooth over.
On Linux every name resolves through `/proc/self/fd/<fd>/<child>`, a magic link
the kernel resolves against the inode the descriptor already holds. The names
above it are never re-walked, so an attacker who renames a parent between the
check and the use cannot redirect the operation. The race is impossible, not
merely detected.
macOS has no such path. `/dev/fd/<fd>` is a devfs node, not a magic link: it can
be opened, but nothing can be resolved through it. `open("/dev/fd/<fd>/child")`
returns `ENOENT`, and `realpath` of it returns `/dev/fd/<fd>` rather than the
directory's path — measured on macOS 26, not inferred. Node exposes no `openat`,
no `dir_fd` parameter, and no FFI, so on macOS the writer resolves names
lexically with `O_NOFOLLOW` at every component, holds an open descriptor on
every directory in the chain for the whole operation, and proves before *and*
after each step that the chain still names exactly the inodes it is holding.
Holding the descriptors is what makes the recorded inode numbers trustworthy:
an open descriptor pins its inode, so a freed number cannot be recycled beneath
the walk.
What that buys is detection rather than prevention. A parent swapped inside the
window between a check and its use is caught by the check that follows, and the
operation aborts having written nothing — but on Linux it could not have
happened at all. No published byte escapes verification on either platform.
`--replace` accepts only a pre-existing regular file and is reserved for
Deepen; without it, accidental overwrite is rejected. It also requires the
exact canonical `generated_plan_path` and `plan_digest` from the same session's

1048
.claude/skills/gitnexus-work/scripts/evidence-provenance.mjs
View file

File diff suppressed because it is too large Load diff

20
.claude/skills/gitnexus/gitnexus-taint-analysis/SKILL.md
View file

@ -148,13 +148,19 @@ finding is NOT proof of safety.
## Adding a source / sink / sanitizer
Edit the language model in `taint/typescript-model.ts` (registered via the
explicit `registerBuiltinTaintModels` seam, keyed by `SupportedLanguages`). The
spec is hashable data (no functions). A sanitizer's `neutralizes` lists the
EXACT sink kinds it defends — never a blanket kill. Add a fixture + assert the
finding (or its absence) in `test/unit/taint/` (real-source harness:
`test/helpers/ts-cfg-harness.ts`); the end-to-end proof is
`test/integration/cfg/`.
Taint models cover four `SupportedLanguages` ids across three files:
TypeScript and JavaScript use `taint/typescript-model.ts`, Python uses
`taint/python-model.ts`, and Java uses `taint/java-model.ts`. Edit the model
for the language you are targeting. The explicit
`registerBuiltinTaintModels` seam in `typescript-model.ts` registers all four;
it is not an import side effect.
The spec is hashable data (no functions). A sanitizer's `neutralizes` lists
the EXACT sink kinds it defends — never a blanket kill. Add a fixture + assert
the finding (or its absence) in `test/unit/taint/`. TypeScript and JavaScript
use the real-source harness `test/helpers/ts-cfg-harness.ts`; Python and Java
model matches are covered by `python-model-match.test.ts` and
`java-model-match.test.ts`. The end-to-end proof is `test/integration/cfg/`.
## Validation checklist for any `--pdg` change

12
.gitattributes vendored
View file

@ -15,3 +15,15 @@
*.so binary
*.dll binary
*.dylib binary
# TypeScript sources are always text for diff purposes. Git's binary
# heuristic fires when EITHER blob in a pair carries a NUL, so a source
# file that carried one on a base commit still renders as "Binary files
# differ" — with no hunks and no inline comments — long after the byte
# itself is gone from the working tree. A head-side guard cannot see
# that, by construction. This does not mark the files binary or change
# how they are stored; it only stops the heuristic from hiding a diff.
*.ts diff
*.tsx diff
*.mts diff
*.cts diff

17
.github/actions/setup-gitnexus-web/action.yml vendored
View file

@ -11,12 +11,19 @@ runs:
cache: npm
cache-dependency-path: gitnexus-web/package-lock.json
- name: Build gitnexus-shared
run: npm install && npm run build
shell: bash
working-directory: gitnexus-shared
- name: Install web dependencies
run: npm ci
shell: bash
working-directory: gitnexus-web
env:
# Browsers are installed explicitly by e2e. Typecheck only needs types.
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: '1'
# Compile shared with the web package's TypeScript 5. Do not npm-ci
# gitnexus-shared (TypeScript 7 optional-platform install, ~7 minutes).
- name: Build gitnexus-shared
# node + lib/tsc.js — same on Windows/macOS/Linux. Do not use .bin/tsc
# (tsc.cmd on Windows; execFileSync cannot launch .cmd without a shell).
run: node ../gitnexus-web/node_modules/typescript/lib/tsc.js
shell: bash
working-directory: gitnexus-shared

31
.github/actions/setup-gitnexus/action.yml vendored
View file

@ -6,6 +6,13 @@ inputs:
description: Whether to run npm run build after install
required: false
default: 'false'
lifecycle-scripts:
description: >
Run npm lifecycle scripts (prepare/postinstall) during gitnexus npm ci.
Typecheck-only and pack-only jobs should set this to false: they do not
need dist/ or native grammar builds.
required: false
default: 'true'
runs:
using: composite
@ -16,16 +23,30 @@ runs:
cache: npm
cache-dependency-path: gitnexus/package-lock.json
- name: Build gitnexus-shared
run: npm install && npm run build
shell: bash
working-directory: gitnexus-shared
# Do not npm-ci gitnexus-shared. Its TypeScript 7 install is a 7-minute
# stall (optional platform packages) and is not in the CLI npm cache.
# prepare/build.js compiles shared with gitnexus's tsc; typecheck does
# the same below after an ignore-scripts install.
- name: Install dependencies
if: ${{ inputs.lifecycle-scripts != 'false' }}
run: npm ci
shell: bash
working-directory: gitnexus
- name: Install dependencies
if: ${{ inputs.lifecycle-scripts == 'false' }}
run: npm ci --ignore-scripts
shell: bash
working-directory: gitnexus
- name: Build gitnexus-shared
if: ${{ inputs.lifecycle-scripts == 'false' }}
# node + lib/tsc.js — same on Windows/macOS/Linux. Do not use .bin/tsc
# (tsc.cmd on Windows; execFileSync cannot launch .cmd without a shell).
run: node ../gitnexus/node_modules/typescript/lib/tsc.js
shell: bash
working-directory: gitnexus-shared
- name: Build
if: ${{ inputs.build == 'true' }}
run: npm run build

2
.github/scripts/check-tree-sitter-upgrade-readiness.py vendored
View file

@ -68,7 +68,9 @@ GRAMMARS: dict[str, tuple[str, str, str]] = {
"tree-sitter-typescript": ("tree-sitter/tree-sitter-typescript", "master", "typescript/src/parser.c"),
# Vendored parsers — kept here so the upstream coords for drift
# detection are co-located with every other grammar's coords.
"tree-sitter-objc": ("tree-sitter-grammars/tree-sitter-objc", "master", "src/parser.c"),
"tree-sitter-proto": ("coder3101/tree-sitter-proto", "main", "src/parser.c"),
"tree-sitter-zig": ("tree-sitter-grammars/tree-sitter-zig", "master", "src/parser.c"),
}
# npm-installed grammars deliberately held below npm latest (surfaced so reviewers

20
.github/scripts/test_check_tree_sitter_upgrade_readiness.py vendored
View file

@ -9,8 +9,8 @@ which is deliberately dependency-free so it runs on any vanilla runner. Run with
(pytest also discovers ``unittest.TestCase`` classes, so a future pytest CI job
picks these up unchanged.)
These tests lock in the #858 fix: the 5 vendored grammars
(c/swift/kotlin/dart/proto) are classified from the shared manifest
These tests lock in the #858 fix: the 7 vendored grammars
(c/swift/kotlin/dart/objc/proto/zig) are classified from the shared manifest
(.github/vendored-grammars.json), their ABI is read from gitnexus/vendor/<name>,
and the report never renders a bare ``?`` placeholder. All network is mocked.
"""
@ -192,7 +192,7 @@ class AssertCurrent(TestCase):
def test_assert_current_is_network_free_and_passes(self):
report, code = self._run_assert_current() # raises if any urlopen fires
self.assertEqual(code, 0)
# All 5 vendored grammars are introspected from the repo (ABI 14), not skipped.
# All 7 vendored grammars are introspected from the repo (ABI 14), not skipped.
for name in readiness.VENDORED_NAMES:
self.assertIn(f"{name}: vendored ABI", report)
@ -324,6 +324,7 @@ class ReportRendering(TestCase):
# which is what removes the old "? (fetch failed)" for tree-sitter-proto.
self.assertNotIn("tree-sitter-proto", _render_report.last_npm_calls)
self.assertNotIn("tree-sitter-dart", _render_report.last_npm_calls)
self.assertNotIn("@tree-sitter-grammars/tree-sitter-zig", _render_report.last_npm_calls)
self.assertNotIn("Could not check", self.report)
self.assertNotIn("fetch failed", self.report)
@ -343,11 +344,11 @@ class ReportRendering(TestCase):
cells = [c.strip() for c in self._matrix_row("tree-sitter-swift").strip().strip("|").split("|")]
self.assertEqual(cells[6], "n/a") # Upstream ABI column
def test_row_diff_regex_captures_all_fifteen_grammar_statuses(self):
def test_row_diff_regex_captures_all_grammar_statuses(self):
# The change-detection bot keys on this regex: group 1 = grammar name,
# group 2 = the Status cell ONLY (not the whole tail). It must match every
# row after the format change so status transitions keep being detected.
self.assertEqual(len(self.rows), 15)
self.assertEqual(len(self.rows), len(readiness.GRAMMARS))
for name in readiness.VENDORED_NAMES:
self.assertIn(name, self.rows)
# group 2 is the Status cell — held c renders exactly "Vendored — held",
@ -364,15 +365,16 @@ class ReportRendering(TestCase):
# Counts are derived from _render_report()'s mock corpus (all npm peer
# deps mocked permissive): of the 10 npm-installed grammars, 9 render
# Ready and 1 — tree-sitter-cpp — is the intentional pin (#1242), so it is
# not counted ready. The 3 blockers are that same pinned tree-sitter-cpp
# plus two held vendored grammars: ABI-held tree-sitter-c (#1242/#858) and
# not counted ready. The 4 blockers are that same pinned tree-sitter-cpp
# plus three held vendored grammars: ABI-held tree-sitter-c (#1242/#858),
# tree-sitter-kotlin (pinned to an unreleased fwcd main commit for `fun
# interface` support — ABI 14 is in range, but a hold counts as a blocker
# until it is lifted). If a grammar is added/removed or a pin/hold changes,
# until it is lifted), and tree-sitter-objc. If a grammar is added/removed
# or a pin/hold changes,
# update _render_report()'s mock AND these expected counts together; a
# mismatch here means the report prose drifted, not the regex.
self.assertEqual(ready.groups(), ("9", "10"))
self.assertEqual(blockers.group(1), "3")
self.assertEqual(blockers.group(1), "4")
def _matrix_row(self, name: str) -> str:
for line in self.report.splitlines():

2
.github/scripts/update-vendored-grammars.mjs vendored
View file

@ -21,7 +21,7 @@
* node update-vendored-grammars.mjs # detect only → JSON report on stdout
* node update-vendored-grammars.mjs --apply X # re-vendor grammar X in place
*
* tree-sitter-c is MONITORED but report-only (`hold`): it is ABI-pinned at 0.21.4
* tree-sitter-c and tree-sitter-objc are MONITORED but report-only (`hold`): c is ABI-pinned at 0.21.4
* (#1242/#858) and must not auto-bump without a tree-sitter runtime upgrade, so an
* available c update is detected + reported but never auto-applied — even if it is
* ABI-13/14. A maintainer re-vendors it deliberately.

274
.github/scripts/verify-workflow-run-pr-identity.cjs vendored Normal file
View file

@ -0,0 +1,274 @@
// Resolve the open PR for a trusted workflow_run consumer.
//
// Shared by commit-fork-prebuilds.yml and pr-autofix-publish.yml.
// workflow_run.pull_requests[] is empty on fork PRs, and
// GET /repos/{base}/commits/{sha}/pulls is also empty because the fork head
// commit is not in the base repo's commit graph. The authoritative lookup is
// GET /repos/{base}/pulls?head={owner}:{branch}&state=open using
// workflow_run.head_repository + workflow_run.head_branch (server-controlled).
// That same query works for same-repo PRs (owner is the base repo owner).
//
// The current PR tip may have moved past the SHA the producer built; that is
// not an identity failure — the caller decides whether to lease-push or just
// comment. Two open PRs from the same fork head (same owner:branch into this
// repo) are an identity failure: artifact pr_number is untrusted and must not
// pick among them. Set SCHEMA_PATTERN to the artifact schema allowlist
// (defaults to the tree-sitter prebuild schema).
'use strict';
const fs = require('node:fs');
const { spawnSync } = require('node:child_process');
const SCHEMA_PATTERN = /^gitnexus\.ts-prebuild\/v[0-9]+$/;
const IDENTITY_PATTERNS = {
pr_number: /^[0-9]+$/,
head_sha: /^[0-9a-f]{40}$/,
head_ref: /^[A-Za-z0-9._/-]+$/,
repo: /^[A-Za-z0-9._-]+\/[A-Za-z0-9._-]+$/,
};
function allowlistField(key, value, pattern) {
const text = value == null ? '' : String(value);
if (!text || !pattern.test(text)) {
throw new Error(`metadata.${key} failed allowlist (got: ${JSON.stringify(text)})`);
}
return text;
}
function forkHeadOwner(headRepo) {
const slash = headRepo.indexOf('/');
if (slash <= 0 || slash === headRepo.length - 1) {
throw new Error(`head_repo must be owner/name (got: ${JSON.stringify(headRepo)})`);
}
return headRepo.slice(0, slash);
}
function compileSchemaPattern(value) {
if (value instanceof RegExp) return value;
if (typeof value === 'string' && value.length > 0) {
try {
return new RegExp(value);
} catch {
throw new Error('SCHEMA_PATTERN is not a valid regular expression');
}
}
return SCHEMA_PATTERN;
}
function allowlistMetadata(raw, schemaPattern) {
const parsed = typeof raw === 'string' ? JSON.parse(raw) : raw;
if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
throw new Error('metadata.json must be an object');
}
return {
schema: allowlistField('schema', parsed.schema, compileSchemaPattern(schemaPattern)),
pr_number: allowlistField('pr_number', parsed.pr_number, IDENTITY_PATTERNS.pr_number),
head_sha: allowlistField('head_sha', parsed.head_sha, IDENTITY_PATTERNS.head_sha),
head_ref: allowlistField('head_ref', parsed.head_ref, IDENTITY_PATTERNS.head_ref),
head_repo: allowlistField('head_repo', parsed.head_repo, IDENTITY_PATTERNS.repo),
base_repo: allowlistField('base_repo', parsed.base_repo, IDENTITY_PATTERNS.repo),
};
}
function allowlistAuthority(authority) {
return {
head_sha: allowlistField('head_sha', authority.head_sha, IDENTITY_PATTERNS.head_sha),
head_repo: allowlistField('head_repo', authority.head_repo, IDENTITY_PATTERNS.repo),
head_branch: allowlistField('head_ref', authority.head_branch, IDENTITY_PATTERNS.head_ref),
base_repo: allowlistField('base_repo', authority.base_repo, IDENTITY_PATTERNS.repo),
};
}
function verifyArtifactAgainstWorkflowRun(meta, authority) {
if (meta.head_sha !== authority.head_sha) {
throw new Error(
`Artifact head_sha (${meta.head_sha}) != workflow_run.head_sha (${authority.head_sha}) — refusing.`,
);
}
if (meta.head_repo !== authority.head_repo) {
throw new Error(
`Artifact head_repo (${meta.head_repo}) != workflow_run.head_repository (${authority.head_repo}) — refusing.`,
);
}
if (meta.base_repo !== authority.base_repo) {
throw new Error('Artifact base_repo does not match $GITHUB_REPOSITORY — refusing.');
}
if (meta.head_ref !== authority.head_branch) {
throw new Error(
`Artifact head_ref (${meta.head_ref}) != workflow_run.head_branch (${authority.head_branch}) — refusing.`,
);
}
}
function matchOpenPullsFromForkHead(pulls, { headRepo, headBranch, baseRepo }) {
if (!Array.isArray(pulls)) {
throw new Error('GitHub pulls?head= lookup returned a non-array');
}
return pulls.filter((pr) => {
return (
pr &&
pr.state === 'open' &&
Number.isInteger(pr.number) &&
pr.head &&
pr.head.repo &&
pr.head.repo.full_name === headRepo &&
pr.head.ref === headBranch &&
pr.base &&
pr.base.repo &&
pr.base.repo.full_name === baseRepo
);
});
}
function resolveVerifiedPullRequest({ meta, authority, pulls, schemaPattern }) {
const cleanMeta = allowlistMetadata(meta, schemaPattern);
const cleanAuthority = allowlistAuthority(authority);
verifyArtifactAgainstWorkflowRun(cleanMeta, cleanAuthority);
const matched = matchOpenPullsFromForkHead(pulls, {
headRepo: cleanAuthority.head_repo,
headBranch: cleanAuthority.head_branch,
baseRepo: cleanAuthority.base_repo,
});
if (matched.length === 0) {
throw new Error(
`No open PR from ${cleanAuthority.head_repo}:${cleanAuthority.head_branch} targeting ${cleanAuthority.base_repo} — refusing.`,
);
}
// Artifact pr_number is untrusted. Do not use it to pick among several open
// PRs that share this fork head (same owner:branch into this repo, different
// base branches). Fail closed unless GitHub-controlled fields leave exactly one.
if (matched.length !== 1) {
throw new Error(
`Ambiguous open PRs from ${cleanAuthority.head_repo}:${cleanAuthority.head_branch} targeting ${cleanAuthority.base_repo} (${matched
.map((pr) => pr.number)
.join(',')}) — refusing.`,
);
}
const chosen = matched[0];
const expected = Number(cleanMeta.pr_number);
if (chosen.number !== expected) {
throw new Error(
`Artifact pr_number (${cleanMeta.pr_number}) is not the open PR(s) from this fork head (${chosen.number}) — refusing.`,
);
}
const currentHeadSha = typeof chosen.head.sha === 'string' ? chosen.head.sha : '';
return {
pr_number: String(chosen.number),
head_ref: cleanAuthority.head_branch,
head_sha: cleanAuthority.head_sha,
head_repo: cleanAuthority.head_repo,
current_head_sha: currentHeadSha,
branch_moved: Boolean(currentHeadSha && currentHeadSha !== cleanAuthority.head_sha),
};
}
function flattenGhListPages(parsed) {
if (!Array.isArray(parsed)) {
throw new Error('GitHub pulls?head= lookup returned a non-array');
}
if (parsed.length === 0) return parsed;
if (parsed.every((page) => Array.isArray(page))) {
return parsed.flat();
}
return parsed;
}
function listOpenPullsByHead({ ghRepo, headOwner, headBranch, runGh }) {
const run = runGh || ((args) => spawnSync('gh', args, { encoding: 'utf8' }));
const result = run([
'api',
'--paginate',
'--slurp',
'-X',
'GET',
`repos/${ghRepo}/pulls`,
'-f',
'state=open',
'-f',
`head=${headOwner}:${headBranch}`,
]);
if (result.status !== 0) {
const err = (result.stderr || result.stdout || '').trim();
throw new Error(`GitHub pulls?head= lookup failed: ${err || `exit ${result.status}`}`);
}
const stdout = (result.stdout || '').trim();
if (!stdout) {
throw new Error('GitHub pulls?head= lookup returned an empty body');
}
let parsed;
try {
parsed = JSON.parse(stdout);
} catch {
throw new Error('GitHub pulls?head= lookup returned non-JSON');
}
return flattenGhListPages(parsed);
}
function main() {
const schemaPattern = compileSchemaPattern(process.env.SCHEMA_PATTERN);
const raw = fs.readFileSync(process.env.META_PATH, 'utf8');
const meta = allowlistMetadata(raw, schemaPattern);
const authority = allowlistAuthority({
head_sha: process.env.WF_HEAD_SHA,
head_repo: process.env.WF_HEAD_REPO,
head_branch: process.env.WF_HEAD_BRANCH,
base_repo: process.env.GH_REPO,
});
const pulls = listOpenPullsByHead({
ghRepo: authority.base_repo,
headOwner: forkHeadOwner(authority.head_repo),
headBranch: authority.head_branch,
});
const verified = resolveVerifiedPullRequest({ meta, authority, pulls, schemaPattern });
if (verified.branch_moved) {
console.log(
`PR head moved to ${verified.current_head_sha}; delivering against built SHA ${verified.head_sha} (lease will refuse if the branch moved).`,
);
}
console.log(
`Verified identity: PR=${verified.pr_number} head_sha=${verified.head_sha} head_repo=${verified.head_repo} head_ref=${verified.head_ref}.`,
);
const out = process.env.GITHUB_OUTPUT;
if (!out) {
throw new Error('GITHUB_OUTPUT is unset');
}
fs.appendFileSync(
out,
[
`pr_number=${verified.pr_number}`,
`head_ref=${verified.head_ref}`,
`head_sha=${verified.head_sha}`,
`head_repo=${verified.head_repo}`,
].join('\n') + '\n',
);
}
if (require.main === module) {
try {
main();
} catch (err) {
console.error(`::error::${err instanceof Error ? err.message : String(err)}`);
process.exit(1);
}
}
module.exports = {
SCHEMA_PATTERN,
IDENTITY_PATTERNS,
allowlistField,
compileSchemaPattern,
allowlistMetadata,
allowlistAuthority,
forkHeadOwner,
verifyArtifactAgainstWorkflowRun,
matchOpenPullsFromForkHead,
flattenGhListPages,
resolveVerifiedPullRequest,
listOpenPullsByHead,
main,
};

9
.github/vendored-grammars.json vendored
View file

@ -6,6 +6,11 @@
"upstream": { "npm": "tree-sitter-c" },
"hold": "ABI-pinned at 0.21.4 (#1242/#858) — needs a tree-sitter runtime upgrade before bumping"
},
"objc": {
"name": "tree-sitter-objc",
"upstream": { "npm": "tree-sitter-objc" },
"hold": "Pinned at 3.0.2 for the Objective-C provider MVP; carries darwin/linux arm64+x64 prebuilds compatible with the current tree-sitter runtime (linux-arm64 built from vendored source because the upstream npm artifact is mislabeled)"
},
"swift": {
"name": "tree-sitter-swift",
"upstream": { "npm": "tree-sitter-swift" }
@ -22,6 +27,10 @@
"proto": {
"name": "tree-sitter-proto",
"upstream": { "github": "coder3101/tree-sitter-proto" }
},
"zig": {
"name": "tree-sitter-zig",
"upstream": { "npm": "@tree-sitter-grammars/tree-sitter-zig" }
}
}
}

86
.github/workflows/build-tree-sitter-prebuilds.yml vendored
View file

@ -7,7 +7,7 @@ name: Build tree-sitter prebuilds
#
# Grammars covered here (the at-risk set — everything else already ships 6
# upstream prebuilds AND stays dependency-review-tracked, so it is left alone).
# All five are vendored under gitnexus/vendor/; `kind` (below) only picks where
# All seven are vendored under gitnexus/vendor/; `kind` (below) only picks where
# the build job fetches the C source to compile:
# - tree-sitter-c (vendored prebuild-only; built from the published npm
# package — closes upstream's 4/6 ARM gap #2116 for a
@ -17,15 +17,22 @@ name: Build tree-sitter prebuilds
# - tree-sitter-kotlin (vendored source; built from gitnexus/vendor/ — pinned to
# an unreleased main commit for `fun interface` support
# (#169) that no npm release carries yet)
# - tree-sitter-objc (vendored source; built from gitnexus/vendor/ — pinned
# for the Objective-C provider MVP)
# - tree-sitter-swift (vendored source; built from gitnexus/vendor/ — its
# prebuilds were originally upstream-shipped, now
# GitNexus-cross-built like the rest for uniformity)
# - tree-sitter-zig (vendored source; built from gitnexus/vendor/ — moved
# off npm optionalDependency so `npm i -g gitnexus`
# no longer warns on peerOptional tree-sitter@^0.22.1.
# Upstream linux-arm64 prebuild is a mispackaged
# x86-64 binary; this workflow rebuilds all seven.)
#
# Output: gitnexus/vendor/<grammar>/prebuilds/<platform-arch>/<grammar>.node for
# all 6 targets ({linux,darwin,win32}-{x64,arm64}). tree-sitter grammars are
# N-API, so one ABI-stable .node per platform-arch works across all Node majors.
#
# COST DISCIPLINE — this is a HEAVY native matrix (up to 3 grammars x 6 runners,
# COST DISCIPLINE — this is a HEAVY native matrix (up to 7 grammars x 6 runners,
# incl. macOS + arm64). It is DELIBERATELY NOT wired into normal PR/push CI. It
# runs only:
# 1. on manual dispatch (workflow_dispatch); or
@ -33,8 +40,9 @@ name: Build tree-sitter prebuilds
# OR an edit to the grammar's build-affecting source (parser.c / grammar.js /
# binding.gyp / scanner / bindings). The `guard` job is the real gate (it
# diffs BOTH the recorded version AND the source files vs the PR base); the
# `paths:` filter below keeps ordinary code PRs at ZERO matrix time and
# excludes the prebuilds the job commits back, so it never retriggers itself.
# `paths:` filter below keeps ordinary code PRs at ZERO matrix time. PR
# filters see the cumulative diff, so the guard separately skips updates
# containing only the prebuilds the job commits back.
# Net effect: an ordinary code PR triggers nothing; touching one grammar's source
# costs exactly one matrix run for that grammar. Delivery of the rebuilt binaries:
# - same-repo PR -> committed straight onto the PR's own branch (in the SAME PR);
@ -55,7 +63,7 @@ on:
workflow_dispatch:
inputs:
grammars:
description: 'Comma-separated grammar shortnames to build (c,dart,proto,kotlin,swift), or "all".'
description: 'Comma-separated grammar shortnames to build (c,dart,proto,kotlin,objc,swift,zig), or "all".'
required: false
type: string
default: 'all'
@ -80,13 +88,14 @@ on:
# Any build-affecting change under a vendored grammar triggers a rebuild —
# not just a version bump — so editing the vendored source (parser.c,
# grammar.js, binding.gyp, scanner, bindings) re-cuts the prebuilds too.
# The prebuilds we commit back are EXCLUDED (negated last) so the bot's own
# in-PR commit can never retrigger this workflow (no build->commit->build loop).
# Excludes PRs containing only prebuilds. A source PR still matches after a
# bot commit because PR filters use the cumulative diff; `guard` stops the
# build->commit->build loop using the synchronize event's before/head diff.
- 'gitnexus/vendor/tree-sitter-*/**'
- '!gitnexus/vendor/tree-sitter-*/prebuilds/**'
# Self-test: re-run the guard if a future grammar pin is reintroduced in
# the main package.json (optionalDependencies fallback). No-op otherwise —
# all five grammars are now fully vendored (kotlin included).
# all seven grammars are now fully vendored (including kotlin, objc, and zig).
- 'gitnexus/package.json'
# Self-test: re-run the guard (normally a no-op) when the recipe changes.
- '.github/workflows/build-tree-sitter-prebuilds.yml'
@ -123,6 +132,9 @@ jobs:
id: decide
env:
EVENT: ${{ github.event_name }}
ACTION: ${{ github.event.action }}
BEFORE_SHA: ${{ github.event.before }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
# Untrusted dispatch inputs — read via env only, validated in JS.
INPUT_GRAMMARS: ${{ inputs.grammars }}
INPUT_REF: ${{ inputs.ref }}
@ -131,7 +143,7 @@ jobs:
run: |
set -euo pipefail
node --input-type=module - <<'NODE'
import { execSync } from 'node:child_process';
import { execFileSync, execSync } from 'node:child_process';
import fs from 'node:fs';
import { appendFileSync } from 'node:fs';
@ -153,10 +165,18 @@ jobs:
// unreleased main commit for `fun interface` support (#169) that no
// npm release carries yet — so it must build from the vendored source.
kotlin: { name: 'tree-sitter-kotlin', kind: 'vendored' },
// Objective-C is vendored WITH its source and its native bindings
// must be recut together with the pinned grammar snapshot.
objc: { name: 'tree-sitter-objc', kind: 'vendored' },
// swift is vendored WITH its source (parser.c/scanner.c/binding.gyp),
// so it builds from gitnexus/vendor/ like dart/proto. Its prebuilds
// were originally upstream-shipped; rebuilding them here unifies it.
swift: { name: 'tree-sitter-swift', kind: 'vendored' },
// zig is vendored WITH its source (parser.c/binding.gyp). Moved off
// the npm optionalDependency so published installs no longer warn
// on peerOptional tree-sitter@^0.22.1. Upstream linux-arm64
// prebuild is a mispackaged x86-64 binary; rebuild here.
zig: { name: 'tree-sitter-zig', kind: 'vendored' },
};
const PLATFORMS = [
{ platform_arch: 'linux-x64', os: 'ubuntu-24.04' },
@ -187,6 +207,30 @@ jobs:
const event = process.env.EVENT;
const force = process.env.FORCE === 'true';
// PR path filters and the source/version checks below see the entire
// PR, so excluding prebuilds there does NOT prevent a rebuild loop.
// Check the whole push (not HEAD^ or the author's identity): a push
// containing source edits followed by a binary commit must still build.
if (event === 'pull_request' && process.env.ACTION === 'synchronize') {
const before = process.env.BEFORE_SHA;
const head = process.env.HEAD_SHA;
for (const sha of [before, head]) {
if (!sha || !/^[0-9a-fA-F]{40}$/.test(sha)) {
throw new Error('synchronize requires valid before/head SHAs; refusing an unbounded rebuild');
}
}
// Fail closed if either commit is unavailable. Never fall back to
// the cumulative PR diff, which would re-enable the loop.
const changed = execFileSync('git', [
'diff', '--name-only', '--no-renames', '-z', before, head, '--',
], { encoding: 'utf8' }).split('\0').filter(Boolean);
if (changed.every((p) => /^gitnexus\/vendor\/tree-sitter-[^/]+\/prebuilds\//.test(p))) {
appendFileSync(process.env.GITHUB_OUTPUT, 'any=false\nmatrix={"include":[]}\n');
console.log('::notice::Push changes only prebuild outputs (or no files) — skipping native matrix.');
process.exit(0);
}
}
// Select which grammar shortnames are in play.
let selected;
if (event === 'workflow_dispatch') {
@ -243,9 +287,9 @@ jobs:
} else {
// pull_request: build when the recorded version changed OR any
// build-affecting source file under the vendored grammar changed vs
// the PR base. The prebuilds/ subtree is excluded from the diff so
// the bot's own in-PR commit (which adds ONLY prebuilds) never reads
// as a source change — this is the other half of the no-loop guard.
// the PR base. Exclude generated outputs from build inputs; the
// synchronize check above prevents rebuilding the original source
// change after every generated-prebuild commit.
const base = recordedVersion(baseRoot, name);
const versionChanged = !!head && head !== base;
let sourceChanged = false;
@ -358,7 +402,7 @@ jobs:
- name: Ensure Python (arm64 Windows only)
if: matrix.platform_arch == 'win32-arm64'
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: '3.12'
@ -414,7 +458,13 @@ jobs:
# prebuilds/<platform>-<arch>/<something>.node.
( cd "$pkgdir" && npx --no-install prebuildify --napi --strip )
out=$(find "$pkgdir/prebuilds" -name '*.node' -print -quit)
# `|| true` so the `test -n` below is the thing that reports a missing
# prebuild. `rm -rf` above deletes the directory, so a prebuildify
# run that emits nothing without failing leaves `find` searching a
# path that no longer exists — it exits 1 and `-e` would kill the step
# before the `::error::` line, which is exactly the case that line
# exists to explain.
out=$(find "$pkgdir/prebuilds" -name '*.node' -print -quit || true)
test -n "$out" || { echo "::error::prebuildify produced no .node"; exit 1; }
produced=$(basename "$(dirname "$out")")
[ "$produced" = "$PLATFORM_ARCH" ] || { echo "::error::built $produced, expected $PLATFORM_ARCH"; exit 1; }
@ -460,12 +510,16 @@ jobs:
dart: "void main() { print(\"hi\"); }",
proto: "syntax = \"proto3\";\nmessage M { int32 id = 1; }",
kotlin: "fun main() { println(\"hi\") }",
objc: "@interface GNValidationProbe : NSObject\n@end",
swift: "func greet() { print(\"hi\") }",
zig: "pub fn main() void {}",
};
const src = snippets[process.env.GRAMMAR];
if (!src) throw new Error("no validate snippet for grammar: " + process.env.GRAMMAR);
const lang = require("node-gyp-build")(process.cwd());
const Parser = require("tree-sitter");
const p = new Parser(); p.setLanguage(lang);
const tree = p.parse(snippets[process.env.GRAMMAR]);
const tree = p.parse(src);
if (!tree || !tree.rootNode || tree.rootNode.hasError) {
throw new Error("parse failed/error: " + (tree && tree.rootNode && tree.rootNode.type));
}
@ -561,7 +615,7 @@ jobs:
NODE
- name: Attest build provenance (SLSA)
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1
uses: actions/attest-build-provenance@4d101475d8b20a2381f78447822ac1eab6504dd8 # v4.2.2
with:
subject-path: 'gitnexus/vendor/tree-sitter-*/prebuilds/**/*.node'

2
.github/workflows/ci-e2e.yml vendored
View file

@ -17,7 +17,7 @@ jobs:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- uses: dorny/paths-filter@7b450fff21473bca461d4b92ce414b9d0420d706 # v3
- uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v3
id: filter
with:
filters: |

17
.github/workflows/ci-quality.yml vendored
View file

@ -9,7 +9,9 @@ permissions:
jobs:
format:
runs-on: ubuntu-latest
timeout-minutes: 5
# Same root npm ci as lint. A cold install already took 4m19s here and
# canceled prettier at the 5-minute job cap; lint needed 7m41s the same run.
timeout-minutes: 10
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
@ -19,7 +21,7 @@ jobs:
node-version: 22
cache: npm
cache-dependency-path: package-lock.json
- run: npm ci
- run: npm ci --ignore-scripts
- run: npx prettier --check .
lint:
@ -34,7 +36,7 @@ jobs:
node-version: 22
cache: npm
cache-dependency-path: package-lock.json
- run: npm ci
- run: npm ci --ignore-scripts
- run: npx eslint .
typecheck:
@ -44,13 +46,20 @@ jobs:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
# tsc --noEmit reads source + gitnexus-shared/dist. Skip prepare/postinstall
# so a cold shared install cannot eat the 10-minute budget on a second tsc.
- uses: ./.github/actions/setup-gitnexus
with:
lifecycle-scripts: 'false'
- run: npx tsc --noEmit
working-directory: gitnexus
typecheck-web:
runs-on: ubuntu-latest
timeout-minutes: 10
# Cold gitnexus-web npm ci is several minutes (mermaid/langchain/playwright).
# A 10-minute cancel prevents setup-node from saving the cache, so the next
# run is cold again.
timeout-minutes: 15
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:

24
.github/workflows/ci-report.yml vendored
View file

@ -256,21 +256,37 @@ jobs:
fi
}
# ── Helper: first matching file, tolerating an absent root ──
# `coverage-merge` (ci-tests.yml) is `needs: tests` with no
# `if: always()`, so a failing shard skips it and the `test-reports`
# artifact is never uploaded. A bare `find` on the missing directory
# exits 1; `-o pipefail` carries that through `| head -1` and `-e`
# then killed this step — silently, because stderr is discarded and
# stdout is redirected to $GITHUB_OUTPUT. That skipped "Comment on
# PR" and failed the run precisely when a PR had failing tests, which
# is when the report matters most. Degrade to "" instead so the
# coverage-unavailable fallback below can do its job.
find_first() {
local root=$1 name=$2
[ -d "$root" ] || return 0
find "$root" -name "$name" -type f 2>/dev/null | head -1 || true
}
# ── Read coverage reports ──
UNIT_SUMMARY=$(find "$DIR/test-reports" -name "coverage-summary.json" -type f 2>/dev/null | head -1)
UNIT_SUMMARY=$(find_first "$DIR/test-reports" "coverage-summary.json")
read_cov "U" "$UNIT_SUMMARY"
# ── Read base branch coverage (main) ──
BASE_SUMMARY=""
if [ "$BASE_FOUND" = "true" ] && [ -n "$BASE_DIR" ]; then
BASE_SUMMARY=$(find "$BASE_DIR/base" -name "coverage-summary.json" -type f 2>/dev/null | head -1)
BASE_SUMMARY=$(find_first "$BASE_DIR/base" "coverage-summary.json")
fi
read_cov "B" "$BASE_SUMMARY"
# ── Locate test results ──
RESULTS_FILE=$(find "$DIR/test-reports" -name "test-results.json" -type f 2>/dev/null | head -1)
WEB_RESULTS_FILE=$(find "$DIR/test-reports" -name "web-test-results.json" -type f 2>/dev/null | head -1)
RESULTS_FILE=$(find_first "$DIR/test-reports" "test-results.json")
WEB_RESULTS_FILE=$(find_first "$DIR/test-reports" "web-test-results.json")
sum_results() {
local file=$1

335
.github/workflows/ci-tests.yml vendored
View file

@ -24,8 +24,14 @@ jobs:
shard: ${{ fromJSON(needs.shard-plan.outputs.cov_shards) }}
# Fail loudly (don't silently skip) if the FTS extension is unavailable, so
# FTS-dependent lbug integration suites are guaranteed to run in CI.
# Same contract for Zig's vendored grammar: this runner is linux-x64,
# which vendor/tree-sitter-zig ships a prebuild for, so an absent grammar
# here is a packaging regression and not an unsupported platform. Without
# it every Zig suite skips and the job is green having never executed the
# native Zig parser once.
env:
GITNEXUS_REQUIRE_FTS: '1'
GITNEXUS_REQUIRE_ZIG: '1'
steps:
# persist-credentials: false — runs tests + uploads a blob artifact; the
# default-persisted token must not be capturable through it (zizmor
@ -278,8 +284,16 @@ jobs:
shell: bash
run: python3 .github/scripts/check-tree-sitter-upgrade-readiness.py --assert-current
# GITNEXUS_REQUIRE_ZIG=1: every OS in this matrix has a committed
# vendored tree-sitter-zig prebuild (linux-arm64 is rebuilt by the
# prebuild workflow; ubuntu/windows/macos latest are x64/arm64 with
# shipped binaries), so the smoke's "optional grammar may be absent"
# exemption is revoked here and an ABI-broken Zig binding fails the
# job instead of being accepted as a clean absence.
- name: Run parser-loader ABI load-smoke (dynamic)
run: npx vitest run test/unit/parser-loader-abi.test.ts
env:
GITNEXUS_REQUIRE_ZIG: '1'
working-directory: gitnexus
# End-to-end smoke test for the #1728 packaging fix: pack the published
@ -295,7 +309,9 @@ jobs:
matrix:
os: [windows-latest, ubuntu-latest]
runs-on: ${{ matrix.os }}
timeout-minutes: 15
# Windows pack + web install regularly exceeds 15 minutes when setup also
# runs prepare/postinstall/build before prepack compiles the same tree again.
timeout-minutes: 20
steps:
# persist-credentials: false — this job runs npm pack + npm install -g
# from a tarball and never pushes back; the token in .git/config would
@ -304,9 +320,20 @@ jobs:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
# Skip prepare/postinstall/build here. `npm pack` runs prepack, which
# compiles CLI + web into the tarball this job actually installs.
- uses: ./.github/actions/setup-gitnexus
with:
build: 'true'
lifecycle-scripts: 'false'
# `npm pack` runs prepack, which builds the web UI into gitnexus/web/
# so the tarball matches what `npm publish` ships. Install those deps
# here, in their own visible step, rather than letting build.js do it
# from inside an execSync.
- name: Install gitnexus-web dependencies
shell: bash
run: npm ci
working-directory: gitnexus-web
- name: Pack gitnexus tarball
shell: bash
@ -348,6 +375,10 @@ jobs:
fi
echo "Installed package at: $INSTALLED"
# The npm package contract includes the built web UI. Validate the
# installed artifact, not just the source workflow that produced it.
node "$INSTALLED/scripts/assert-web-assets.mjs" "$INSTALLED/web"
# #836 invariant: no node_modules/ or build/ under any vendor/*.
BAD=$(find "$INSTALLED/vendor" \( -name node_modules -o -name build \) -print 2>/dev/null || true)
if [ -n "$BAD" ]; then
@ -407,9 +438,6 @@ jobs:
node-version: '22'
cache: npm
cache-dependency-path: gitnexus/package-lock.json
- name: Build gitnexus-shared
run: npm ci && npm run build
working-directory: gitnexus-shared
- name: Install and build gitnexus
shell: bash
run: |
@ -481,7 +509,29 @@ jobs:
node --import tsx bench/python-scope/import-target-fingerprint.mjs --check
working-directory: gitnexus
- name: Java wildcard-static route constant guards (#3110)
if: ${{ !cancelled() }}
# Build-free: named-import control vs wildcard materialization;
# fingerprints bindings and guards scaling + absolute wall time.
run: node --import tsx bench/java-wildcard-route-constants/measure.mjs --check
working-directory: gitnexus
- name: Kotlin package-star route constant guards (#3110)
if: ${{ !cancelled() }}
# Build-free: explicit-import control vs package-star folding;
# fingerprints route facts and guards scaling + widening overhead.
run: node --import tsx bench/kotlin-star-route-constants/measure.mjs --check
working-directory: gitnexus
- name: Cross-language scope-capture fingerprint + scaling guards
# Runs even after an earlier guard fails (#2895). Every step here was
# fail-fast, so the FIRST failing --check aborted the job and every guard
# after it reported `skipped` — which reads identically to "nothing to do".
# Audited across 13 benchmark runs on #2856: the job succeeded zero times
# and the last two guards executed zero times for the life of the PR, while
# two reviews read the checks summary and saw nothing wrong. `!cancelled()`
# rather than `always()` so an explicit cancel still stops the job.
if: ${{ !cancelled() }}
# Build-free: asserts emit<Lang>ScopeCaptures output is unchanged
# (fingerprint) and stays linear (scaling < 1.5) for go/csharp/rust/php/
# ruby/cobol. Catches an O(n^2) re-regression without the worker pool.
@ -489,6 +539,7 @@ jobs:
working-directory: gitnexus
- name: Callable-value-flow target-index guards (#2693)
if: ${{ !cancelled() }}
# Build-free: asserts buildGraphTargetIndex resolves an unchanged target
# set (fingerprint), stays linear in def count, and that the #2693
# widened gate — which now considers VALUE bindings, a population that
@ -500,7 +551,202 @@ jobs:
run: node --import tsx bench/callable-value-flow/measure.mjs --check
working-directory: gitnexus
- name: Java Lombok accessor synthesis guards (#2885)
if: ${{ !cancelled() }}
# Build-free: no-Lombok vs Lombok-heavy corpora; fingerprint over
# synthetic Method ids; scaling + widening overhead budgets.
run: node --import tsx bench/java-lombok-synthesis/measure.mjs --check
working-directory: gitnexus
- name: Kotlin JVM accessor synthesis guards (#2885)
if: ${{ !cancelled() }}
# Build-free: no-property vs data-class corpora; fingerprint over
# synthetic Method ids; scaling + widening overhead budgets.
run: node --import tsx bench/kotlin-jvm-accessors/measure.mjs --check
working-directory: gitnexus
- name: Kotlin Spring config-consumer capture guards (#2412)
if: ${{ !cancelled() }}
# Build-free: explicit-import control vs wildcard-import feature path;
# fingerprints @Value / @ConfigurationProperties facts and guards scaling
# + widening overhead. The parity check is the regression gate: each file
# declares a sibling nested type named `Value`, which must not suppress
# the imported Spring annotation (file-wide shadowing dropped 2 of every
# 3 facts on this corpus).
run: node --import tsx bench/spring-config-bindings/measure.mjs --check
working-directory: gitnexus
- name: Re-export closure scaling guards (#2864)
# Build-free: asserts buildReexportClosures stays linear in chain depth
# and within an absolute ceiling on a wide package corpus. #2864 changed
# this pass's input class from TypeScript barrels (a handful of shallow
# edges) to every module-level Python `from m import x`, which is where
# its two quadratic corners became reachable. The depth arm specifically
# guards MAX_VIA_LENGTH — the bound that was removed once already, in
# fc919ad6, and stayed invisible for as long as the input was shallow.
run: node --import tsx bench/finalize-reexport/measure.mjs --check
working-directory: gitnexus
- name: Parse dispatch-round cadence guards (#3194, #3196)
if: ${{ !cancelled() }}
# Build-free: asserts parse-cache pack membership is unchanged
# (fingerprint — every cache key derives from it), that a fixed corpus
# still batches into a fixed number of dispatch rounds, and that the
# round budget counts UTF-8 bytes rather than UTF-16 code units. Round
# boundaries are deliberately invisible to graph output, so no test can
# see these regress. Rationale and history: see the header of
# bench/parse-dispatch-rounds/measure.mjs.
run: node --import tsx bench/parse-dispatch-rounds/measure.mjs --check
working-directory: gitnexus
- name: Python workspace import-scan guards (#3254)
if: ${{ !cancelled() }}
# Build-free: same baseline approach as parse-dispatch-rounds —
# exact link/lookalike floors plus a fingerprint, then ratio timing
# only (scan scaling and from-token prefilter advantage). See
# bench/python-workspace-import-scan/measure.mjs.
run: node --import tsx bench/python-workspace-import-scan/measure.mjs --check
working-directory: gitnexus
- name: MCP tools/list countRepos vs listRepos guards (#3259, #3184)
if: ${{ !cancelled() }}
# Build-free: exact registry cardinality + tool-roster + schema-flag
# floors, then ratio timing only (countRepos/listRepos and
# listTools/listRepos). No millisecond ceiling — this repo has
# already been bitten by a fixed ms budget. Isolated GITNEXUS_HOME;
# fixture is N real git repos so listRepos pays rev-list. See
# bench/mcp-tools-list/measure.mjs.
run: node --import tsx bench/mcp-tools-list/measure.mjs --check
working-directory: gitnexus
- name: C++ qualified-namespace resolution guards (#2788)
if: ${{ !cancelled() }}
# Build-free: asserts resolveCppQualifiedNamespaceMember resolves an
# unchanged symbol set (fingerprint) and that per-call-site cost stays
# independent of corpus size. Rationale and history: see the header of
# bench/cpp-qualified-ns/measure.mjs.
run: node --import tsx bench/cpp-qualified-ns/measure.mjs --check
working-directory: gitnexus
- name: Import-target resolution guards (every registered language, #2877–#2909, PR #2911)
if: ${{ !cancelled() }}
# Build-free: runs EVERY import-target resolver registered in
# SCOPE_RESOLVERS — plus C# a second time WITH csproj configs, over the
# identical corpus, because the no-csproj arm returns before it can
# reach the leg #2902 indexed. One arm per registered language over ONE
# shared corpus, and no registered language ungated. That is enforced,
# not enumerated: measure.mjs derives its list from a LANG_REGISTRY
# table and its --check inventory arm reconciles that table against
# SCOPE_RESOLVERS in both directions, so a language roster typed out
# here would only be a second copy that can go stale — this one did.
# A C/C++ #include is an import site for this purpose and is gated like
# every other registered language (its headers arrive through
# resolutionConfig rather than allFilePaths, which is the one structural
# difference — see `newPass`).
#
# Asserts each returns an unchanged target set (a fingerprint per
# language AND per arm), that per-import cost stays independent of
# corpus size AND of path depth, that the absolute small-arm cost holds
# — a constant-factor regression that grows both scale arms equally
# passes every ratio — and that the per-pass index eight of them retain
# stays within an absolute byte ceiling. The corpus SHAPE is asserted
# too: a fingerprint alone cannot tell a legitimate resolution change
# from a corpus quietly shrunk below the size the timing arms need.
#
# Several arms exist because an arm that stops MEASURING otherwise
# passes. The heap arms drive real resolvers and carry a FLOOR as well
# as a ceiling: when buildSuffixIndex's suffix maps went lazy, four arms
# that called the builder directly read 0 B, and 0 B is under every
# ceiling. EVERY budget is checked for PRESENCE first, timing and heap
# alike, because `got > undefined` is false and `got < ceiling *
# undefined` is false too, so deleting a budget key deleted its gate —
# and the two heap scalars gate all eight heap arms at once. The heap
# arm's own corpus shape (its two file counts, its path depth and the
# probe it resolves) is asserted by the same loop as the timing arms,
# because those four decide WHAT it measures. And an inventory arm
# reconciles the bench's language table against SCOPE_RESOLVERS itself,
# so a newly registered resolver cannot ship ungated the way JavaScript
# did.
#
# The resolvers gated first were added as their own O(imports × files)
# scans were indexed away (Ruby rebuilt a suffix index per `require`;
# COBOL scanned twice per `COPY`), and the same corpus shape scores >3.3
# against those pre-fix implementations. The rest were ungated until
# this PR, which is not a theoretical gap: PR #2911 found JavaScript
# reaching suffixResolve with no index at all — 25 972 µs per import at
# 8000 files, protected only by unit tests. This step is what stops the
# next one shipping.
#
# SCOPE: "independent of corpus size" holds for UNIQUE-LEAF layouts,
# where no two directories share a last segment and no two files share a
# basename — which is what the small/large/deep arms are, and where
# every index bucket holds exactly one entry. The `collide` arm runs the
# identical workload on the layout these languages are actually written
# in (svcN/internal, SrcN/Models, a repeated basename per package, four
# SPM modules instead of fifty); there the bucket grows with the file
# count by construction and go, csharp, dart, java, swift and c/cpp
# legitimately score 2.1–3.9, so that arm carries its own per-language
# budget. It is a scope limit, not a regression — the indexed code is
# still faster on that shape than the pre-change scan. Rust is the one
# language whose collide arm is NOT a shared-leaf layout: it probes
# candidate paths and is provably flat in the file count, so its arm is
# a deep module tree that varies `::` segment count instead — the axis
# its cost actually has.
#
# --expose-gc enables the retained-heap arm; --check REFUSES to run
# without it rather than passing with the memory gate silently skipped.
# ~44–45 s, which is essentially unchanged from the ~46 s it cost
# before: the timing phase did fall from 39.8 s to 28.7 s when the
# min-of-N estimator became per-language, but the inventory arm's one
# dynamic import (pipeline/registry.ts pulls in every registered
# provider) costs 6–10 s depending on the box and consumes almost all of
# that. Report mode, which does not load the registry, is the mode that
# got faster: ~33–35 s. Kept as-is because this job runs minutes clear
# of the sharded coverage job that gates the merge, so the seconds buy
# no merge latency — see COST in the bench header. The ts
# family (javascript/typescript/vue) is still the largest block, 8.8 s,
# because suffixResolve probes ~39 extensions per path part on a miss.
# If this ever has to shrink, drop collide/collide_large for typescript
# and vue (−3.9 s) — the only cut that removes near-duplicate work
# rather than coverage. N is 15 (matching bench/cfg) for every language
# whose cheapest arm is under 5 ms, because depth_ratio divides two
# sub-3 ms numbers and at 5 or 7 it tripped its own budget roughly 1 run
# in 20; the six languages whose cheapest arm is 20-28 ms drop to 7-8,
# where the measured overshoot is at most 6.3%. The estimator was fixed
# rather than the budget widened; distributions in _arms_note.
# The Kotlin arm here is a second corpus, not a replacement for the
# kotlin-import-target bench below, which carries declared-package
# correctness probes this shared corpus does not.
# It sits with the other resolver-index guards rather than at the end of
# the job: parking a new gate last is not safety, it is the slot least
# likely to execute (#2895 measured the last two guards running zero
# times in 13 runs). #2899 landed the `if: ${{ !cancelled() }}` below,
# which is what makes position irrelevant — a failing step no longer
# aborts the ones after it.
# Rationale, budgets and the measured blind spot: see the header of
# measure.mjs and _blind_spot in baselines.json.
run: node --expose-gc --import tsx bench/import-target/measure.mjs --check
working-directory: gitnexus
- name: Kotlin declared-package import correctness + scaling guards
if: ${{ !cancelled() }}
# Build-free: fingerprints declared-package evidence, external decoy
# rejection, top-level/member/wildcard imports, overload sets and root
# packages, then guards one package-index build per workspace against
# file-count and path-depth scaling. Rationale and history: see the
# header of bench/kotlin-import-target/measure.mjs.
run: node --import tsx bench/kotlin-import-target/measure.mjs --check
working-directory: gitnexus
- name: Ruby gem-boundary correctness + scaling guards (#3096)
if: ${{ !cancelled() }}
# Includes real manifest loading; checks scoped resolution and scaling
# as sibling projects or declared gem counts grow independently.
run: node --import tsx bench/ruby-gem-resolution/measure.mjs --check
working-directory: gitnexus
- name: Receiver-resolution drop guards
if: ${{ !cancelled() }}
# NOT build-free: this one runs the real pipeline, so it needs dist/
# (the setup action above builds). ~2m15s.
#
@ -509,13 +755,24 @@ jobs:
# recorder gates on the receiver's punctuation, not on what the
# reference is, so property reads would inflate it by ~20%. The shape
# arm asserts the state of each receiver spelling by EDGE PRESENCE,
# which is the only arm that can see the shapes the recorder is blind
# to (`?.`, explicit type args, `repos[0]`): those emit no edge AND no
# drop, so fixing them moves the count by zero.
# which is the only arm that can see shapes the recorder is blind to:
# they emit no edge AND no drop, so fixing them moves the count by zero.
#
# `repos[0]` is no longer among them (#2766): Case 0's gate now accepts
# a minted receiver chain instead of testing the receiver's punctuation,
# so subscript receivers record a drop and ARE countable. 13 shapes moved
# INVISIBLE -> VISIBLE that way. `?.` and explicit type args remain
# invisible on some languages, so the shape arm still earns its keep.
#
# The check is EXACT-MATCH, which is strictly stronger than a ratchet:
# the count cannot rise without a deliberate rebaseline, and the
# rebaseline path demands the movement be explained. No separate
# drop-ratchet gate is needed on top of this.
run: node --import tsx bench/receiver-resolution/measure.mjs --check
working-directory: gitnexus
- name: Scope-emission guards (#2699)
if: ${{ !cancelled() }}
# Build-free: asserts the JS/TS scope set is unchanged. Block scopes are
# what make `let`/`const` in sibling blocks distinct bindings, but a
# scope per `statement_block` triples the count and deepens every
@ -527,7 +784,37 @@ jobs:
run: node --import tsx bench/scope-emission/measure.mjs --check
working-directory: gitnexus
- name: Zig cross-file static-gating guards (#3162)
if: ${{ !cancelled() }}
# Build-free: fingerprints cross-file dead-call classification and
# guards the workspace enrichment pass across file-count scaling.
run: node --import tsx bench/zig-cross-file-resolution/measure.mjs --check
working-directory: gitnexus
- name: Objective-C workspace resolution guards (#3179)
if: ${{ !cancelled() }}
# Build-free: fingerprints spread (typed self/super/sibling) and
# protocol-candidate evidence, and gates linear file-count scaling
# of emitPostResolutionEdges. Import lookup is the shared
# import-target `objc` arm; this is the C#/Zig analog for the
# workspace message-send pass.
run: node --import tsx bench/objective-c-resolution/measure.mjs --check
working-directory: gitnexus
- name: Callable-value reference resolution guards (#3399)
if: ${{ !cancelled() }}
# Build-free: pins the resolved-target SET of `resolveValueRefTarget`
# (exact site/resolved/declined counts plus an order-independent
# fingerprint) and asserts its per-site cost stays independent of
# workspace size across a 4x file-count step. The pass resolves a
# qualified receiver through `scopes.qualifiedNames`, a workspace-wide
# index: keyed it is O(1) per site, scanned it is O(files) — a
# regression a fixture cannot see and a 257-file binding table can.
run: node --import tsx bench/value-ref-resolution/measure.mjs --check
working-directory: gitnexus
- name: CFG construction time / disk / memory guards (#2081 M1)
if: ${{ !cancelled() }}
# Build-free: asserts collectFunctionCfgs output is unchanged
# (fingerprint) and that wall-time, cfgSideChannel disk bytes, AND
# retained heap all stay sub-quadratic for the straight-line /
@ -538,6 +825,7 @@ jobs:
working-directory: gitnexus
- name: Emit-persistence throughput / byte-identity guards (#2203)
if: ${{ !cancelled() }}
# Build-free: asserts streamAllCSVsToDisk output is byte-identical
# (order-independent CSV-line fingerprint — the #2203 U2/U3 emit
# optimisations must not change graph content) and that emit wall-time
@ -547,6 +835,7 @@ jobs:
working-directory: gitnexus
- name: Streaming PDG-emit byte-identity / bounded-RSS guards (#2202)
if: ${{ !cancelled() }}
# Build-free: asserts the streaming PdgEmitSink emits a CSV row SET
# byte-identical to the whole-graph streamAllCSVsToDisk emit, AND that
# the in-memory graph retains zero BasicBlock nodes (the O(chunk) peak-RSS
@ -556,14 +845,25 @@ jobs:
working-directory: gitnexus
- name: Cross-language pipeline benchmarks (GITNEXUS_BENCH, serial)
if: ${{ !cancelled() }}
# cpp-adl-benchmark.test.ts and csharp-razor-view-components-benchmark.test.ts
# are not `*-pipeline-benchmark.test.ts` files but belong here for the
# same reason: they are skipIf-gated on GITNEXUS_BENCH, so the scaling
# guards they hold never run in the main coverage job.
env:
GITNEXUS_BENCH: '1'
GITNEXUS_WORKER_READY_TIMEOUT_MS: '60000'
run: >-
npx vitest run --no-file-parallelism
test/integration/cobol-pipeline-benchmark.test.ts
test/integration/csharp-pipeline-benchmark.test.ts
test/integration/csharp-razor-view-components-benchmark.test.ts
test/integration/objective-c-pipeline-benchmark.test.ts
test/integration/cpp-adl-benchmark.test.ts
test/integration/data-route-table-benchmark.test.ts
test/integration/instance-ownership-pipeline-benchmark.test.ts
test/integration/spring-bean-resource-benchmark.test.ts
test/integration/spring-dynamic-lookup-benchmark.test.ts
test/integration/rust-pipeline-benchmark.test.ts
test/integration/php-pipeline-benchmark.test.ts
test/integration/ruby-pipeline-benchmark.test.ts
@ -597,6 +897,11 @@ jobs:
timeout-minutes: 20
env:
GITNEXUS_REQUIRE_BWRAP_CANARY: '1'
# This job installs bubblewrap, the pinned runtime and a built GitNexus,
# so the offline sweep runs here with nothing provisioning-stubbed: real
# containment, real mounts, real graph. A missing piece fails the job
# rather than silently falling back to the stubbed path.
GITNEXUS_REQUIRE_FULL_SWEEP: '1'
GITNEXUS_REQUIRE_CLAUDE_CANARY: '1'
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
@ -619,7 +924,7 @@ jobs:
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get install --yes --no-install-recommends bubblewrap socat
sudo apt-get install --yes --no-install-recommends bubblewrap ripgrep socat
apparmor_userns=/proc/sys/kernel/apparmor_restrict_unprivileged_userns
if [[ -r "${apparmor_userns}" ]] && [[ "$(<"${apparmor_userns}")" == '1' ]]; then
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
@ -642,11 +947,6 @@ jobs:
"${canary_runtime}/node_modules/@anthropic-ai/claude-code/package.json"
test "$("${canary_runtime}/node_modules/@anthropic-ai/claude-code-linux-x64/claude" --version)" = \
'2.1.214 (Claude Code)'
- name: Build pinned shared runtime
run: |
npm ci
npm run build
working-directory: gitnexus-shared
- name: Install and build pinned GitNexus runtime
run: |
npm ci
@ -660,7 +960,9 @@ jobs:
tests/test_process_control.py
tests/test_proposer_sandbox.py
tests/test_workflow_bench_sessions.py
tests/test_ce_plugin_runtime.py -q
tests/test_ce_plugin_runtime.py
tests/test_offline_sweep_integration.py
tests/test_mock_provider.py -q
working-directory: eval
# Native Windows Job Object canary. POSIX-only tests skip by platform, while
@ -682,5 +984,6 @@ jobs:
- name: Prove Windows process-tree ownership
run: >-
uv run --locked --extra dev python -m pytest
tests/test_process_control.py -q
tests/test_process_control.py tests/test_model_gateway.py
-k "not locked_litellm" -q
working-directory: eval

10
.github/workflows/codeql.yml vendored
View file

@ -48,7 +48,7 @@ jobs:
persist-credentials: false
- name: Initialize CodeQL
uses: github/codeql-action/init@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3
uses: github/codeql-action/init@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9
with:
languages: ${{ matrix.language }}
queries: security-and-quality
@ -71,8 +71,14 @@ jobs:
# deliberately contain use-before-init / unused-variable shapes).
- '**/test/fixtures/**'
- '**/test/**/fixtures/**'
# GET /api/grep intentionally builds RegExp from the query string
# (literal=1 escapes). ReDoS is handled by worker terminate() —
# see SECURITY.md. Inline codeql[] comments do not clear the
# GitHub PR CodeQL gate, so this file is excluded to avoid
# re-filing js/regex-injection on every push of the same line.
- 'gitnexus/src/server/grep-params.ts'
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3
uses: github/codeql-action/analyze@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9
with:
category: '/language:${{ matrix.language }}'

79
.github/workflows/commit-fork-prebuilds.yml vendored
View file

@ -135,68 +135,53 @@ jobs:
# workflow_run event. The allowlist above only proves the fields are
# well-formed — not that they refer to the PR/SHA that actually triggered
# us. A fork-controlled build could mutate metadata.json to reference
# another PR/SHA and redirect our write-scoped push. Authority sources are
# all server-controlled: workflow_run.head_sha, head_repository.full_name,
# and pull_requests[].number (empty on forks -> commits/{sha}/pulls).
# another PR/SHA and redirect our write-scoped push.
#
# This job's `if:` already restricts to forks, so pull_requests[] is empty
# by design and GET /repos/{base}/commits/{sha}/pulls is also empty (the
# fork commit is not in the base graph). Authority is workflow_run.head_sha
# + head_repository.full_name + head_branch, resolved via
# pulls?head={owner}:{branch}. The script comes from THIS default-branch
# checkout (the same trust anchor as this workflow file).
- name: Checkout identity verifier
if: steps.meta.outputs.deliver == 'true'
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
sparse-checkout: .github/scripts/verify-workflow-run-pr-identity.cjs
sparse-checkout-cone-mode: false
path: trusted
- name: Verify metadata against workflow_run authority
id: verify
if: steps.meta.outputs.deliver == 'true'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}
META_PR_NUMBER: ${{ steps.meta.outputs.pr_number }}
META_HEAD_SHA: ${{ steps.meta.outputs.head_sha }}
META_HEAD_REPO: ${{ steps.meta.outputs.head_repo }}
META_PATH: meta-in/metadata.json
SCHEMA_PATTERN: '^gitnexus\.ts-prebuild/v[0-9]+$'
WF_HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
WF_HEAD_REPO: ${{ github.event.workflow_run.head_repository.full_name }}
WF_PR_NUMBERS: ${{ toJSON(github.event.workflow_run.pull_requests.*.number) }}
WF_HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
shell: bash
run: |
set -euo pipefail
# 1) head_sha must match exactly — the commit GitHub ran the producer against.
if [ "${META_HEAD_SHA}" != "${WF_HEAD_SHA}" ]; then
echo "::error::Artifact head_sha (${META_HEAD_SHA}) != workflow_run.head_sha (${WF_HEAD_SHA}) — refusing."
exit 1
fi
# 2) head_repo must match exactly.
if [ "${META_HEAD_REPO}" != "${WF_HEAD_REPO}" ]; then
echo "::error::Artifact head_repo (${META_HEAD_REPO}) != workflow_run.head_repository (${WF_HEAD_REPO}) — refusing."
exit 1
fi
# 3) pr_number must reference an open PR with this head SHA. Forks have
# an empty pull_requests[] by design — fall back to commits/{sha}/pulls.
allowed_numbers=$(jq -c '.' <<< "${WF_PR_NUMBERS}")
if [ "${allowed_numbers}" = "[]" ]; then
echo "workflow_run.pull_requests empty (fork) — using commits/{sha}/pulls."
allowed_numbers=$(gh api "repos/${GH_REPO}/commits/${WF_HEAD_SHA}/pulls" \
--jq '[.[] | select(.state == "open") | .number]' 2>/dev/null || echo "[]")
if [ "${allowed_numbers}" = "[]" ]; then
echo "::error::No open PR for head ${WF_HEAD_SHA} — refusing."
exit 1
fi
fi
if ! jq -e --argjson n "${META_PR_NUMBER}" 'index($n) != null' <<< "${allowed_numbers}" >/dev/null; then
echo "::error::Artifact pr_number (${META_PR_NUMBER}) not in authoritative list (${allowed_numbers}) — refusing."
exit 1
fi
echo "Verified identity: PR=${META_PR_NUMBER} head_sha=${META_HEAD_SHA} head_repo=${META_HEAD_REPO}."
run: node trusted/.github/scripts/verify-workflow-run-pr-identity.cjs
# Pinned to v6.0.3 (same SHA used by build-tree-sitter-prebuilds.yml).
# persist-credentials: false — push auth is provided inline at push time,
# never written to .git/config on disk.
- name: Checkout fork PR head
if: steps.meta.outputs.deliver == 'true'
if: steps.meta.outputs.deliver == 'true' && steps.verify.outcome == 'success'
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
repository: ${{ steps.meta.outputs.head_repo }}
ref: ${{ steps.meta.outputs.head_sha }}
repository: ${{ steps.verify.outputs.head_repo }}
ref: ${{ steps.verify.outputs.head_sha }}
token: ${{ secrets.GITHUB_TOKEN }}
persist-credentials: false
fetch-depth: 0
path: pr-checkout
- name: Place prebuilds into the fork checkout
if: steps.meta.outputs.deliver == 'true'
if: steps.meta.outputs.deliver == 'true' && steps.verify.outcome == 'success'
env:
DL: prebuilds-in
CHECKOUT: pr-checkout
@ -238,12 +223,12 @@ jobs:
- name: Commit and push to the fork branch
id: push
if: steps.meta.outputs.deliver == 'true'
if: steps.meta.outputs.deliver == 'true' && steps.verify.outcome == 'success'
working-directory: pr-checkout
env:
HEAD_REF: ${{ steps.meta.outputs.head_ref }}
HEAD_REPO: ${{ steps.meta.outputs.head_repo }}
HEAD_SHA: ${{ steps.meta.outputs.head_sha }}
HEAD_REF: ${{ steps.verify.outputs.head_ref }}
HEAD_REPO: ${{ steps.verify.outputs.head_repo }}
HEAD_SHA: ${{ steps.verify.outputs.head_sha }}
# Push auth only — supplied via env, never interpolated into the command.
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
shell: bash
@ -303,11 +288,11 @@ jobs:
fi
- name: Comment delivery outcome
if: always() && steps.meta.outputs.deliver == 'true' && steps.push.outcome != 'skipped'
if: always() && steps.meta.outputs.deliver == 'true' && steps.verify.outcome == 'success' && steps.push.outcome != 'skipped'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}
PR: ${{ steps.meta.outputs.pr_number }}
PR: ${{ steps.verify.outputs.pr_number }}
RESULT: ${{ steps.push.outputs.result }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
shell: bash

12
.github/workflows/docker.yml vendored
View file

@ -138,17 +138,17 @@ jobs:
# Required for multi-platform (linux/arm64) emulation.
- name: Set up QEMU
uses: docker/setup-qemu-action@96fe6ef7f33517b61c61be40b68a1882f3264fb8 # v4.2.0
uses: docker/setup-qemu-action@1f40c72289eff860ee54a304f1438e3cff362e0a # v4.3.0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
- name: Install Cosign
uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2
- name: Log in to GitHub Container Registry
if: ${{ github.event_name != 'pull_request' && !inputs.dry_run }}
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
@ -163,7 +163,7 @@ jobs:
# `akonlabs/gitnexus` and `akonlabs/gitnexus-web` repos.
- name: Log in to Docker Hub
if: ${{ github.event_name != 'pull_request' && !inputs.dry_run }}
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
@ -256,7 +256,7 @@ jobs:
# pulling from either GHCR or Docker Hub see the same provenance.
- name: Generate build provenance attestation (GHCR)
if: ${{ github.event_name != 'pull_request' && !inputs.dry_run }}
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1
uses: actions/attest-build-provenance@4d101475d8b20a2381f78447822ac1eab6504dd8 # v4.2.2
with:
subject-name: ghcr.io/${{ github.repository_owner }}/${{ matrix.image.slug }}
subject-digest: ${{ steps.build.outputs.digest }}
@ -264,7 +264,7 @@ jobs:
- name: Generate build provenance attestation (Docker Hub)
if: ${{ github.event_name != 'pull_request' && !inputs.dry_run }}
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4.1.1
uses: actions/attest-build-provenance@4d101475d8b20a2381f78447822ac1eab6504dd8 # v4.2.2
with:
subject-name: docker.io/akonlabs/${{ matrix.image.slug }}
subject-digest: ${{ steps.build.outputs.digest }}

371
.github/workflows/gitnexus-skill-evolution.yml vendored
View file

@ -4,17 +4,22 @@
# overlay. The gate is evidence FOR a PR, never a bypass of one — nothing
# merges without review.
#
# Activation checklist (the scheduled lane is OFF by default).
# [ ] Configure the repository secret GITNEXUS_BENCH_AUTH_TOKEN (an Anthropic
# API key — benchmark sessions bill real usage; the Claude Code OAuth
# subscription token does not work here).
# [ ] Configure the RELEASE_APP_ID and RELEASE_APP_PRIVATE_KEY secrets (the
# Activation and operations checklist.
# [x] Configure at least one model secret on the `gitnexus-evolution`
# Environment: GITNEXUS_BENCH_ANTHROPIC_API_KEY (Anthropic API key — not
# the Claude Code OAuth token; legacy GITNEXUS_BENCH_AUTH_TOKEN is still
# accepted) and/or GITNEXUS_BENCH_OPENAI_API_KEY. Sessions bill real usage.
# OpenAI keys are not native to Claude Code; the loop starts a loopback
# LiteLLM proxy and keeps the OpenAI key off the sandboxed agent. With only
# the OpenAI secret, or with provider=openai, dispatch-time Claude model
# defaults are gpt-5.6-sol with xhigh reasoning effort.
# [x] Configure the RELEASE_APP_ID and RELEASE_APP_PRIVATE_KEY secrets (the
# App that opens the promotion PR). The Mint-App-Token step hard-fails
# without them once a promotion is detected. Verify the App installation
# is scoped to this repo with only Contents: RW + Pull requests: RW.
# [x] Create the protected Environment `gitnexus-evolution` with a
# deployment-branch rule restricting it to `main`, and ideally scope the
# three secrets above to that Environment. workflow_dispatch runs this
# four secrets above to that Environment. workflow_dispatch runs this
# workflow (and eval/workflow_bench/evolve.py) from the *dispatched ref*,
# so this server-side rule — not a code-side guard the branch could edit
# away — is what stops a non-main branch from running with the secrets.
@ -40,13 +45,35 @@
# most weekly. Revisit if run frequency increases or the threat model
# changes; stopping already bounds the exposure window to the job's own
# runtime on 1 day out of 7.
# [ ] Run workflow_dispatch once and confirm: containment preflight passes,
# [ ] Install and verify the runner survival policy below before enabling
# scheduled runs. A run
# spans ~15h and apt-daily-upgrade.timer fires daily (~06:34), so every
# scheduled run crosses it. On 2026-08-02 unattended-upgrades upgraded
# openssl at 07:54:02 and needrestart restarted the Actions runner five
# seconds later: the job went to Canceled, and a cancelled job skips even
# `if: always()`, so the evidence artifact died with it. Keep installing
# updates, but never let them restart services here:
# /etc/needrestart/conf.d/90-gitnexus-evolution.conf
# $nrconf{restart} = 'l';
# A drop-in, so a needrestart package upgrade cannot clobber it. Nothing
# is left unpatched in practice — the box is stopped between runs, so the
# new binaries take effect at the next boot.
# [x] Run workflow_dispatch once and confirm: containment preflight passes,
# the benchmark completes inside the job timeout, the results artifact
# uploads, and a promotion (if any) opens a well-formed PR.
# [ ] Set the repository variable GITNEXUS_EVOLUTION_ENABLED=true.
# Roll back by setting that variable to false. Note: workflow_dispatch always
# runs the full benchmark loop regardless of GITNEXUS_EVOLUTION_ENABLED and
# bills real API usage on GITNEXUS_BENCH_AUTH_TOKEN.
# uploads, and a promotion (if any) opens a well-formed PR. Run
# 29907431284 (2026-07-22) went green end to end in 14h45m and reached a
# gate decision (`insufficient_evidence`, no promotion).
# [ ] Confirm a workers=3 dispatch has zero excluded runs (review sessions in
# 33962002890 averaged ~19m serial, well under the 90m session ceiling).
# Then set GITNEXUS_EVOLUTION_WORKERS=3 and
# GITNEXUS_EVOLUTION_ENABLED=true for scheduled runs. Scheduled runs
# require both values, so leaving the var unset is an immediate rollback.
# Dispatch defaults to 3; pass workers=1 only to debug a contended host.
# Weekly generations reuse matching incumbent/CE cells from the previous
# artifact so the paid matrix is the new candidate, not a 54-cell replay.
# Wall clock is quantised by ceil(cells_per_task / workers), and a review
# task is 9 cells cold, so 4 costs host contention for exactly the wall
# clock of 3. The next step up that buys anything is 5 (3 waves -> 2).
name: GitNexus skill evolution
on:
@ -68,21 +95,51 @@ on:
required: false
default: '3'
type: string
workers:
description: 'Benchmark cells of one task to run at once — 3 fits the evolution box; drop to 1 only if siblings hit the session ceiling'
required: false
default: '3'
type: string
model:
description: 'Model for the benchmark arms (match the model your skill users run)'
required: false
default: 'claude-sonnet-5'
default: 'gpt-5.6-sol'
type: string
proposer_model:
description: 'Model for the proposer/diagnosis session — a stronger model is fine (one session per generation)'
required: false
default: 'claude-opus-4-8'
default: 'gpt-5.6-sol'
type: string
effort:
description: 'Reasoning effort for every proposer and benchmark session'
required: false
default: xhigh
type: choice
options:
- low
- medium
- high
- xhigh
- max
provider:
description: 'Model backend. auto uses Anthropic when that secret exists; openai forces the loopback OpenAI gateway even if an Anthropic key is also configured.'
required: false
default: openai
type: choice
options:
- auto
- openai
- anthropic
include_expensive:
description: 'Include tasks marked expensive: true'
required: false
default: false
type: boolean
seed_from_previous:
description: "Seed the proposer with the previous run's evidence and rejected proposal. Turn off to start from a blank slate — required when the earlier evidence is not trustworthy (e.g. produced before a harness-integrity fix), since a tainted proposal would otherwise propagate into every later generation."
required: false
default: true
type: boolean
concurrency:
group: ${{ github.workflow }}
@ -97,31 +154,101 @@ jobs:
github.repository == 'abhigyanpatwari/GitNexus' &&
(
github.event_name == 'workflow_dispatch' ||
vars.GITNEXUS_EVOLUTION_ENABLED == 'true'
(
vars.GITNEXUS_EVOLUTION_ENABLED == 'true' &&
vars.GITNEXUS_EVOLUTION_WORKERS == '3'
)
)
runs-on: [self-hosted, linux, x64, gitnexus-evolution]
# Gate promotion runs on a protected Environment. An admin must attach a
# deployment-branch rule (main only) and ideally scope the three secrets to
# it — server-side enforcement a dispatched non-main ref cannot bypass by
# deployment-branch rule (main only) and ideally scope the model and App
# secrets to it — server-side enforcement a dispatched non-main ref cannot bypass by
# editing its own workflow copy. See the activation checklist above.
environment: gitnexus-evolution
timeout-minutes: 1440 # self-hosted ceiling is 5 days (7200min); 24h is a generous margin over a single-generation serial run
# Three budgets have to nest, longest first, or the evidence is lost:
# EventBridge instance uptime (24h from ~02:45)
# > this job timeout (21h)
# > the benchmark step timeout (19h, set on the step below)
# A job-level timeout CANCELS the job, so the upload step never runs and a
# multi-hour generation's evidence dies with it; a step-level timeout only
# fails that step, and `if: always()` still uploads what the sweep wrote.
# The instance must outlive the job for the same reason — when the box
# stops the runner just disappears mid-step. Scheduled runs can start well
# after the cron (the 2026-08-01 run was queued 65min late), so the job
# budget has to absorb that delay and still land inside the uptime window.
# A Friday workflow_dispatch on a box that already booted for Saturday's
# cron inherits leftover uptime, not a fresh 24h. Run 33962002890 started
# Friday 10:57 UTC and vanished at the Saturday 03:00 stop — 51 finished
# sessions never uploaded. run-evolution.sh therefore passes
# --max-runtime-from-instance-window, and the CLI derives its cap from
# /proc/uptime at startup, so the sweep fails in-process and this always()
# upload still runs.
timeout-minutes: 1260
permissions:
contents: read # The promotion PR uses a short-lived App token minted below.
actions: read # Read the previous run's evidence artifact to seed the proposer.
env:
GENERATIONS: ${{ inputs.generations || '1' }}
RUNS: ${{ inputs.runs || '3' }}
MODEL: ${{ inputs.model || 'claude-sonnet-5' }}
PROPOSER_MODEL: ${{ inputs.proposer_model || 'claude-opus-4-8' }}
# A manual input wins; scheduled runs use the repository rollout knob.
# Both fall back to serial — see workflow_bench.runner --workers for why.
WORKERS: ${{ inputs.workers || vars.GITNEXUS_EVOLUTION_WORKERS || '1' }}
MODEL: ${{ inputs.model || 'gpt-5.6-sol' }}
PROPOSER_MODEL: ${{ inputs.proposer_model || 'gpt-5.6-sol' }}
EFFORT: ${{ inputs.effort || 'xhigh' }}
PROVIDER: ${{ inputs.provider || 'openai' }}
INCLUDE_EXPENSIVE: ${{ inputs.include_expensive && '1' || '' }}
steps:
- name: Require the benchmark auth secret
env:
HAS_TOKEN: ${{ secrets.GITNEXUS_BENCH_AUTH_TOKEN != '' }}
HAS_ANTHROPIC: ${{ secrets.GITNEXUS_BENCH_ANTHROPIC_API_KEY != '' || secrets.GITNEXUS_BENCH_AUTH_TOKEN != '' }}
HAS_OPENAI: ${{ secrets.GITNEXUS_BENCH_OPENAI_API_KEY != '' }}
run: |
set -euo pipefail
if [[ "${HAS_TOKEN}" != 'true' ]]; then
echo '::error::GITNEXUS_BENCH_AUTH_TOKEN is not configured. The evolution loop runs real benchmark sessions and needs an Anthropic API key (not the Claude Code OAuth token).'
if [[ "${HAS_ANTHROPIC}" != 'true' && "${HAS_OPENAI}" != 'true' ]]; then
echo '::error::Configure GITNEXUS_BENCH_ANTHROPIC_API_KEY (Anthropic API key, not the Claude Code OAuth token) and/or GITNEXUS_BENCH_OPENAI_API_KEY. The evolution loop runs real benchmark sessions.'
exit 1
fi
case "${PROVIDER}" in
openai)
if [[ "${HAS_OPENAI}" != 'true' ]]; then
echo '::error::provider=openai requires GITNEXUS_BENCH_OPENAI_API_KEY on the gitnexus-evolution environment.'
exit 1
fi
;;
anthropic)
if [[ "${HAS_ANTHROPIC}" != 'true' ]]; then
echo '::error::provider=anthropic requires GITNEXUS_BENCH_ANTHROPIC_API_KEY on the gitnexus-evolution environment.'
exit 1
fi
;;
auto)
;;
*)
echo "::error::Unknown provider '${PROVIDER}' (expected auto, openai, or anthropic)."
exit 1
;;
esac
- name: Verify runner survival policy
run: |
set -euo pipefail
needrestart_policy=/etc/needrestart/conf.d/90-gitnexus-evolution.conf
needrestart_line="\$nrconf{restart} = 'l';"
if [[ ! -r "${needrestart_policy}" ]] || ! grep -Fqx "${needrestart_line}" "${needrestart_policy}"; then
echo "::error::${needrestart_policy} must contain: ${needrestart_line}"
exit 1
fi
# The runner sets job processes to 500; the host oom-guard rewrites
# them to -900. Read once and the check loses that race.
oom_score_adjustment="$(</proc/self/oom_score_adj)"
deadline=$((SECONDS + 5))
while (( oom_score_adjustment > -900 && SECONDS < deadline )); do
sleep 0.05
oom_score_adjustment="$(</proc/self/oom_score_adj)"
done
if (( oom_score_adjustment > -900 )); then
echo "::error::Runner.Worker descendants require OOMScoreAdjust=-900 or stronger; effective value is ${oom_score_adjustment}."
exit 1
fi
@ -145,11 +272,26 @@ jobs:
enable-cache: true
cache-dependency-glob: eval/uv.lock
- name: Fetch pinned Compound Engineering review comparator
env:
CE_COMMIT: 3ad9b51bceecf0158e590c882034d0398dbb9c5c
run: |
set -euo pipefail
destination="${RUNNER_TEMP}/compound-engineering-plugin"
rm -rf "${destination}"
git clone --filter=blob:none --no-checkout \
https://github.com/EveryInc/compound-engineering-plugin.git "${destination}"
git -C "${destination}" checkout --detach "${CE_COMMIT}"
test "$(git -C "${destination}" rev-parse HEAD)" = "${CE_COMMIT}"
- name: Install sandbox runtime and pinned Claude CLI
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get install --yes --no-install-recommends bubblewrap socat
# This box is stopped six days a week, so persistent apt timers can
# begin their catch-up run shortly after boot. Wait for dpkg instead
# of racing the same package lock and failing the weekly lane.
sudo apt-get -o DPkg::Lock::Timeout=600 update
sudo apt-get -o DPkg::Lock::Timeout=600 install --yes --no-install-recommends bubblewrap ripgrep socat
apparmor_userns=/proc/sys/kernel/apparmor_restrict_unprivileged_userns
if [[ -r "${apparmor_userns}" ]] && [[ "$(<"${apparmor_userns}")" == '1' ]]; then
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
@ -173,6 +315,16 @@ jobs:
test "$("${canary_runtime}/node_modules/@anthropic-ai/claude-code-linux-x64/claude" --version)" = \
'2.1.214 (Claude Code)'
- name: Verify contained review execution before paid sessions
working-directory: eval
env:
GITNEXUS_REQUIRE_BWRAP_CANARY: '1'
GITNEXUS_REQUIRE_CLAUDE_CANARY: '1'
CLAUDE_CANARY_BIN: ${{ runner.temp }}/claude-canary/node_modules/@anthropic-ai/claude-code-linux-x64/claude
run: |
set -euo pipefail
uv run --locked --extra dev python -m pytest tests/test_proposer_sandbox.py -q
- name: Install monorepo root dependencies
run: |
set -euo pipefail
@ -201,50 +353,169 @@ jobs:
- name: Point the benchmark task repo at the checkout
run: |
set -euo pipefail
# tasks.scenarios.yaml addresses the target repo as ~/GitNexus (the
# tasks.review.scenarios.yaml addresses the target repo as ~/GitNexus (the
# developer-local convention). On the runner the repo is the checkout
# at ${GITHUB_WORKSPACE}; link it so runner_tasks.py can resolve the
# task `repo` path. The benchmark only clones the repo (copy-on-write)
# and mounts dependencies read-only, so the checkout is never mutated.
if [[ -e "${HOME}/GitNexus" && ! -L "${HOME}/GitNexus" ]]; then
echo '::error::~/GitNexus exists and is not a symlink; refusing to place the checkout inside it.'
exit 1
fi
ln -sfn "${GITHUB_WORKSPACE}" "${HOME}/GitNexus"
# The review corpus pins historical object ids. Fetch main so those
# objects are present even when actions/checkout selected another ref.
git -C "${GITHUB_WORKSPACE}" fetch --no-tags --quiet \
"${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git" \
'+refs/heads/main:refs/remotes/origin/main'
baseline_sha="$(git -C "${GITHUB_WORKSPACE}" rev-parse --verify 'refs/remotes/origin/main^{commit}')"
echo "Fetched review corpus history at ${baseline_sha}"
- name: Seed the proposer with the previous run's evidence
id: seed
# Scheduled runs always seed; a dispatch can opt out to start clean.
if: github.event_name != 'workflow_dispatch' || inputs.seed_from_previous
# Best-effort seeding must not consume the benchmark's budget. This
# step walks up to 10 prior runs and every iteration blocks on network
# it does not control (`gh run download` of a multi-hundred-megabyte
# artifact). Unbounded, a wedged download sits here until the 21h job
# timeout CANCELS the job — and a cancelled job skips even
# `if: always()`, so the sweep never starts and nothing is uploaded.
# Bounding the step instead fails it in minutes, which is a loud,
# cheap, re-runnable failure rather than a silent 21h loss. 15 minutes
# is an order of magnitude above the observed walk (well under a
# minute) and a rounding error against the 19h sweep it protects.
timeout-minutes: 15
env:
GH_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
# Without this the weekly lane is memoryless: `--seed-results` is the
# only way a run sees what already lost (evolve stages the prior
# proposal when present and summarizes promotion.json when present),
# and with the default --generations 1 there is no earlier generation
# in-process to supply it. Every Saturday would otherwise propose
# from a blank slate and could re-propose the same rejected candidate
# forever. Best-effort by design: a first run, an expired artifact,
# or a download failure must not cost a whole generation.
if ! command -v gh >/dev/null; then
echo '::warning::gh is not installed on this runner — proposing without prior evidence. The promotion-PR step needs gh too.'
exit 0
fi
if ! previous_runs="$(gh run list \
--repo "${GITHUB_REPOSITORY}" \
--workflow gitnexus-skill-evolution.yml \
--branch main \
--status completed \
--limit 10 \
--json databaseId \
--jq "map(.databaseId) | map(select(. != ${GITHUB_RUN_ID})) | .[]")"; then
echo '::warning::Prior workflow runs could not be listed; proposing without prior evidence.'
exit 0
fi
if [[ -z "${previous_runs}" ]]; then
echo 'No prior completed run to seed from; the proposer starts from the learnings queue only.'
exit 0
fi
seed_root="${RUNNER_TEMP}/wfseed"
rm -rf "${seed_root}"
install -d -m 0700 "${seed_root}"
seed=''
# Failed sweeps deliberately upload partial evidence, so "completed"
# is the right population. Walk newest-first until one still-retained
# artifact actually contains benchmark rows; an empty latest run must
# not hide an older useful one.
for previous in ${previous_runs}; do
if [[ ! "${previous}" =~ ^[0-9]+$ ]]; then
echo "::warning::Ignoring malformed prior run id: ${previous}"
continue
fi
run_root="${seed_root}/${previous}"
install -d -m 0700 "${run_root}"
if ! gh run download "${previous}" --repo "${GITHUB_REPOSITORY}" --dir "${run_root}"; then
echo "::warning::Evidence from run ${previous} could not be downloaded (expired or absent); trying an older run."
continue
fi
unsafe="$(find "${run_root}" ! -type d ! -type f -print -quit)"
if [[ -n "${unsafe}" ]]; then
echo "::warning::Run ${previous} contains a non-regular artifact entry; trying an older run."
continue
fi
# upload-artifact normalizes directories/files to 0755/0644, while
# the evidence reader deliberately requires transcript paths to be
# owner-only. Restore that trust-boundary invariant after download.
if ! chmod -R go-rwx "${run_root}"; then
echo "::warning::Evidence permissions from run ${previous} could not be restricted; trying an older run."
continue
fi
# The artifact holds gen-N/bench/{results.jsonl,promotion.json,...};
# the highest generation is the one that actually reached the gate.
latest="$(find "${run_root}" -type f -path '*/gen-*/bench/results.jsonl' | sort -V | tail -1)"
if [[ -z "${latest}" || -L "${latest}" || ! -f "${latest}" ]]; then
echo "::warning::Run ${previous} uploaded no usable benchmark results; trying an older run."
continue
fi
# Existence is insufficient: an interrupted run may leave an empty,
# malformed, or session/infra-only JSONL. Reuse the same bounded
# selection and transcript/digest preflight the proposer will use,
# so an unusable newer run cannot hide an older useful one.
if uv run --project eval --locked --extra dev python -c \
'from pathlib import Path; import json, sys, tempfile; from workflow_bench.evolve import load_jsonl, select_evidence, stage_proposer_evidence_bundle, summarize_gate; result = Path(sys.argv[1]); root = result.parent; rows = select_evidence(load_jsonl(result)); rows or sys.exit(10); promotion = root / "promotion.json"; gate = summarize_gate(json.loads(promotion.read_text())) if promotion.is_file() else []; prior = root.parent / "proposal.md"; prior = prior if prior.is_file() and not prior.is_symlink() else None; dest = Path(tempfile.mkdtemp(prefix="wfseed-preflight-")) / "bundle"; stage_proposer_evidence_bundle(dest, results_dir=root, evidence=rows, learnings=[], gate_summary=gate, prior_proposal=prior)' \
"${latest}"; then
:
else
usability_status=$?
echo "::warning::Run ${previous} failed evidence preflight (exit ${usability_status}); trying an older run."
continue
fi
seed="$(dirname "${latest}")"
echo "Seeding the proposer from run ${previous}: ${seed}"
break
done
if [[ -z "${seed}" ]]; then
echo '::warning::No usable prior benchmark artifact found; proposing without prior evidence.'
exit 0
fi
echo "seed=${seed}" >> "${GITHUB_OUTPUT}"
- name: Run the propose → benchmark → gate loop
id: loop
# Kill the sweep with time left in the job to upload what it produced.
# See the budget nesting on the job above.
timeout-minutes: 1140
env:
GITNEXUS_BENCH_AUTH_TOKEN: ${{ secrets.GITNEXUS_BENCH_AUTH_TOKEN }}
GITNEXUS_BENCH_ANTHROPIC_API_KEY: ${{ secrets.GITNEXUS_BENCH_ANTHROPIC_API_KEY || secrets.GITNEXUS_BENCH_AUTH_TOKEN }}
GITNEXUS_BENCH_OPENAI_API_KEY: ${{ secrets.GITNEXUS_BENCH_OPENAI_API_KEY }}
# The step's stdout is a pipe, so CPython block-buffers it and a
# multi-hour generation would report nothing until it exits (run
# 29907431284 emitted every line at the same timestamp, 14h45m in).
PYTHONUNBUFFERED: '1'
SEED_RESULTS: ${{ steps.seed.outputs.seed }}
EVOLUTION_PROFILE: review
CE_PLUGIN_DIR: ${{ runner.temp }}/compound-engineering-plugin
CE_PLUGIN_VERSION: 3.24.0
run: |
set -euo pipefail
out_root="${RUNNER_TEMP}/wfevolve"
echo "out_root=${out_root}" >> "${GITHUB_OUTPUT}"
extra=()
if [[ -n "${INCLUDE_EXPENSIVE}" ]]; then
extra+=(--include-expensive)
fi
uv run --locked --extra dev python -m workflow_bench.evolve \
--tasks workflow_bench/tasks.scenarios.yaml \
--model "${MODEL}" \
--proposer-model "${PROPOSER_MODEL}" \
--generations "${GENERATIONS}" \
--runs "${RUNS}" \
--claude-bin "${RUNNER_TEMP}/claude-canary/node_modules/@anthropic-ai/claude-code-linux-x64/claude" \
--out-root "${out_root}" \
--apply \
"${extra[@]}"
./workflow_bench/run-evolution.sh --apply
working-directory: eval
- name: Upload benchmark evidence
if: always() && steps.loop.outputs.out_root != ''
# Unconditional: the sweep writes results.jsonl and transcripts as it
# goes, so a killed or failed generation still has evidence worth
# keeping — and that is exactly the run whose evidence is needed.
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: gitnexus-evolution-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ steps.loop.outputs.out_root }}
# Addressed directly rather than carried from the sweep step: that is
# the step whose death is the reason this upload matters, and a value
# threaded from it would not be there when it counts.
path: ${{ runner.temp }}/wfevolve
retention-days: 14
if-no-files-found: warn
- name: Detect and bound the applied promotion
id: promotion
env:
OUT_ROOT: ${{ steps.loop.outputs.out_root }}
run: |
set -euo pipefail
changed="$(git status --porcelain)"
@ -259,7 +530,7 @@ jobs:
while IFS= read -r line; do
path="${line:3}"
case "${path}" in
.claude/skills/*|gitnexus/skills/*|gitnexus-claude-plugin/skills/*) ;;
.claude/skills/gitnexus-review/*|gitnexus/skills/gitnexus-review/*|gitnexus-claude-plugin/skills/gitnexus-review/*|gitnexus-cursor-integration/skills/gitnexus-review/*) ;;
*)
echo "::error::Promotion touched a path outside the skill trees: ${path}"
exit 1
@ -273,7 +544,7 @@ jobs:
# generation's decisions could surface in the PR body. The heredoc
# uses a per-run random delimiter so a summary value that ever
# contains the marker cannot close the block early and inject keys.
promotion_file="$(find "${OUT_ROOT}" -name promotion.json | sort -V | tail -1)"
promotion_file="$(find "${RUNNER_TEMP}/wfevolve" -name promotion.json | sort -V | tail -1)"
delim="PROMOTION_EOF_$(openssl rand -hex 16)"
{
echo "summary<<${delim}"
@ -316,7 +587,7 @@ jobs:
git config user.name 'gitnexus-evolution[bot]'
git config user.email 'gitnexus-evolution[bot]@users.noreply.github.com'
git checkout -b "${branch}"
git add .claude/skills gitnexus/skills gitnexus-claude-plugin/skills
git add .claude/skills gitnexus/skills gitnexus-claude-plugin/skills gitnexus-cursor-integration/skills/gitnexus-review
git commit -m 'feat(skills): promoted evolution overlay (gate-passed)'
# The App token reaches git through GIT_ASKPASS reading step env at

77
.github/workflows/pr-autofix-publish.yml vendored
View file

@ -121,64 +121,35 @@ jobs:
# metadata.json to reference another PR or SHA, redirecting our
# write-scoped sticky/check-run onto an attacker-chosen target.
#
# Authority sources are all server-controlled GitHub event fields:
# - workflow_run.head_sha
# - workflow_run.head_repository.full_name
# - workflow_run.pull_requests[].number (within-repo PRs only;
# empty array on fork PRs — fall back to commits/{sha}/pulls)
# Authority is workflow_run.head_sha + head_repository.full_name +
# head_branch, resolved via pulls?head={owner}:{branch}. That query
# works for same-repo PRs and forks; commits/{sha}/pulls is empty
# for fork SHAs. The script comes from THIS default-branch checkout
# (the same trust anchor as this workflow file).
#
# Always verify — including the changed_lines=0 path — so the
# check-run SHA cannot be an unverified artifact field.
# Mismatch => fail loud BEFORE any sticky/check-run side effect.
- name: Checkout identity verifier
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
sparse-checkout: .github/scripts/verify-workflow-run-pr-identity.cjs
sparse-checkout-cone-mode: false
path: trusted
- name: Verify metadata against workflow_run authority
id: verify
if: steps.meta.outputs.changed_lines != '0'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}
META_PR_NUMBER: ${{ steps.meta.outputs.pr_number }}
META_HEAD_SHA: ${{ steps.meta.outputs.head_sha }}
META_HEAD_REPO: ${{ steps.meta.outputs.head_repo }}
META_PATH: autofix-in/metadata.json
SCHEMA_PATTERN: '^gitnexus\.pr-autofix/v[0-9]+$'
WF_HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
WF_HEAD_REPO: ${{ github.event.workflow_run.head_repository.full_name }}
WF_PR_NUMBERS: ${{ toJSON(github.event.workflow_run.pull_requests.*.number) }}
WF_HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
shell: bash
run: |
set -euo pipefail
# 1) head_sha must match exactly. workflow_run.head_sha is the
# commit GitHub actually ran the producer against — definitive.
if [ "${META_HEAD_SHA}" != "${WF_HEAD_SHA}" ]; then
echo "::error::Artifact head_sha (${META_HEAD_SHA}) does not match workflow_run.head_sha (${WF_HEAD_SHA}) — refusing to publish."
exit 1
fi
# 2) head_repo must match exactly. Same authority anchor.
if [ "${META_HEAD_REPO}" != "${WF_HEAD_REPO}" ]; then
echo "::error::Artifact head_repo (${META_HEAD_REPO}) does not match workflow_run.head_repository (${WF_HEAD_REPO}) — refusing to publish."
exit 1
fi
# 3) pr_number must reference an open PR with this head SHA.
# Within-repo PRs: workflow_run.pull_requests[] is populated.
# Fork PRs: that array is empty by GitHub design — fall back
# to the REST commit-to-PRs lookup. Fail closed if the lookup
# finds no matching open PR (avoids attacker-forged PR ids).
allowed_numbers=$(jq -c '.' <<< "${WF_PR_NUMBERS}")
if [ "${allowed_numbers}" = "[]" ]; then
echo "workflow_run.pull_requests is empty (fork PR) — falling back to commits/{sha}/pulls."
allowed_numbers=$(gh api "repos/${GH_REPO}/commits/${WF_HEAD_SHA}/pulls" \
--jq '[.[] | select(.state == "open") | .number]' 2>/dev/null || echo "[]")
if [ "${allowed_numbers}" = "[]" ]; then
echo "::error::No open PR found for head ${WF_HEAD_SHA} via commits/{sha}/pulls — refusing to publish."
exit 1
fi
fi
if ! jq -e --argjson n "${META_PR_NUMBER}" 'index($n) != null' <<< "${allowed_numbers}" >/dev/null; then
echo "::error::Artifact pr_number (${META_PR_NUMBER}) is not in the authoritative PR list (${allowed_numbers}) — refusing to publish."
exit 1
fi
echo "Verified: metadata identity matches workflow_run authority (PR=${META_PR_NUMBER}, head_sha=${META_HEAD_SHA}, head_repo=${META_HEAD_REPO})."
run: node trusted/.github/scripts/verify-workflow-run-pr-identity.cjs
- name: Upsert sticky summary comment
# Only post when ci-quality found something fixable (= the
@ -187,14 +158,14 @@ jobs:
# so we skip it.
if: >-
always()
&& steps.meta.outputs.pr_number != ''
&& steps.verify.outcome == 'success'
&& steps.meta.outputs.changed_lines != '0'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}
PR: ${{ steps.meta.outputs.pr_number }}
PR: ${{ steps.verify.outputs.pr_number }}
CHANGED: ${{ steps.meta.outputs.changed_lines }}
HEAD_SHA: ${{ steps.meta.outputs.head_sha }}
HEAD_SHA: ${{ steps.verify.outputs.head_sha }}
RUN_ID: ${{ github.run_id }}
shell: bash
run: |
@ -285,11 +256,11 @@ jobs:
# fixes-available → conclusion: neutral
# `neutral` does not block branch-protection required-checks but
# is visually distinct from a green pass.
if: always() && steps.meta.outputs.head_sha != ''
if: always() && steps.verify.outcome == 'success'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}
HEAD_SHA: ${{ steps.meta.outputs.head_sha }}
HEAD_SHA: ${{ steps.verify.outputs.head_sha }}
CHANGED: ${{ steps.meta.outputs.changed_lines }}
shell: bash
run: |

2
.github/workflows/pr-labeler.yml vendored
View file

@ -108,7 +108,7 @@ jobs:
# Pinned to v7.2.0. Verify SHA via:
# gh api repos/release-drafter/release-drafter/git/refs/tags/v7.2.0
# v7 removed `disable-releaser`; use `dry-run: true` to only autolabel.
- uses: release-drafter/release-drafter@eada3c96a64734dd381cfbda23511034e328ddb0 # v7.6.0
- uses: release-drafter/release-drafter@34d80673e067bdc0c24568d3af899c216adcfaa9 # v7.7.0
with:
config-name: release-drafter.yml
dry-run: true

13
.github/workflows/publish.yml vendored
View file

@ -396,14 +396,17 @@ jobs:
# cache-poisoning audit). ~30s slower per release; runs rarely.
package-manager-cache: false
- name: Build gitnexus-shared
run: npm ci && npm run build
working-directory: gitnexus-shared
- name: Install gitnexus dependencies
run: npm ci
working-directory: gitnexus
# The published tarball ships the web UI (`files: [... "web"]`), built
# by prepack during `npm publish`. Install its deps in their own step
# so a slow install is visible here instead of dying inside build.js.
- name: Install gitnexus-web dependencies
run: npm ci
working-directory: gitnexus-web
# ── Stable-only: verify the tag and package.json agree ───────────────
- name: Verify version consistency (stable)
if: needs.route.outputs.mode == 'stable'
@ -828,7 +831,7 @@ jobs:
fi
- name: Create GitHub Release
uses: softprops/action-gh-release@3d0d9888cb7fd7b750713d6e236d1fcb99157228 # v2
uses: softprops/action-gh-release@efb35369e0ad2afab669f228072c1b0d510eae64 # v2
with:
tag_name: ${{ steps.vtag-gate.outputs.vtag }}
name: >-

4
.github/workflows/scorecard.yml vendored
View file

@ -38,7 +38,7 @@ jobs:
persist-credentials: false
- name: Run Scorecard
uses: ossf/scorecard-action@4eaacf0543bb3f2c246792bd56e8cdeffafb205a # v2.4.3
uses: ossf/scorecard-action@2d1146689b8cda280b9bc96326124645441f03bc # v2.4.4
with:
results_file: results.sarif
results_format: sarif
@ -53,6 +53,6 @@ jobs:
retention-days: 5
- name: Upload to Security tab
uses: github/codeql-action/upload-sarif@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3
uses: github/codeql-action/upload-sarif@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9
with:
sarif_file: results.sarif

3
.github/workflows/skill-sync.yml vendored
View file

@ -55,9 +55,6 @@ jobs:
node-version: '22'
cache: npm
cache-dependency-path: gitnexus/package-lock.json
- name: Build gitnexus-shared
run: npm ci && npm run build
working-directory: gitnexus-shared
- name: Install gitnexus
run: npm ci
working-directory: gitnexus

2
.github/workflows/tree-sitter-upgrade-readiness.yml vendored
View file

@ -4,7 +4,7 @@ name: Tree-sitter Upgrade Readiness
# 1. Peer-dep compatibility — can each NPM-installed grammar install cleanly
# with tree-sitter@0.25.0 without --legacy-peer-deps?
# 2. Vendored grammars — each grammar in .github/vendored-grammars.json
# (c/swift/kotlin/dart/proto) is classified by its vendored ABI, read
# (c/swift/kotlin/dart/proto/objc) is classified by its vendored ABI, read
# straight from gitnexus/vendor/<name>/src/parser.c (NOT node_modules,
# which is never populated for vendored grammars — that mismatch is why
# the report used to render bare "?" placeholders, #858).

2
.github/workflows/triage-sweep.yml vendored
View file

@ -66,7 +66,7 @@ jobs:
fetch-depth: 1
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: '3.12'
cache: pip

4
.github/workflows/trivy.yml vendored
View file

@ -50,7 +50,7 @@ jobs:
persist-credentials: false
- name: Setup Buildx
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
- name: Build image (load locally for scan)
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
@ -76,7 +76,7 @@ jobs:
exit-code: '0'
- name: Upload to Security tab
uses: github/codeql-action/upload-sarif@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3
uses: github/codeql-action/upload-sarif@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9
with:
sarif_file: trivy-${{ matrix.image.name }}.sarif
category: trivy-${{ matrix.image.name }}

4
.github/workflows/workflow-lint.yml vendored
View file

@ -58,7 +58,7 @@ jobs:
persist-credentials: false
- name: Setup Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: '3.12'
@ -76,7 +76,7 @@ jobs:
continue-on-error: true
- name: Upload SARIF
uses: github/codeql-action/upload-sarif@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3
uses: github/codeql-action/upload-sarif@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9
with:
sarif_file: zizmor.sarif
category: zizmor

11
.github/zizmor.yml vendored
View file

@ -18,9 +18,11 @@ rules:
# untrusted half (pr-autofix.yml) runs fork code with permissions:{}
# and produces only a diff artifact (data, not executable code). The
# publish job consumes the artifact, allowlist-validates every field
# of metadata.json before exporting to $GITHUB_OUTPUT, never checks
# out fork code, and never executes anything fork-controlled. Header
# comment in the file documents the split.
# of metadata.json, then cross-checks identity against
# workflow_run.head_sha / head_repository / head_branch via
# pulls?head=owner:branch (commits/{sha}/pulls is empty for fork SHAs).
# It never checks out fork code and never executes anything
# fork-controlled. Header comment in the file documents the split.
- pr-autofix-publish.yml
# workflow_run is the trusted half of the vendored-grammar prebuild
@ -29,7 +31,8 @@ rules:
# validates the .node prebuilds and uploads them as artifacts. This
# consumer downloads ONLY those artifacts + metadata.json,
# allowlist-validates every metadata field, cross-checks identity against
# the workflow_run authority (head_sha / head_repo / pr_number), and
# workflow_run.head_sha / head_repository / head_branch via
# pulls?head=owner:branch (commits/{sha}/pulls is empty for fork SHAs), and
# checks out the fork head pinned to that HEAD SHA solely to ADD prebuild
# files (never executes fork code) before pushing. Header comment in the
# file documents the split.

4
.gitignore vendored
View file

@ -31,6 +31,8 @@ npm-debug.log*
# Testing
coverage/
.tmp-test/
gitnexus/.tmp-test/
# Misc
*.local
@ -70,6 +72,8 @@ eval/.hypothesis/
# Local docs — planning output (gitnexus-plan / gitnexus-work) stays local, not tracked
docs/*
!docs/fork/
!docs/fork/**
gitnexus/test/fixtures/mini-repo/*.md
gitnexus/test/fixtures/mini-repo/.claude

2
.gitleaksignore
View file

@ -1,2 +1,4 @@
# Deleted README placeholder from PR #2458; no credential was present.
c9fdab17f25ebaf332fba6e6ba55ee328f20fe66:README.md:curl-auth-header:348
# Synthetic Kotlin Actuator fixture value from PR #3107; no credential was present.
3951079300a18b14e79f5b5f5dd778ae19ced6e3:gitnexus/test/integration/spring-actuator-kotlin-runtime-pipeline.test.ts:generic-api-key:8

37
AGENTS.md
View file

@ -1,7 +1,7 @@
<!-- version: 1.14.0 -->
<!-- Last updated: 2026-07-16 -->
<!-- version: 1.15.0 -->
<!-- Last updated: 2026-09-07 -->
Last reviewed: 2026-07-16
Last reviewed: 2026-09-07
**Project:** GitNexus · **Environment:** dev · **Maintainer:** repository maintainers (see GitHub)
@ -39,6 +39,7 @@ Commands and gotchas live under **Repo reference** below and in **[CONTRIBUTING.
## Reference docs
- **[ARCHITECTURE.md](ARCHITECTURE.md)**, **[CONTRIBUTING.md](CONTRIBUTING.md)**, **[GUARDRAILS.md](GUARDRAILS.md)**
- **Objective-C provider work:** read **[docs/languages/objective-c-provider.md](docs/languages/objective-c-provider.md)** before changing Objective-C parsing or resolution.
- **Call & inheritance resolution (RFC #909 Ring 3):** See ARCHITECTURE.md § Scope-Resolution Pipeline. All languages resolve calls and inheritance through the scope-resolution pipeline (`Registry.lookup`, `preEmitInheritanceEdges`, `emitHeritageEdges`, `buildMro` → `MethodDispatchIndex`). **Shared code in `gitnexus/src/core/ingestion/` must not name languages** — plug language behavior in via `LanguageProvider` / `ScopeResolver` hooks. A language plugs in by implementing `ScopeResolver` (`scope-resolution/contract/scope-resolver.ts`) and registering it in `SCOPE_RESOLVERS`. (The legacy call-resolution DAG + `@heritage` capture path were removed in RING4-1 #942.)
- **Cursor:** `.cursor/index.mdc` (always-on); `.cursor/rules/*.mdc` (glob-scoped). Legacy `.cursorrules` deprecated.
- **GitNexus:** standard skills in `.claude/skills/gitnexus-*/`; MCP rules in `gitnexus:start` block below.
@ -90,6 +91,7 @@ mirror. `gitnexus/test/unit/shipped-skills-sync.test.ts` guards the copies. Toke
| Date | Version | Change |
|------|---------|--------|
| 2026-09-07 | 1.15.0 | Added the Objective-C provider guide as the required reference before changing Objective-C parsing or resolution. |
| 2026-07-20 | 1.14.0 | `gitnexus-review` gains a coordinated swarm: six `ci-personas/` lanes the CI review agent dispatches as subagents (via the `Agent` tool), with a bounded critic gate and sidechain-excluded evidence. |
| 2026-07-16 | 1.13.0 | `gitnexus-plan` asks plan depth up front (quick/standard/deep) in interactive runs; `gitnexus-lfg` gate slimmed to proceed/stop (Deepen stays as the route-back mechanism). |
| 2026-07-16 | 1.12.0 | Renamed `gitnexus-pr-review` to `gitnexus-review`; added PR URL/number, branch/range, and local-change targets plus install migration (setup warns on a legacy `gitnexus-pr-review` dir and leaves it in place; uninstall removes it). |
@ -111,30 +113,31 @@ mirror. `gitnexus/test/unit/shipped-skills-sync.test.ts` guards the copies. Toke
<!-- gitnexus:start -->
# GitNexus — Code Intelligence
This project is indexed by GitNexus as **GitNexus** (20319 symbols, 54304 relationships, 300 execution flows). Use the GitNexus MCP tools to understand code, assess impact, and navigate safely.
This project is indexed by GitNexus as **GitNexus** (248612 symbols, 565510 relationships, 918 execution flows).
> Index stale? Run `node .gitnexus/run.cjs analyze` from the project root — it auto-selects an available runner. No `.gitnexus/run.cjs` yet? `npx gitnexus analyze` (npm 11 crash → `npm i -g gitnexus`; #1939).
> Index stale? Run `node .gitnexus/run.cjs analyze --index-only` from the project root — it auto-selects an available runner. No `.gitnexus/run.cjs` yet? Bootstrap with `npx`, `bunx`, or `pnpm dlx` — e.g. `bunx gitnexus@latest analyze` (npm 11 npx crash; #1939).
## Always Do
- **MUST run impact analysis before editing any symbol.** Before modifying a function, class, or method, run `impact({target: "symbolName", direction: "upstream"})` and report the blast radius (direct callers, affected processes, risk level) to the user.
- **MUST run `detect_changes()` before committing** to verify your changes only affect expected symbols and execution flows. For regression review, compare against the default branch: `detect_changes({scope: "compare", base_ref: "main"})`.
- **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits.
- When exploring unfamiliar code, use `query({search_query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance.
- When you need full context on a specific symbol — callers, callees, which execution flows it participates in — use `context({name: "symbolName"})`.
- **MUST run impact analysis before editing.** Use `impact({target: "symbolName", direction: "upstream"})` (MCP) or `node .gitnexus/run.cjs impact "symbolName" --direction upstream --repo .` (CLI fallback); report callers, processes, and risk. Never substitute grep for graph analysis. For unified PDG impact, add `mode: "pdg"` with optional `line: <N>` — it returns statement-level `affectedStatements` over CDG + REACHING_DEF and inter-procedural symbols in `interproceduralByDepth`/`byDepth`; no-layer/degraded PDG results are UNKNOWN-risk notes (`--pdg` layer). CLI equivalent: `node .gitnexus/run.cjs impact "symbolName" --direction upstream --mode pdg --line <N> --repo .`.
- **MUST analyze graph changes before committing.** Use `detect_changes({scope: "all"})` (MCP) or `node .gitnexus/run.cjs detect-changes --scope all --repo .` (CLI fallback). `partial: true` or `truncated: true` is not a clean check — a zero means unseen, not unaffected; re-run it. For regression review: `detect_changes({scope: "compare", base_ref: "main"})` or `node .gitnexus/run.cjs detect-changes --scope compare --base-ref "main" --repo .`.
- MUST warn on HIGH/CRITICAL `risk` pre-edit; never use `riskSharedAxes` to waive a HIGH/CRITICAL `risk` warning. Compare File/symbol: MCP File omits axes; Graph-RAG expands File.
- **MUST treat `risk: UNKNOWN` as unresolved, not as low.** An empty caller set is not evidence the symbol is unused — it can also mean the callers are not resolvable by the index (plain-object property access, dynamic dispatch, cross-language calls). `impact` pairs `UNKNOWN` with a `riskNote` saying so. Confirm with a text search before treating the symbol as safe to change or delete; do not proceed on the strength of a zero.
- **MUST use `query({search_query: "concept"})` for concepts/flows, `context({name: "symbolName"})` for a named symbol, or `impact` for blast radius, on read-only callers, dependencies, imports, or execution flow.** Graph first; text search only for empty/`UNKNOWN`/literals.
- For security review, `explain({target: "fileOrSymbol"})` lists taint findings (source→sink flows; needs `analyze --pdg`).
- For control/data dependence, `pdg_query({mode: "controls", target: "fileOrSymbol"})` answers "under what condition does X run?" (CDG, incl. guard clauses) and `pdg_query({mode: "flows", target, variable})` traces "where does variable Y flow?" (REACHING_DEF). `--pdg` layer.
## Never Do
- NEVER edit a function, class, or method without first running `impact` on it.
- NEVER ignore HIGH or CRITICAL risk warnings from impact analysis.
- NEVER edit a function, class, or method before MCP/CLI impact analysis.
- NEVER ignore HIGH or CRITICAL risk warnings from impact analysis, and never read `UNKNOWN` as an all-clear — it means the walk could not answer, which is the one verdict that requires confirming by other means.
- NEVER rename symbols with find-and-replace — use `rename` which understands the call graph.
- NEVER commit changes without running `detect_changes()` to check affected scope.
- NEVER commit before MCP/CLI graph change analysis.
## Resources
| Resource | Use for |
|----------|---------|
| --- | --- |
| `gitnexus://repo/GitNexus/context` | Codebase overview, check index freshness |
| `gitnexus://repo/GitNexus/clusters` | All functional areas |
| `gitnexus://repo/GitNexus/processes` | All execution flows |
@ -143,7 +146,7 @@ This project is indexed by GitNexus as **GitNexus** (20319 symbols, 54304 relati
## CLI
| Task | Read this skill file |
|------|---------------------|
| --- | --- |
| Understand architecture / "How does X work?" | `.claude/skills/gitnexus-exploring/SKILL.md` |
| Blast radius / "What breaks if I change X?" | `.claude/skills/gitnexus-impact-analysis/SKILL.md` |
| Trace bugs / "Why is X failing?" | `.claude/skills/gitnexus-debugging/SKILL.md` |
@ -191,6 +194,6 @@ npx gitnexus serve # HTTP API on port 4747 (from any ind
### Gotchas
- `npm install` in `gitnexus/` triggers `prepare` (builds via `tsc`) and `postinstall` (materializes the vendored grammars into `node_modules/`, then prefers a committed prebuild per platform-arch and only source-builds when none matches). A C/C++ toolchain (`python3`, `make`, `g++`) is needed only for that source-build fallback.
- The vendored grammars `tree-sitter-{c,dart,proto,swift,kotlin}` are handled uniformly: c is required; dart/proto/swift/kotlin are optional and skippable via `GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1`. Install warnings appear only when no prebuild matches the platform-arch and no toolchain is present, and are non-fatal — only that language's parsing is unavailable.
- `npm install` in `gitnexus/` triggers `prepare` (builds via `tsc`) and `postinstall` (`build-tree-sitter-grammars.cjs` activates committed prebuilds in place under `vendor/`, and only source-builds when none matches). A C/C++ toolchain (`python3`, `make`, `g++`) is needed only for that source-build fallback.
- The vendored grammars `tree-sitter-{c,dart,proto,swift,kotlin,zig}` are handled uniformly: c is required; dart/proto/swift/kotlin/zig are optional and skippable via `GITNEXUS_SKIP_OPTIONAL_GRAMMARS=1`. Install warnings appear only when no prebuild matches the platform-arch and no toolchain is present, and are non-fatal — only that language's parsing is unavailable.
- ESLint configured via `eslint.config.mjs` (TS, React Hooks, unused-imports). No `npm run lint` script; use `npx eslint .`. Prettier runs via lint-staged. CI checks both in `ci-quality.yml`.

352
ARCHITECTURE.md
View file

@ -4,18 +4,18 @@ Monorepo: **CLI/MCP** (`gitnexus/`) + **browser UI** (`gitnexus-web/`).
## Repository layout
| Path | Role |
|------|------|
| `gitnexus/` | npm package `gitnexus`: CLI, MCP server (stdio), HTTP API, ingestion pipeline, LadybugDB graph, embeddings. |
| `gitnexus-web/` | Vite + React thin client: graph explorer + AI chat. All queries via `gitnexus serve` HTTP API. |
| `gitnexus-shared/` | Shared TypeScript types and constants (consumed by CLI and Web). |
| `.claude/`, `gitnexus-claude-plugin/`, `gitnexus-cursor-integration/` | Agent skills and plugin metadata. |
| `eval/` | Evaluation harnesses for benchmarking tool usage. |
| `.github/` | CI workflows + composite actions (`setup-gitnexus/`, `setup-gitnexus-web/`). |
| Path | Role |
| --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| `gitnexus/` | npm package `gitnexus`: CLI, MCP server (stdio), HTTP API, ingestion pipeline, LadybugDB graph, embeddings. |
| `gitnexus-web/` | Vite + React thin client: graph explorer + AI chat. All queries via `gitnexus serve` HTTP API. |
| `gitnexus-shared/` | Shared TypeScript types and constants (consumed by CLI and Web). |
| `.claude/`, `gitnexus-claude-plugin/`, `gitnexus-cursor-integration/` | Agent skills and plugin metadata. |
| `eval/` | Evaluation harnesses for benchmarking tool usage. |
| `.github/` | CI workflows + composite actions (`setup-gitnexus/`, `setup-gitnexus-web/`). |
## End-to-end flow: index → graph → tools
1. **Ingestion** — `analyze.ts` → `runFullAnalysis` (`run-analyze.ts`) → `runPipelineFromRepo` (`pipeline.ts`). DAG of 15 phases builds a `KnowledgeGraph` in memory, then loads into LadybugDB under `.gitnexus/`. Repo registered in `~/.gitnexus/registry.json` for MCP discovery.
1. **Ingestion** — `analyze.ts` → `runFullAnalysis` (`run-analyze.ts`) → `runPipelineFromRepo` (`pipeline.ts`). The default DAG of 19 phases builds a `KnowledgeGraph` in memory, then loads into LadybugDB under `.gitnexus/`. Repo registered in `~/.gitnexus/registry.json` for MCP discovery.
2. **Persistence** — `repo-manager.ts` (paths, registry, LadybugDB cleanup). `lbug-adapter.ts` (graph load, queries, embedding batches).
@ -24,57 +24,57 @@ Monorepo: **CLI/MCP** (`gitnexus/`) + **browser UI** (`gitnexus-web/`).
- **HTTP bridge:** `serve.ts` → Express (`api.ts`, `mcp-http.ts`) for web UI
- **CLI direct:** `gitnexus query|context|impact|cypher` in `tool.ts`
4. **Staleness** — `staleness.ts` compares indexed `lastCommit` to `HEAD`, surfaces hints.
4. **Staleness** — `core/git-staleness.ts` compares indexed `lastCommit` to `HEAD` and classifies the result as `current`, `behind`, `diverged` (HEAD moved off the indexed commit, gap uncountable) or `unknown`; `core/staleness-status.ts` builds the one `staleness` payload that MCP `list_repos`, the read tools and the `serve` repo routes all emit.
## MCP tools
| Tool | Purpose |
|------|---------|
| `list_repos` | Discover indexed repos |
| `query` | Hybrid BM25 + vector search over the graph |
| `cypher` | Ad hoc Cypher against the schema |
| `context` | Callers, callees, processes for one symbol |
| `impact` | Blast radius (upstream/downstream) with risk summary |
| `detect_changes` | Map git diffs to affected symbols and processes |
| `rename` | Graph-assisted multi-file rename with `dry_run` preview |
| `api_impact` | Pre-change impact report for an API route handler |
| `trace` | Shortest directed path between two symbols (call + class-member edges); group-aware (`repo: "@<group>"`) for cross-repo traces |
| `route_map` | API route → handler → consumer mappings |
| `tool_map` | MCP/RPC tool definitions and handlers |
| `shape_check` | Response shape vs consumer property access mismatches |
| `explain` | Persisted taint findings (source→sink data flows) — needs `analyze --pdg` |
| `pdg_query` | Control/data dependence — CDG (`mode: controls`) / REACHING_DEF (`mode: flows`) — needs `analyze --pdg` |
| `group_list` | List repo groups or details for one group |
| `group_sync` | Rebuild group Contract Registry (`contracts.json`) and bridge graph |
| Tool | Purpose |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `list_repos` | Discover indexed repos |
| `query` | Hybrid BM25 + vector search over the graph |
| `cypher` | Ad hoc Cypher against the schema |
| `context` | Callers, callees, processes for one symbol |
| `impact` | Blast radius (upstream/downstream) with risk summary |
| `detect_changes` | Map git diffs to affected symbols and processes |
| `rename` | Graph-assisted multi-file rename with `dry_run` preview |
| `api_impact` | Pre-change impact report for an API route handler |
| `trace` | Shortest directed path between two symbols (call + class-member edges); group-aware (`repo: "@<group>"`) for cross-repo traces |
| `route_map` | API route → handler → consumer mappings |
| `tool_map` | MCP/RPC tool definitions and handlers |
| `shape_check` | Response shape vs consumer property access mismatches |
| `explain` | Persisted taint findings (source→sink data flows) — needs `analyze --pdg` |
| `pdg_query` | Control/data dependence — CDG (`mode: controls`) / REACHING_DEF (`mode: flows`) — needs `analyze --pdg` |
| `group_list` | List repo groups or details for one group |
| `group_sync` | Rebuild group Contract Registry (`contracts.json`) and bridge graph |
`query`, `context`, and `impact` are group-aware: pass `repo: "@<groupName>"` (or `"@<groupName>/<memberPath>"` to scope to one member) plus optional `service: "<monorepo/path>"`. Group-mode `query` merges per-repo results via Reciprocal Rank Fusion; group-mode `impact` runs the local walk in the chosen member and fans out across boundaries via the Contract Bridge (`gitnexus/src/core/group/cross-impact.ts`). `trace` is also group-aware via `repo: "@<groupName>"` — but, unlike the others, it resolves `from`/`to` across **all** members (a `@<groupName>/<memberPath>` suffix is advisory for trace, not a scope); pass `from_uid`/`to_uid` to disambiguate a symbol name that occurs in more than one member.
Group-mode `trace` (`gitnexus/src/core/group/cross-trace.ts`) stitches a path that crosses repositories: it resolves `from`/`to` across all members, and when they live in different repos it joins the home-repo segment to the target-repo segment over a single `ContractLink` boundary (an HTTP consumer→provider link, joined on `Contract.symbolUid`), reported as a `CONTRACT_LINK` hop in `crossings[]`. The crossing is clamped to one boundary (`MAX_SUPPORTED_CROSS_DEPTH`, shared with cross-impact); deeper `crossDepth` is reported via `notes[]`. With `pdg: true` (experimental, opt-in), each boundary-adjacent segment is enriched with its intra-procedural REACHING_DEF data-flow when that repo was indexed with `--pdg` (reusing the same anchored `flows` query as `pdg_query`); data flow never crosses the repo boundary, and a missing PDG layer degrades to call-level hops with a note. Two stores meet only at the `symbolUid` grain — the per-repo PDG/call graph and the group bridge — so this is the documented join; full cross-program (SDG-like) data flow across the boundary remains deferred (see `docs/plans/2026-06-18-002-feat-unified-pdg-impact-evaluation-plan.md`). The previously-planned `group_query`, `group_context`, `group_impact`, `group_contracts`, `group_status` MCP tools are intentionally not introduced — group-level state is exposed via resources instead:
| Resource URI | Purpose |
|--------------|---------|
| Resource URI | Purpose |
| ----------------------------------- | -------------------------------------------------------- |
| `gitnexus://group/{name}/contracts` | Contract Registry (provider/consumer rows + cross-links) |
| `gitnexus://group/{name}/status` | Per-member index + Contract Registry staleness |
| `gitnexus://group/{name}/status` | Per-member index + Contract Registry staleness |
## Where to change what
| Concern | Start in |
|---------|----------|
| CLI commands/flags | `src/cli/` (`index.ts`, per-command modules) |
| Parsing/graph construction | `src/core/ingestion/pipeline-phases/` + `pipeline.ts` |
| Graph schema/DB | `src/core/lbug/` (`schema.ts`, `lbug-adapter.ts`) |
| MCP tools/resources | `src/mcp/server.ts`, `tools.ts`, `resources.ts` |
| Cross-repo groups (sync, contracts, `@<group>` routing) | `src/core/group/` (`service.ts`, `cross-impact.ts`, `sync.ts`, `bridge-db.ts`) |
| Search ranking | `src/core/search/` (BM25, hybrid fusion) |
| Embeddings | `src/core/embeddings/` + `src/core/run-analyze.ts` |
| Wiki generation | `src/core/wiki/` |
| Language support | `src/core/ingestion/languages/` + `tree-sitter-queries.ts` + `gitnexus-shared/src/languages.ts` |
| Import resolution | `src/core/ingestion/import-processor.ts` + `import-resolvers/configs/` + `model/resolution-context.ts` |
| Call resolution/inheritance/MRO | `src/core/ingestion/scope-resolution/` (pipeline, passes, graph-bridge) |
| Type extraction | `src/core/ingestion/type-extractors/` |
| Worker pool | `src/core/ingestion/workers/` |
| Web UI | `gitnexus-web/src/` |
| CI | `.github/workflows/*.yml`, `.github/actions/` |
| Concern | Start in |
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| CLI commands/flags | `src/cli/` (`index.ts`, per-command modules) |
| Parsing/graph construction | `src/core/ingestion/pipeline-phases/` + `pipeline.ts` |
| Graph schema/DB | `src/core/lbug/` (`schema.ts`, `lbug-adapter.ts`) |
| MCP tools/resources | `src/mcp/server.ts`, `tools.ts`, `resources.ts` |
| Cross-repo groups (sync, contracts, `@<group>` routing) | `src/core/group/` (`service.ts`, `cross-impact.ts`, `sync.ts`, `bridge-db.ts`) |
| Search ranking | `src/core/search/` (BM25, hybrid fusion) |
| Embeddings | `src/core/embeddings/` + `src/core/run-analyze.ts` |
| Wiki generation | `src/core/wiki/` |
| Language support | `src/core/ingestion/languages/` + `tree-sitter-queries.ts` + `gitnexus-shared/src/languages.ts` |
| Import resolution | `src/core/ingestion/import-processor.ts` + `import-resolvers/configs/` + `model/resolution-context.ts` |
| Call resolution/inheritance/MRO | `src/core/ingestion/scope-resolution/` (pipeline, passes, graph-bridge) |
| Type extraction | `src/core/ingestion/type-extractors/` |
| Worker pool | `src/core/ingestion/workers/` |
| Web UI | `gitnexus-web/src/` |
| CI | `.github/workflows/*.yml`, `.github/actions/` |
> Paths above are relative to `gitnexus/` unless they start with `gitnexus-web/` or `.github/`.
@ -82,30 +82,35 @@ Group-mode `trace` (`gitnexus/src/core/group/cross-trace.ts`) stitches a path th
## Pipeline Phase DAG
15 phases defined in `gitnexus/src/core/ingestion/pipeline-phases/`, each with explicit `deps` and typed output.
19 default phases are defined in `gitnexus/src/core/ingestion/pipeline-phases/`, each with explicit `deps` and typed output. `--pdg` adds `taintSummaries` and `callSummaries` (21 total).
```
scan → structure → [markdown, cobol] → parse → [routes, tools, orm]
→ crossFile → scopeResolution → pruneLocalSymbols → mro → di → communities → processes
scan → structure → [springConfig, markdown, cobol] → parse → [routes, tools, orm]
→ crossFile → scopeResolution → [springAutoConfiguration, springAop]
→ pruneLocalSymbols → mro → springAopInheritance → di → communities → processes
```
| Phase | File | Deps | Output |
|-------|------|------|--------|
| `scan` | `scan.ts` | (root) | File paths + sizes |
| `structure` | `structure.ts` | `scan` | File/Folder nodes, CONTAINS edges, `allPathSet` |
| `markdown` | `markdown.ts` | `structure` | Section nodes, cross-link edges from .md/.mdx |
| `cobol` | `cobol.ts` | `structure` | COBOL program/paragraph/section nodes (regex, no tree-sitter) |
| `parse` | `parse.ts` + `parse-impl.ts` | `structure`, `markdown`, `cobol` | Symbol nodes, IMPORTS/CALLS/EXTENDS edges, extracted routes/tools/ORM queries |
| `routes` | `routes.ts` | `parse` | Route nodes + HANDLES_ROUTE edges (Next.js, Expo, PHP, decorators) |
| `tools` | `tools.ts` | `parse` | Tool nodes + HANDLES_TOOL edges |
| `orm` | `orm.ts` | `parse` | QUERIES edges (Prisma, Supabase) |
| `crossFile` | `cross-file.ts` + `cross-file-impl.ts` | `parse`, `routes`, `tools`, `orm` | Cross-file type propagation in topological import order |
| `scopeResolution` | `scope-resolution/pipeline/phase.ts` | `parse`, `crossFile`, `structure` | Binding/reference + inheritance edges; disposes BindingAccumulator |
| `pruneLocalSymbols` | `prune-local-symbols.ts` | `scopeResolution` | Drops inert block-local `Const`/`Variable`/`Static` nodes (only a `File→DEFINES` edge) post-resolution |
| `mro` | `mro.ts` | `crossFile`, `scopeResolution`, `pruneLocalSymbols`, `structure` | METHOD_OVERRIDES + METHOD_IMPLEMENTS edges |
| `di` | `di.ts` | `mro` | INJECTS edges from consumer Classes or factory Methods to provider Classes/declaration CodeElements (framework-neutral DI resolution; per-language matchers registered in `di-extractors/`) |
| `communities` | `communities.ts` | `mro`, `pruneLocalSymbols`, `structure` | Community nodes + MEMBER_OF edges (Leiden algorithm) |
| `processes` | `processes.ts` | `communities`, `routes`, `tools`, `pruneLocalSymbols`, `structure` | Process nodes + STEP_IN_PROCESS edges |
| Phase | File | Deps | Output |
| ------------------------- | -------------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `scan` | `scan.ts` | (root) | File paths + sizes |
| `structure` | `structure.ts` | `scan` | File/Folder nodes, CONTAINS edges, `allPathSet` |
| `springConfig` | `spring-config.ts` | `structure` | Spring configuration-property nodes and metadata |
| `markdown` | `markdown.ts` | `structure` | Section nodes, cross-link edges from .md/.mdx |
| `cobol` | `cobol.ts` | `structure` | COBOL program/paragraph/section nodes (regex, no tree-sitter) |
| `parse` | `parse.ts` + `parse-impl.ts` | `structure`, `markdown`, `cobol` | Symbol nodes, IMPORTS/CALLS/EXTENDS edges, extracted routes/tools/ORM queries |
| `routes` | `routes.ts` | `parse` | Route nodes + HANDLES_ROUTE edges (Next.js, Expo, PHP, decorators, and JS/TS static route sources — see below) |
| `tools` | `tools.ts` | `parse` | Tool nodes + HANDLES_TOOL edges |
| `orm` | `orm.ts` | `parse` | QUERIES edges (Prisma, Supabase) |
| `crossFile` | `cross-file.ts` + `cross-file-impl.ts` | `parse`, `routes`, `tools`, `orm` | Cross-file type propagation in topological import order |
| `scopeResolution` | `scope-resolution/pipeline/phase.ts` | `parse`, `crossFile`, `structure` | Binding/reference + inheritance edges; disposes BindingAccumulator |
| `springAutoConfiguration` | `spring-auto-configuration.ts` | `structure`, `scopeResolution` | DECLARES and CONDITIONAL_ON metadata for Spring configuration candidates |
| `springAop` | `spring-aop.ts` | `scopeResolution` | Direct declarative/advice ADVISED_BY edges and pointcut evidence |
| `pruneLocalSymbols` | `prune-local-symbols.ts` | `scopeResolution` | Drops inert block-local `Const`/`Variable`/`Static` nodes (only a `File→DEFINES` edge) post-resolution |
| `mro` | `mro.ts` | `crossFile`, `scopeResolution`, `pruneLocalSymbols`, `structure` | METHOD_OVERRIDES + METHOD_IMPLEMENTS edges |
| `springAopInheritance` | `spring-aop.ts` | `springAop`, `mro` | Propagates declarative behavior through class/interface inheritance decisions |
| `di` | `di.ts` | `mro` | INJECTS edges from consumer Classes, factory Methods, or AST-captured programmatic lookup callables to provider Classes/declaration CodeElements (framework-neutral DI resolution; per-language matchers registered in `di-extractors/`) |
| `communities` | `communities.ts` | `mro`, `pruneLocalSymbols`, `structure` | Community nodes + MEMBER_OF edges (Leiden algorithm) |
| `processes` | `processes.ts` | `communities`, `routes`, `tools`, `pruneLocalSymbols`, `structure` | Process nodes + STEP_IN_PROCESS edges |
**Non-phase files in the same directory:** `parse-impl.ts`, `cross-file-impl.ts` (implementation), `wildcard-synthesis.ts` (whole-module import expansion), `types.ts`, `runner.ts`, `index.ts`.
@ -124,6 +129,7 @@ scan → structure → [markdown, cobol] → parse → [routes, tools, orm]
4. **Timing** — per-phase `durationMs` in `PhaseResult`, dev-mode console logging.
**Design patterns:**
- **Single graph accumulator** — all phases mutate the same `KnowledgeGraph` in `ctx`; the graph is the primary output.
- **Typed phase access** — `getPhaseOutput<T>(deps, 'name')` for type-safe upstream results.
- **Binding accumulator lifecycle** — created in `parse`, disposed by `crossFile` (in `finally`). No other phase should take ownership.
@ -141,7 +147,9 @@ import type { PipelinePhase, PhaseResult } from './types.js';
import { getPhaseOutput } from './types.js';
import type { ParseOutput } from './parse.js';
export interface MyPhaseOutput { /* ... */ }
export interface MyPhaseOutput {
/* ... */
}
export const myPhase: PipelinePhase<MyPhaseOutput> = {
name: 'myPhase',
@ -149,11 +157,64 @@ export const myPhase: PipelinePhase<MyPhaseOutput> = {
async execute(ctx, deps) {
const { allPaths } = getPhaseOutput<ParseOutput>(deps, 'parse');
// ... write to ctx.graph ...
return { /* typed output */ };
return {
/* typed output */
};
},
};
```
### Where routes come from
`route-extractors/` holds four independent ways a route can be discovered, all
converging on the routes phase's `(method, url)` registry:
| Source | Shape | Examples |
| --- | --- | --- |
| Filesystem convention | path → URL, no parsing | Next.js `app/`, Expo, PHP |
| Single-file framework route | `isRouteFile` + worker extraction | Laravel `routes/*.php` |
| Cross-file framework route | `discoverRootRouteFiles` + `extractRoutes` | Django `urlpatterns` |
| AST-level route in a normal file | `extractDecoratorRoutes` | Spring, FastAPI, NestJS (`@Controller` + `@Get`/`@Post`/…; URLs are controller-relative — `setGlobalPrefix` and URI versioning live in the bootstrap file and are not applied), **JS/TS dispatch guards and static data route tables** |
The last row is the one whose name undersells it. A route is DECLARED by a
decorator, but it can also be **inferred** from a raw `node:http` server's own
dispatch — `if (req.method === 'GET' && pathname === '/api/x')` is a route with
a path, a verb and a handler, and nothing else in the pipeline could see it.
`route-extractors/dispatch-guard.ts` reads that shape; the transport, dedup and
handler resolution are shared with decorator routes, and
`ExtractedDecoratorRoute.source` carries the provenance difference through to
the `HANDLES_ROUTE` edge.
JS/TS data route tables share that transport when a route-named array contains
direct object literals with static `path`, `method`, and `handler` fields and a
same-scope `for...of` dispatcher positively compares the path and method before
directly invoking the handler. Dynamic values, computed keys, spreads,
inline/called handlers, unknown verbs, and ambiguous handler bindings are
suppressed. Bare import aliases and single-level member handlers are attributed
only through declared import and owner provenance; an unproven receiver never
falls back to a global name guess.
That extractor is deliberately **precision-weighted**: `route_map` presents its
output as fact, so a `startsWith` namespace test, a bare `pathname === '/'`
without a verb, and any regex it cannot translate exactly are all dropped rather
than guessed at. A missing route is a coverage limit; an invented one is a lie.
Two rules there need more than one comparison to decide, and are worth knowing
about before changing either:
- **Same-file constant folding.** `` pathname === `${basePath}/rules` `` is
common enough that refusing it loses whole route modules — and loses them
invisibly, since a module with unfoldable paths and a module with no routes
produce the same empty answer. Folding is same-file, string literals only, one
alias hop, and refuses on ambiguity (a name declared twice with different
values is dropped, never guessed).
- **Whole-repo reconciliation** (`reconcileDispatchGuardRoutes`, applied in the
routes phase). A split route table — one module listing every path it
recognises so the dispatcher can 404 early, handlers in others — otherwise
lists every route twice, once verb-less with the table as its "handler". It
applies to dispatch-guard routes only: a framework route with no verb is
method-agnostic *by declaration*, which is a fact, not a weaker observation.
---
## Semantic model
@ -204,6 +265,9 @@ Language-agnostic scope-resolution resolver. This is the resolution path for eve
│ emitReferencesViaLookup ── uses handledSites + deferred-site skip set
│ emitPropertyDispatchCalls ── registration USES + conservative CALLS
│ emitCallableValueFlow ── assigned/passed callable invocation CALLS
│ emitImportedValueReferences ── cross-file value reads via finalized imports
│ emitUniqueNamePropertyAccesses ── LAST-RESORT property reads by name,
│ narrowed same-file → direct-import, refusing to choose otherwise
│ emitImportEdges
▼
KnowledgeGraph (IMPORTS / CALLS / ACCESSES / INHERITS / USES)
@ -222,13 +286,27 @@ The solver is flow-insensitive but bounded: dependency-indexed work items rerun
Property-key dispatch remains a separate conservative fallback. Its per-key fan-out cap is 32; capped keys synthesize no partial calls and are reported at warning level with language, skipped-key count, dropped key names (bounded), and cap; the count also travels in `RunScopeResolutionStats.propertyDispatchSkippedKeys`.
Interface-dispatch fan-out walks the subtype closure of the receiver's interface and is **generic-instantiation aware** (#2912): a call through `IValidator<string>` must not reach an implementor of `IValidator<int>`, which shares its declaration and therefore its subtype list. Each heritage clause's arguments reach resolution by one of three routes — read off the `@reference.inherits` anchor's own spelling where that anchor spans the whole base (most languages, no query change), through the `@reference.type-arguments` sub-tag where the anchor is the bare name and moving it would renumber inheritance edge ids (Rust `impl T<A> for S`, Dart `extends`), or on a heritage MARKER payload for clauses that never become reference sites (Dart `implements`/`with`). Whichever pass emits the edge records the pair through one sink: `preEmitInheritanceEdges` for heritage clauses, `ScopeResolver.emitHeritageEdges` for the rest.
The walk then carries a substitution: a subtype's own type parameters bind to the receiver's arguments, so `class Wrapper<T> : IValidator<T>` stays reachable from every instantiation while `class IntValidator : IValidator<int>` is pruned from the `string` one. Receiver arguments come from the declared type (Case 4), a class-level field's declared type (Case 6), or — for a compound receiver such as `this._repo` — the spelling the compound fold typed that position from, reported back through `recordReceiverType` and accepted only when it names the class the fold returned.
The filter prunes only on positive evidence: an unknown instantiation on either side, an argument list whose arity does not line up, a name that may be a type variable the language's captures never recorded, or an unresolved spelling whose simple name matches all keep the target. A type parameter of the declaration ENCLOSING either side is recognised as such and never compared — `void Run<T>(IValidator<T> v)` writes a receiver with no known instantiation, so it keeps the unfiltered fan-out. That recognition is what generic METHODS now carry `@declaration.type-parameters` for in C#, Java and Kotlin (TypeScript already did): without it an unbounded `T` grounds to nothing and a bounded one grounds to its BOUND, and both compare unequal to an implementor's concrete argument. Languages that capture neither type arguments nor type parameters therefore emit exactly the pre-#2912 fan-out. The fan-out cap (32, `GITNEXUS_MAX_INTERFACE_DISPATCH_FANOUT`) and its skipped-target reporting are unchanged and apply after filtering. Note the fan-out itself still fires only for a receiver whose folded type is an `Interface` symbol, so a Rust `Trait` or a Dart abstract `Class` receiver emits no secondary targets to filter in the first place.
Standalone (regex-based) providers such as COBOL participate via `ScopeResolver.scopeResolutionEdgeMode: 'callable-flow-only'`: `runScopeResolution` runs for them, but every ordinary emission path — heritage, interface implementations, receiver-bound, free-call fallback, reference/import edges, post-resolution hooks — is gated off, so their legacy phase (e.g. `cobolPhase`) remains the sole owner of structural edges and the callable solver's `CALLS` are purely additive. A callable-flow-only provider whose files emitted no callable facts exits early, before finalize, keeping the opt-in proportional to source scanning.
### Receiver chains and the drop census (#2766)
A compound receiver (`svc.getUser().address.save()`) is captured as a compact string on `ReferenceSite.receiverChain`. `utils/receiver-chain-codec.ts` is the ONE encoder/decoder — capture emitters, the scope-resolution fold, and the durable ParsedFile store all import it rather than hand-rolling the format.
Wire format is **v2**: `2|<base>|<step>|<step>…`, one-character version prefix, then base-first steps, each a one-character kind sigil plus the member name (`c` = call, `f` = field). `a` (await) and `i` (index) are **name-free** and encode as a bare sigil — an awaited call's name already lives on its `c` step, and a subscript key is a value, not a lookup-able identifier. The version went 1 → 2 when those two kinds were added, and a decoder REFUSES a foreign version rather than decoding the prefix it understands: a chain missing its await/index hop decodes cleanly as a different, shorter chain and would type the receiver against the wrong member. The format is unescaped (`|` and `~` cannot occur in an identifier), so an unencodable name is refused rather than escaped, and the payload is capped at `MAX_RECEIVER_CHAIN_BYTES` / `MAX_CHAIN_DEPTH` steps. Because these strings live in the incremental parse cache and the durable ParsedFile store, a format change requires a `PARSE_CACHE_VERSION` schema bump — a stale cache would otherwise replay v1 chains this build discards.
Receivers the resolver could not type are not silently dropped. Each records a `ResolutionOutcome` (`scope-resolution/resolution-outcome.ts`) carrying the receiver's *shape* (`classifyReceiverShape`: `chain-call` / `chain-field` / `chain-mixed` / `chain-unwrap` / `no-chain` — the bench censuses these) and its *origin* (`in-program` / `external` / `unknown`). `scope-resolution/unresolved-receivers.ts` aggregates them per member name into the index-persisted `unresolvedReceiverMembers` summary, keeping in-program and external counts under separate keys. Only in-program drops make a count short: an external-rooted call (`System.out.println`, `fetch(...)`) has no in-graph node an edge could have reached, so it is reported but does not hedge. `impact` / `context` read that summary and publish `epistemic: 'exact' | 'lower-bound'`, prose `boundaries`, and the machine-readable `causes` split (`EpistemicCauses` in `mcp/local/local-backend.ts`).
### Optional CFG/PDG emission (`--pdg`, #2081–#2086)
On a `--pdg` run the parse worker builds a per-function control-flow graph from the tree-sitter AST (`LanguageProvider.cfgVisitor`; TypeScript/JavaScript today) and serializes it onto `ParsedFile.cfgSideChannel` as plain data. Scope-resolution then emits the program-dependence layers from that side-channel **inside Phase 4 of `runScopeResolution`, while the disk-backed ParsedFile store is still live** — the only window where the worker-built CFGs are loaded (the store is cleared right after the phase returns). A standalone post-`mro` phase would read an empty store, so the emit deliberately lives in-phase, mirroring the `applyCaptureSideChannel` pattern. The opt-in is off by default (graph byte-identical), folded into the parse-cache key (a pdg-off warm cache is never reused on a `--pdg` run), and each layer is bounded by a per-function edge cap that logs any dropped edges. All layers are `BasicBlock → BasicBlock` edges in the single `CodeRelation` table, keyed by `type`; there is **no** `Function → BasicBlock` edge — the symbol↔block join is reconstructed from the BasicBlock id prefix + line span. The layers build on each other:
- **M1 — CFG** (#2081): `BasicBlock` nodes + `CFG` edges. Edge *kind* (`seq`/`cond-true`/`loop-back`/…) rides the `reason` column (CFG is one `CodeRelation` type, not one per kind).
- **M1 — CFG** (#2081): `BasicBlock` nodes + `CFG` edges. Edge _kind_ (`seq`/`cond-true`/`loop-back`/…) rides the `reason` column (CFG is one `CodeRelation` type, not one per kind).
- **M2 — REACHING_DEF** (#2082): GEN/KILL def→use data dependence from a pure fixpoint solver; the variable name rides `reason`.
- **M3/M4 — TAINTED / SANITIZES / TAINT_PATH** (#2083–#2084): intra- and inter-procedural taint (source→sink) — the `explain` tool's data.
- **M5 — CDG** (#2085): Ferrante control dependence over a Cooper–Harvey–Kennedy post-dominator tree (the EXIT-rooted reverse CFG); branch sense (`'T'`/`'F'`) rides `reason`. A CFG whose EXIT is unreachable from some block is skipped for CDG (post-dominance would be unsound) while its CFG/REACHING_DEF layers are kept.
@ -241,24 +319,26 @@ See `core/ingestion/cfg/` (emit + the pure CFG / post-dominator / control-depend
Single interface a language implements to plug into the pipeline. Contract fully documented in `scope-resolution/contract/scope-resolver.ts`.
| Hook | Purpose |
|------|---------|
| `languageProvider` | Base `LanguageProvider` (tree-sitter query, `emitScopeCaptures`, import/binding interpreters, hooks) |
| `populateOwners(parsed)` | Fill deferred `ownerId` fields on method defs (captures can't always know the owning class at parse time) |
| `buildMro(graph, parsed, nodeLookup)` | Produce `mroByClassDefId: Map<DefId, DefId[]>` — C3, Ruby-mixin, or first-wins per language |
| `resolveImportTarget(target, fromFile, allFiles)` | `(rawImportPath, sourceFile) → targetFilePath` (PEP-328 for Python, etc.) |
| `mergeBindings(existing, incoming, scopeId)` | Shadowing / LEGB precedence |
| `arityCompatibility` | Provider consumed by registry during `MethodRegistry.lookup` Step 2 |
| `importEdgeReason` | Confidence-tier string for IMPORTS edge reason field |
| `propagatesReturnTypesAcrossImports?` | Opt out of cross-file return-type propagation (default on) |
| `fieldFallbackOnMethodLookup?` | Statically-typed languages turn this OFF — the heuristic over-connects (default on) |
| `unwrapCollectionAccessor?` | Property-style collection views (`data.Values` on Dictionary-like receivers) — default off |
| `collapseMemberCallsByCallerTarget?` | One CALLS edge per (caller, target) instead of per-site — default off |
| `populateNamespaceSiblings?` | Cross-file implicit visibility (compiler-implicit namespace sharing) — default off; ctx carries `treeCache` |
| `hoistTypeBindingsToModule?` | Walk up to Module scope when looking up a method's return-type typeBinding — default off; enable only when bindings are stored at module level |
| `hasFileLocalCallableLinkage?` | Precise internal-linkage predicate used only when joining callable declarations/prototypes to cross-file definitions; C/C++ use it for `static` free functions |
| `constructorCallTargetsClass?` | A constructor-form call `Type(...)` links to the Class def rather than its explicit Constructor def — default off; Swift and Dart opt in |
| `constructionSyntax?` | How the language spells construction, so an INLINE constructor receiver (`Service(db).m()`, `new Service(db).m()`, `Service.new.m()`) can be typed — `bare` / `keyword` / `selector`; default off, opt in per language only where measured to be needed (#2708) |
| Hook | Purpose |
| ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `languageProvider` | Base `LanguageProvider` (tree-sitter query, `emitScopeCaptures`, import/binding interpreters, hooks) |
| `populateOwners(parsed)` | Fill deferred `ownerId` fields on method defs (captures can't always know the owning class at parse time) |
| `buildMro(graph, parsed, nodeLookup)` | Produce `mroByClassDefId: Map<DefId, DefId[]>` — C3, Ruby-mixin, or first-wins per language |
| `resolveImportTarget(target, fromFile, allFiles)` | `(rawImportPath, sourceFile) → targetFilePath` (PEP-328 for Python, etc.) |
| `isNamespaceImport(parsedImport, targetFile, fromFile)` | Optionally reclassify a resolved named import as a namespace handle when the imported symbol is itself a module |
| `mergeBindings(existing, incoming, scopeId)` | Shadowing / LEGB precedence |
| `arityCompatibility` | Provider consumed by registry during `MethodRegistry.lookup` Step 2 |
| `importEdgeReason` | Confidence-tier string for IMPORTS edge reason field |
| `propagatesReturnTypesAcrossImports?` | Opt out of cross-file return-type propagation (default on) |
| `fieldFallbackOnMethodLookup?` | Statically-typed languages turn this OFF — the heuristic over-connects (default on) |
| `elementTypeOf?` | `(containerType, via: {kind:'index'} \| {kind:'accessor',name}) → elementType \| undefined` — element type of a container, reached by subscript (`repos[0]`) or by a property-style collection view (`data.Values`). ONE hook for both routes (it replaced the split `unwrapCollectionAccessor` / `unwrapCollectionElement`, where implementing one silently answered nothing for the other). Consulted only where the source actually performed the access — never as a general type-name normalizer |
| `stripTypePreservingDecoration?` | `(typeName) → strippedName \| undefined` — strip ONE layer of TYPE-PRESERVING decoration (pointer, reference, `const`, nullable, borrow, sigil) so a receiver declared `*Host` still finds the `Host` binding (#2766). Never a container: unwrapping `Repo[]` here would fold `repos.find(x)` to `Repo.find` — that is `elementTypeOf`'s job, and only after a real subscript. Consulted only after every undecorated lookup fails, and only by receiver-chain base/step resolution — default off |
| `collapseMemberCallsByCallerTarget?` | One CALLS edge per (caller, target) instead of per-site — default off |
| `populateNamespaceSiblings?` | Cross-file implicit visibility (compiler-implicit namespace sharing) — default off; ctx carries `treeCache` |
| `hoistTypeBindingsToModule?` | Walk up to Module scope when looking up a method's return-type typeBinding — default off; enable only when bindings are stored at module level |
| `hasFileLocalCallableLinkage?` | Precise internal-linkage predicate used only when joining callable declarations/prototypes to cross-file definitions; C/C++ use it for `static` free functions |
| `constructorCallTargetsClass?` | A constructor-form call `Type(...)` links to the Class def rather than its explicit Constructor def — default off; Swift and Dart opt in |
| `constructionSyntax?` | How the language spells construction, so an INLINE constructor receiver (`Service(db).m()`, `new Service(db).m()`, `Service.new.m()`) can be typed — `bare` / `keyword` / `selector`; default off, opt in per language only where measured to be needed (#2708) |
### Per-language registration
@ -269,21 +349,21 @@ CI auto-discovers the set via `tsx`. No workflow edit required.
### Code references
| Module | Purpose |
|--------|---------|
| `scope-resolution/contract/scope-resolver.ts` | `ScopeResolver` interface + shared types |
| `scope-resolution/pipeline/run.ts` | Generic orchestrator |
| `scope-resolution/pipeline/phase.ts` | Pipeline-phase wrapper (deps: `parse`, `structure`) |
| `scope-resolution/pipeline/registry.ts` | `SCOPE_RESOLVERS` map |
| `scope-resolution/passes/*.ts` | Reference-resolution passes (receiver-bound, free-call fallback, compound-receiver, MRO, cross-file return-type propagation) |
| `scope-resolution/graph-bridge/*.ts` | CLI-local translation from resolved references → `KnowledgeGraph` edges |
| `scope-resolution/scope/*.ts` | Generic scope-chain walkers + namespace targets |
| `scope-resolution/workspace-index.ts` | Build-once O(1) lookup index |
| `languages/python/index.ts` | Python `ScopeResolver` hooks + known-limitation docs |
| `languages/python/captures.ts` | `emitPythonScopeCaptures` (honors cross-phase Tree cache) |
| `languages/csharp/index.ts` | C# `ScopeResolver` hooks + known-limitation docs |
| `languages/csharp/captures.ts` | `emitCsharpScopeCaptures` (honors cross-phase Tree cache) |
| `languages/csharp/namespace-siblings.ts` | Cross-file implicit-namespace visibility hook (reads `treeCache`) |
| Module | Purpose |
| --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `scope-resolution/contract/scope-resolver.ts` | `ScopeResolver` interface + shared types |
| `scope-resolution/pipeline/run.ts` | Generic orchestrator |
| `scope-resolution/pipeline/phase.ts` | Pipeline-phase wrapper (deps: `parse`, `structure`) |
| `scope-resolution/pipeline/registry.ts` | `SCOPE_RESOLVERS` map |
| `scope-resolution/passes/*.ts` | Reference-resolution passes (receiver-bound, free-call fallback, compound-receiver, MRO, cross-file return-type propagation) |
| `scope-resolution/graph-bridge/*.ts` | CLI-local translation from resolved references → `KnowledgeGraph` edges |
| `scope-resolution/scope/*.ts` | Generic scope-chain walkers + namespace targets |
| `scope-resolution/workspace-index.ts` | Build-once O(1) lookup index |
| `languages/python/index.ts` | Python `ScopeResolver` hooks + known-limitation docs |
| `languages/python/captures.ts` | `emitPythonScopeCaptures` (honors cross-phase Tree cache) |
| `languages/csharp/index.ts` | C# `ScopeResolver` hooks + known-limitation docs |
| `languages/csharp/captures.ts` | `emitCsharpScopeCaptures` (honors cross-phase Tree cache) |
| `languages/csharp/namespace-siblings.ts` | Cross-file implicit-namespace visibility hook (reads `treeCache`) |
### Performance notes
@ -297,7 +377,7 @@ CI auto-discovers the set via `tsx`. No workflow edit required.
## Language-agnostic graph feeding
16 languages → single unified graph. Four abstraction layers:
18 languages → single unified graph. Four abstraction layers:
```
Unified Graph Schema (44 node types, 21 relationship types)
@ -313,18 +393,19 @@ CI auto-discovers the set via `tsx`. No workflow edit required.
Each language implements `LanguageProvider` (`language-provider.ts`). Key fields:
| Field | Purpose |
|-------|---------|
| `id`, `extensions` | Language identity and file matching |
| `treeSitterQueries` | S-expression queries for AST extraction |
| `importSemantics` | `named` / `wildcard-leaf` / `wildcard-transitive` / `namespace` |
| `importResolver` | Language-specific path → file resolution |
| `exportChecker` | Public/exported symbol detection |
| `typeConfig` | Type annotation extraction rules |
| `mroStrategy` | `first-wins` / `c3` / `none` |
| Field | Purpose |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id`, `extensions` | Language identity and file matching |
| `treeSitterQueries` | S-expression queries for AST extraction |
| `importSemantics` | `named` / `wildcard-leaf` / `wildcard-transitive` / `namespace` |
| `importResolver` | Language-specific path → file resolution |
| `exportChecker` | Public/exported symbol detection |
| `typeConfig` | Type annotation extraction rules |
| `mroStrategy` | `first-wins` / `c3` / `none` |
| `descriptionExtractor` | Optional hook returning a symbol's doc-comment text as its `description`; feeds the embedding metadata header so doc-only terms are semantically searchable (issue #2270). Most languages register `createLeadingDocDescriptionExtractor` (shared, language-neutral; per-language comment/wrapper config passed at the call site) |
| `definitionPropertiesExtractor` | Optional language-owned hook for structured, clone-safe definition metadata. Shared ingestion persists these properties opaquely; the owning provider supplies the extraction semantics. |
16 providers in `languages/index.ts` via `satisfies Record<SupportedLanguages, LanguageProvider>` — missing a language is a compile error.
18 providers in `languages/index.ts` via `satisfies Record<SupportedLanguages, LanguageProvider>` — missing a language is a compile error.
### Unified capture tags
@ -336,22 +417,23 @@ Per-language import resolution uses the **configs + factory** pattern (like call
Unified 3-tier algorithm (`model/resolution-context.ts`), per-language `importSemantics` controls which tier activates:
| Tier | Confidence | Mechanism |
|------|-----------|-----------|
| 1 — same-file | 0.95 | Symbol table for caller's file |
| 2 — import-scoped | 0.9 | `NamedImportMap` chains (named) or all files in `importMap` (wildcard) |
| 3 — global | 0.5 | O(1) index lookups: class, impl, callable. Fallback only |
| Tier | Confidence | Mechanism |
| ----------------- | ---------- | ---------------------------------------------------------------------- |
| 1 — same-file | 0.95 | Symbol table for caller's file |
| 2 — import-scoped | 0.9 | `NamedImportMap` chains (named) or all files in `importMap` (wildcard) |
| 3 — global | 0.5 | O(1) index lookups: class, impl, callable. Fallback only |
| Import strategy | Languages | Behavior |
|----------------|-----------|----------|
| `named` | TS, JS, Java, C#, Rust, PHP, Kotlin | Only explicitly imported names visible |
| `wildcard-leaf` | Go, Ruby, Swift, Dart | Whole-package import, no transitive re-exports |
| `wildcard-transitive` | C, C++ | `#include` closure chains through re-exports |
| `namespace` | Python | Module aliases resolved at call site |
| Import strategy | Languages | Behavior |
| --------------------- | ----------------------------------- | ---------------------------------------------- |
| `named` | TS, JS, Java, C#, Rust, PHP, Kotlin | Only explicitly imported names visible |
| `wildcard-leaf` | Go, Ruby, Swift, Dart | Whole-package import, no transitive re-exports |
| `wildcard-transitive` | C, C++ | `#include` closure chains through re-exports |
| `namespace` | Python | Module aliases resolved at call site |
### Chunked parse-and-resolve
`parse` processes files in ~20 MB byte-budget chunks to bound memory. Per chunk:
1. Worker pool dispatches files (the sole parse path — there is no sequential fallback; `skipWorkers`, `--workers 0`, and `GITNEXUS_WORKER_POOL_SIZE=0` are rejected with an actionable error)
2. Each worker: detect language → load grammar → run queries → return unified `ParseWorkerResult`
3. Synthesize wildcard bindings (`wildcard-synthesis.ts`)
@ -362,11 +444,12 @@ Inheritance edges are emitted later, by the scope-resolution phase (`preEmitInhe
Workers: `workers/worker-pool.ts`, `workers/parse-worker.ts`.
**Worker-serialized ParsedFiles (#2038).** To index very large repos (e.g. the Linux kernel) without OOM, the worker pool is the *sole* parse path and workers serialize each file's `ParsedFile` (plus its capture side-channel) in parallel, streaming them to scope-resolution through a disk-backed store. Scope-resolution consumes the pre-extracted artifact instead of re-parsing every file on the main thread — tree-sitter's native input buffers are not GC-reclaimable, so the former main-thread re-parse leaked native memory until the process died. Pool creation is lazy / cache-miss-gated, so a warm all-cache-hit run replays cached worker output without spawning a worker (hence `usedWorkerPool` can be false even when the repo has parseable files).
**Worker-serialized ParsedFiles (#2038).** To index very large repos (e.g. the Linux kernel) without OOM, the worker pool is the _sole_ parse path and workers serialize each file's `ParsedFile` (plus its capture side-channel) in parallel, streaming them to scope-resolution through a disk-backed store. Scope-resolution consumes the pre-extracted artifact instead of re-parsing every file on the main thread — tree-sitter's native input buffers are not GC-reclaimable, so the former main-thread re-parse leaked native memory until the process died. Pool creation is lazy / cache-miss-gated, so a warm all-cache-hit run replays cached worker output without spawning a worker (hence `usedWorkerPool` can be false even when the repo has parseable files).
### Inheritance and MRO
Inheritance is captured by the `@reference.inherits` tag and emitted by the scope-resolution phase: `preEmitInheritanceEdges` resolves each base in scope, then `emitHeritageEdges` writes the `EXTENDS`/`IMPLEMENTS` edges. The phase then computes method resolution order via each `ScopeResolver`'s `buildMro` hook, feeding a `MethodDispatchIndex` used for owner-scoped lookups. Per-language strategy:
- **`first-wins`** — Java, C#, C++, TS, Ruby, Go
- **`c3`** — Python (C3 linearization)
- **`ruby-mixin`** — Ruby (mixin-aware linearization)
@ -418,7 +501,7 @@ Defined in `lbug/schema.ts`. Separate node tables per type, single `CodeRelation
**Node tables:** File, Folder, Function, Class, Interface, Method, Constructor, CodeElement, Struct, Enum, Macro, Typedef, Union, Namespace, Trait, Impl, TypeAlias, Const, Static, Property, Record, Delegate, Annotation, Template, Module, Community, Process, Route, Tool, Section, Embedding.
**Relation types** (`CodeRelation.type`): CONTAINS, DEFINES, CALLS, IMPORTS, EXTENDS, IMPLEMENTS, HAS_METHOD, HAS_PROPERTY, ACCESSES, METHOD_OVERRIDES, METHOD_IMPLEMENTS, MEMBER_OF, STEP_IN_PROCESS, HANDLES_ROUTE, FETCHES, HANDLES_TOOL, ENTRY_POINT_OF.
**Relation types** (`CodeRelation.type`): CONTAINS, DEFINES, CALLS, IMPORTS, INHERITS, EXTENDS, IMPLEMENTS, USES, DECORATES, HAS_METHOD, HAS_PROPERTY, ACCESSES, METHOD_OVERRIDES, METHOD_IMPLEMENTS, MEMBER_OF, STEP_IN_PROCESS, HANDLES_ROUTE, FETCHES, HANDLES_TOOL, ENTRY_POINT_OF, WRAPS, QUERIES, INJECTS, CONDITIONAL_ON, DECLARES, ADVISED_BY, BINDS_EVENT_HANDLER, EMITS_EVENT.
**Optional `--pdg` additions** (off by default, opt-in via `gitnexus analyze --pdg`; see _Optional CFG/PDG emission_ above): a `BasicBlock` node table, plus the PDG relation types `CFG`, `REACHING_DEF`, `CDG`, `TAINTED`, `SANITIZES`, and `TAINT_PATH` on the same `CodeRelation` table. These are deliberately kept out of the default `VALID_RELATION_TYPES` / web graph schema — query them via `cypher`, `explain`, or `pdg_query`.
@ -446,12 +529,12 @@ Node IDs use arity suffix (`#<paramCount>`): `Method:file:Class.method#1` vs `#2
**METHOD_IMPLEMENTS confidence tiering:**
| Match quality | Confidence |
|---|---|
| Exact parameter types match | 1.0 |
| Arity match, types unavailable | 1.0 |
| Variadic vs fixed | 0.7 |
| Insufficient info | 0.7 |
| Match quality | Confidence |
| ------------------------------ | ---------- |
| Exact parameter types match | 1.0 |
| Arity match, types unavailable | 1.0 |
| Variadic vs fixed | 0.7 |
| Insufficient info | 0.7 |
## Related docs
@ -459,4 +542,5 @@ Node IDs use arity suffix (`#<paramCount>`): `Method:file:Class.method#1` vs `#2
- [RUNBOOK.md](RUNBOOK.md) — operational commands and recovery
- [GUARDRAILS.md](GUARDRAILS.md) — safety boundaries for humans and agents
- [TESTING.md](TESTING.md) — how to run tests
- [docs/languages/objective-c-provider.md](docs/languages/objective-c-provider.md) — Objective-C provider behavior and limits
- `AGENTS.md` / `CLAUDE.md` — agent workflows and tool usage

25
CLAUDE.md
View file

@ -62,30 +62,31 @@ See the `<!-- gitnexus:start --> … <!-- gitnexus:end -->` block in **[AGENTS.m
<!-- gitnexus:start -->
# GitNexus — Code Intelligence
This project is indexed by GitNexus as **GitNexus** (20319 symbols, 54304 relationships, 300 execution flows). Use the GitNexus MCP tools to understand code, assess impact, and navigate safely.
This project is indexed by GitNexus as **GitNexus** (248612 symbols, 565510 relationships, 918 execution flows).
> Index stale? Run `node .gitnexus/run.cjs analyze` from the project root — it auto-selects an available runner. No `.gitnexus/run.cjs` yet? `npx gitnexus analyze` (npm 11 crash → `npm i -g gitnexus`; #1939).
> Index stale? Run `node .gitnexus/run.cjs analyze --index-only` from the project root — it auto-selects an available runner. No `.gitnexus/run.cjs` yet? Bootstrap with `npx`, `bunx`, or `pnpm dlx` — e.g. `bunx gitnexus@latest analyze` (npm 11 npx crash; #1939).
## Always Do
- **MUST run impact analysis before editing any symbol.** Before modifying a function, class, or method, run `impact({target: "symbolName", direction: "upstream"})` and report the blast radius (direct callers, affected processes, risk level) to the user.
- **MUST run `detect_changes()` before committing** to verify your changes only affect expected symbols and execution flows. For regression review, compare against the default branch: `detect_changes({scope: "compare", base_ref: "main"})`.
- **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits.
- When exploring unfamiliar code, use `query({search_query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance.
- When you need full context on a specific symbol — callers, callees, which execution flows it participates in — use `context({name: "symbolName"})`.
- **MUST run impact analysis before editing.** Use `impact({target: "symbolName", direction: "upstream"})` (MCP) or `node .gitnexus/run.cjs impact "symbolName" --direction upstream --repo .` (CLI fallback); report callers, processes, and risk. Never substitute grep for graph analysis. For unified PDG impact, add `mode: "pdg"` with optional `line: <N>` — it returns statement-level `affectedStatements` over CDG + REACHING_DEF and inter-procedural symbols in `interproceduralByDepth`/`byDepth`; no-layer/degraded PDG results are UNKNOWN-risk notes (`--pdg` layer). CLI equivalent: `node .gitnexus/run.cjs impact "symbolName" --direction upstream --mode pdg --line <N> --repo .`.
- **MUST analyze graph changes before committing.** Use `detect_changes({scope: "all"})` (MCP) or `node .gitnexus/run.cjs detect-changes --scope all --repo .` (CLI fallback). `partial: true` or `truncated: true` is not a clean check — a zero means unseen, not unaffected; re-run it. For regression review: `detect_changes({scope: "compare", base_ref: "main"})` or `node .gitnexus/run.cjs detect-changes --scope compare --base-ref "main" --repo .`.
- MUST warn on HIGH/CRITICAL `risk` pre-edit; never use `riskSharedAxes` to waive a HIGH/CRITICAL `risk` warning. Compare File/symbol: MCP File omits axes; Graph-RAG expands File.
- **MUST treat `risk: UNKNOWN` as unresolved, not as low.** An empty caller set is not evidence the symbol is unused — it can also mean the callers are not resolvable by the index (plain-object property access, dynamic dispatch, cross-language calls). `impact` pairs `UNKNOWN` with a `riskNote` saying so. Confirm with a text search before treating the symbol as safe to change or delete; do not proceed on the strength of a zero.
- **MUST use `query({search_query: "concept"})` for concepts/flows, `context({name: "symbolName"})` for a named symbol, or `impact` for blast radius, on read-only callers, dependencies, imports, or execution flow.** Graph first; text search only for empty/`UNKNOWN`/literals.
- For security review, `explain({target: "fileOrSymbol"})` lists taint findings (source→sink flows; needs `analyze --pdg`).
- For control/data dependence, `pdg_query({mode: "controls", target: "fileOrSymbol"})` answers "under what condition does X run?" (CDG, incl. guard clauses) and `pdg_query({mode: "flows", target, variable})` traces "where does variable Y flow?" (REACHING_DEF). `--pdg` layer.
## Never Do
- NEVER edit a function, class, or method without first running `impact` on it.
- NEVER ignore HIGH or CRITICAL risk warnings from impact analysis.
- NEVER edit a function, class, or method before MCP/CLI impact analysis.
- NEVER ignore HIGH or CRITICAL risk warnings from impact analysis, and never read `UNKNOWN` as an all-clear — it means the walk could not answer, which is the one verdict that requires confirming by other means.
- NEVER rename symbols with find-and-replace — use `rename` which understands the call graph.
- NEVER commit changes without running `detect_changes()` to check affected scope.
- NEVER commit before MCP/CLI graph change analysis.
## Resources
| Resource | Use for |
|----------|---------|
| --- | --- |
| `gitnexus://repo/GitNexus/context` | Codebase overview, check index freshness |
| `gitnexus://repo/GitNexus/clusters` | All functional areas |
| `gitnexus://repo/GitNexus/processes` | All execution flows |
@ -94,7 +95,7 @@ This project is indexed by GitNexus as **GitNexus** (20319 symbols, 54304 relati
## CLI
| Task | Read this skill file |
|------|---------------------|
| --- | --- |
| Understand architecture / "How does X work?" | `.claude/skills/gitnexus-exploring/SKILL.md` |
| Blast radius / "What breaks if I change X?" | `.claude/skills/gitnexus-impact-analysis/SKILL.md` |
| Trace bugs / "Why is X failing?" | `.claude/skills/gitnexus-debugging/SKILL.md` |

1
CONTRIBUTING.md
View file

@ -73,6 +73,7 @@ Commits within a PR may use any style — only the **merged PR title** shows up
- [ ] Typecheck passes: `npx tsc --noEmit` in `gitnexus/` and `npx tsc -b --noEmit` in `gitnexus-web/`.
- [ ] No secrets, tokens, or machine-specific paths committed.
- [ ] Documentation updated if behavior or public CLI/MCP contract changes.
- [ ] Every new `GITNEXUS_*` environment variable has a row in the **Environment variables** table in [README.md](README.md) — variable, default, effect, and when to tune it.
- [ ] Pre-commit hook runs clean (`.husky/pre-commit` — formatting via lint-staged + typecheck for staged packages; tests run in CI only).
## Code review

12
Dockerfile.cli
View file

@ -51,8 +51,9 @@ RUN npm run postinstall --prefix gitnexus
# node:22-bookworm-slim
FROM node:22-bookworm-slim@sha256:9f6d5975c7dca860947d3915877f85607946403fc55349f39b4bc3688448bb6e AS runtime
# curl for the healthcheck; git for cloning; ca-certificates for TLS verification.
RUN apt-get update && apt-get install -y --no-install-recommends curl git ca-certificates && rm -rf /var/lib/apt/lists/* \
# curl for the healthcheck; git for cloning; procps for watch process identity;
# ca-certificates for TLS verification.
RUN apt-get update && apt-get install -y --no-install-recommends curl git procps ca-certificates && rm -rf /var/lib/apt/lists/* \
&& rm -rf /usr/local/lib/node_modules/npm \
&& rm -rf /usr/local/lib/node_modules/corepack \
&& rm -f /usr/local/bin/npm /usr/local/bin/npx /usr/local/bin/corepack
@ -120,10 +121,13 @@ USER node
# The web UI defaults to http://localhost:4747 - keep that contract.
ENV GITNEXUS_HOME=/data/gitnexus \
GITNEXUS_NO_UPDATE_NOTIFIER=1 \
NODE_ENV=production \
PORT=4747
EXPOSE 4747
# Bind to 0.0.0.0 so the server is reachable from the host's mapped port.
CMD ["node", "gitnexus/dist/cli/index.js", "serve", "--host", "0.0.0.0", "--port", "4747"]
# Bind 0.0.0.0 for the host's mapped port, honoring an injected $PORT (Render
# sets one). `sh -c` expands it; `exec` keeps the server PID 1 so SIGTERM still
# reaches it. Platforms can rely on this instead of a dockerCommand override.
CMD ["sh", "-c", "exec gitnexus serve --host 0.0.0.0 --port \"${PORT:-4747}\""]

28
GUARDRAILS.md
View file

@ -31,20 +31,38 @@ Format: **Trigger → Instruction → Reason**. Append new Signs when the same m
### Stale graph after edits
- **Trigger:** MCP warns index is behind `HEAD`, or search doesn't match latest commit.
- **Do:** `npx gitnexus analyze` (plus `--embeddings` if used). Runs incrementally by default — the pipeline parses every file every run (cross-file resolution requires it), but tree-sitter dispatch is skipped for unchanged file chunks via the content-addressed cache, and only changed-file rows (plus their importers, transitively) are rewritten in LadybugDB. When the effective write set exceeds ~50% of the repo's files (minimum 50 files), the run transparently switches to the full wipe + bulk-COPY write plan and logs "switching to a full DB write" — expected behavior, not a bug, and file-level bookkeeping stays incremental.
- **Do:** `npx gitnexus analyze` (plus `--embeddings` if used). Runs incrementally by default — the pipeline parses every file every run (cross-file resolution requires it), but tree-sitter dispatch is skipped for unchanged file chunks via the content-addressed cache, and only changed-file rows (plus their importers, transitively) are rewritten in LadybugDB. When the effective write set exceeds ~50% of the repo's files (minimum 50 files), the run transparently switches to the full wipe + bulk-COPY write plan and logs "switching to a full DB write" — expected behavior, not a bug, and file-level bookkeeping stays incremental. That same line also appears — regardless of write-set size, even for a one-file change — when a LadybugDB extension the existing index depends on cannot load on this machine (VECTOR, #2623; FTS, #2841), because a DB carrying those indexes refuses all row-level DML until the extension is loaded; run `gitnexus doctor` for live extension status and re-run with `GITNEXUS_LBUG_EXTENSION_INSTALL=auto` (with network access) to allow one bounded install attempt. The rebuild is one-shot: it clears the indexes, so the next run goes back to the incremental plan.
- **Why:** Tools query LadybugDB from last analyze; git changes are invisible until re-indexed.
### Index seems corrupt or "incremental" is misbehaving
- **Trigger:** `analyze` produces unexpected results, or `incrementalInProgress` is set in the index metadata (`.gitnexus/gitnexus.json` / legacy `meta.json`), or the index is in a half-state after a crash.
- **Do:** `npx gitnexus analyze --force` to rebuild from scratch. The dirty-flag check forces this automatically when a previous incremental run didn't complete cleanly, but `--force` is the manual escape hatch. A dirty-flag recovery rebuild parks the interrupted run's sidecars beside the DB as `lbug.wal.dirty-recovery` / `lbug.shadow.dirty-recovery` for post-mortem debugging — harmless, and removable with `npx gitnexus clean --lbug-sidecars`. Safe to delete the `.gitnexus/parse-cache/` directory (and any legacy `.gitnexus/parse-cache.json`) at any time — content-addressed, will be regenerated.
- **Do:** `npx gitnexus analyze --force` to rebuild the graph and FTS indexes. This may reuse unchanged parser output; when debugging parser/capture changes, use `npx gitnexus analyze --no-parse-cache` to rebuild that output too. The dirty-flag check forces the graph rebuild automatically when a previous incremental run didn't complete cleanly. A dirty-flag recovery rebuild parks the interrupted run's sidecars beside the DB as `lbug.wal.dirty-recovery` / `lbug.shadow.dirty-recovery` for post-mortem debugging — harmless, and removable with `npx gitnexus clean --lbug-sidecars`. Safe to delete the `.gitnexus/parse-cache/` directory (and any legacy `.gitnexus/parse-cache.json`) at any time — content-addressed, will be regenerated.
- **Why:** Incremental writeback is selective DB row replacement; if the on-disk state is inconsistent for any reason, a full rebuild is the cheapest path back to a known-good index.
### Embeddings vanished after analyze
- **Trigger:** Semantic search quality drops; `stats.embeddings` in the index metadata (`gitnexus.json` / legacy `meta.json`) is 0 after refresh.
- **Do:** Re-run `npx gitnexus analyze --embeddings` to regenerate. Check the analyze log for a `Warning: could not load cached embeddings` line — if present, the cache restore failed (corrupt DB / schema mismatch) and the rebuild had nothing to preserve. If you intentionally passed `--drop-embeddings`, this is expected.
- **Why:** Plain `analyze` preserves prior vectors by re-inserting them after the rebuild; the only ways to end up at zero are an explicit `--drop-embeddings`, a cache-load failure (now logged), or a model/dimension change that invalidates the cache. A dirty-recovery run that cannot move the crashed WAL aside now either discards it (logged: forensics lost, embeddings still preserved) or fails fast with a lock error naming the holder — it never silently zeroes embeddings.
- **Why:** Plain `analyze` preserves prior vectors by re-inserting them after the rebuild; ways to end up at zero include an explicit `--drop-embeddings`, a cache-load failure (now logged), or a model/dimension change that invalidates the cache — but zero is no longer the only embedding-loss signature to watch for; see the Sign below for the non-zero, partial-failure case. A dirty-recovery run that cannot move the crashed WAL aside now either discards it (logged: forensics lost, embeddings still preserved) or fails fast with a lock error naming the holder — it never silently zeroes embeddings.
### Analyze finishes but embeddings are incomplete (partial embedding index)
- **Trigger:** `npx gitnexus status` reports `incompleteReasons: ["embedding-checkpoint-pending"]` (or the human-readable "Index incomplete reasons" line); `stats.embeddings` is honest and **non-zero**, and the preceding analyze log showed a `Warning: N node(s) lost their embeddings to embedding-endpoint failures` line (#2790).
- **Do:** Re-run plain `npx gitnexus analyze` — no `--embeddings` flag needed. A retained `embeddingCheckpoint` in the index metadata forces embedding generation for exactly the pending nodes regardless of flags, and clears once they succeed. `--drop-embeddings` abandons the pending nodes instead of retrying them; `--force` also discards the checkpoint (with a warning) and rebuilds without resuming it.
- **Why:** A long analyze run against a flaky HTTP embedding endpoint tolerates bounded sub-batch failures instead of aborting the whole run: it deletes the affected nodes' embedding rows (so they hold zero rows, never a partial set) and records those nodes as pending in `embeddingCheckpoint`. `stats.embeddings` stays an honest, non-zero count of everything that did succeed, so this state never trips the "Embeddings vanished" Sign above — `embedding-checkpoint-pending` is the only reliable signal.
### Scope extraction is incomplete
- **Trigger:** `npx gitnexus status` reports `incompleteReasons: ["scope-extraction-failed"]` when files were omitted, or `incompleteReasons: ["scope-extraction-unverified"]` when the index predates the completeness receipt or its metadata is unreadable. `impact`/`context` reports the same uncertainty as `epistemic: "lower-bound"`; confirmed omissions set `causes.scopeExtractionFiles > 0`.
- **Do:** Re-run `npx gitnexus analyze` (`--force` for a full graph rebuild). If the reason persists, inspect the scope-extraction warnings and treat impact counts as floors until the affected source is supported or corrected.
- **Why:** Parsing continued, but scope captures for the reported file count could not be produced even after the main-thread fallback. Calls, inheritance, imports, or accesses originating there may therefore be absent from the graph.
### Analyze reports INCOMPLETE with a collapsed graph write
- **Trigger:** `npx gitnexus status` reports `incompleteReasons: ["graph-write-collapsed"]`; the analyze summary printed `Repository indexed INCOMPLETELY` naming an expected and a persisted relationship count, and the CLI exited non-zero.
- **Do:** Re-run `npx gitnexus analyze --force`. If it recurs, check free disk space on the volume holding `.gitnexus/`, confirm no second `analyze` is running against the same repo (both stage through `.gitnexus/csv`), then run `npx gitnexus doctor`.
- **Why:** The run finished and wrote metadata, but far fewer relationships are readable back than the pipeline produced. Nothing throws: the DB holds rows and the metadata is valid, so every query answers with missing edges rather than an error — a confident empty answer, which is worse than a failure because it looks like a result. Unlike `incremental-in-progress` and `embedding-checkpoint-pending`, which describe a run that did what it said and left work for next time, this one means most of your edges are gone, so it is the one incomplete reason that also fails the exit code. The check compares in-memory totals (including rows streamed out of the heap) against the post-write count, refuses to answer when the count cannot be read, and is skipped on incremental runs where whole-scope counts are not comparable.
### MCP lists no repos
@ -55,8 +73,8 @@ Format: **Trigger → Instruction → Reason**. Append new Signs when the same m
### Wrong repo in multi-repo setups
- **Trigger:** Query/impact results belong to another project.
- **Do:** Call `list_repos`, then pass `repo` on subsequent tools.
- **Why:** Default target is ambiguous when multiple repos are registered.
- **Do:** Confirm an MCP default is configured or the GitNexus process was launched inside the intended registered path without crossing into an unindexed nested Git checkout. Otherwise call `list_repos`, then pass `repo` on subsequent tools; pass it for mutating tools when multiple repos are registered and no MCP default exists.
- **Why:** Read-only tools derive their default from MCP configuration or a process cwd that stays within one registered Git boundary. Outside those paths the target remains ambiguous, and mutating tools stay explicit unless configuration supplies the target.
### LadybugDB lock / "database busy"

132
MIGRATION.md
View file

@ -106,6 +106,19 @@ Running `npx gitnexus analyze` writes both `gitnexus.json` and `meta.json`
with identical content. A pre-existing repo that only has `meta.json` gets
`gitnexus.json` bootstrapped from it on the first run.
### Process ids are not stable across this release
`Process` ids are positional (`proc_<idx>_<entry>`), and this release changes
both which execution flows are detected and the order they are selected in:
tracing is depth-first, sibling branches follow source order, and selection
round-robins across terminals so one flow cannot take every slot. A given
`proc_7_handle` before the upgrade is not the same flow afterwards.
Nothing in GitNexus persists or joins on a raw process id across a re-index —
the MCP resource keys by label — so this is one-time index churn rather than a
broken consumer. If you have external tooling that stored a process id, re-
resolve it by label after the next analyze.
### What about rollback?
Downgrading to an older GitNexus version is safe: `meta.json` is always
@ -117,3 +130,122 @@ repo as never analyzed.
The `meta.json` mirror will remain until a future major version. Removal
will be announced in this file and in the changelog before it happens.
## Ambiguous responses report the true match count (PR #2796, issue #2787)
The MCP symbol resolver returns at most 20 candidate rows. Every ambiguous
response used to take its count from that capped window, so a name with 92
matches (`constructor`, in this repo's own index) reported 20. The same PR
pinned the window with an `ORDER BY`, which turned that undercount from
flaky into stable — and a stable wrong number reads as authoritative.
Three consumer-visible changes follow:
- **`impact`'s `totalCandidates` changed meaning.** It was the length of the
capped 20-row window; it is now the true `COUNT(*)` of matching symbols.
Callers using `totalCandidates === candidates.length` as a "not truncated"
proxy will now see the two diverge. This is a bug fix — the old number was
wrong — but it is still a value change on a published field.
- **`totalCandidates` and `candidatesTruncated` are new on other tools.**
They now also appear on `context`, `trace`, the `explain` / `pdg_query`
block-anchor path, and on `rename` (which returns `context`'s ambiguous
payload verbatim). `candidatesTruncated: true` is present only when
`candidates[]` is shorter than `totalCandidates` — absent otherwise, never
`false`.
- **The `message` template gained a `(showing M)` suffix.** It follows the
total — `Found 92 symbols matching 'constructor' (showing 20). …` — and
appears only when the returned window is smaller than the total. `impact`
uses the longer `(showing M of N)` form.
### Do I need to migrate?
**Only if you read `totalCandidates` or parse `message`.** The last two
changes are purely additive — no field was removed or renamed and
`candidates[]` keeps its shape — so PR #888's "no existing field has changed.
No migration required for `context` callers" still holds for `context`.
- Reading `totalCandidates` on `impact`: it is a true total now. Detect a
shortened window with `candidatesTruncated` (or `totalCandidates >
candidates.length`) rather than by comparing it to an array length.
- Parsing `message` for a count: the total is still the first number, but a
`(showing M)` parenthetical may now follow it. Prefer the structured
`totalCandidates` field over the string.
### What happens on re-index?
Nothing — this is an MCP-surface change only. The graph schema, indexer,
and stored data are untouched.
## `schemaVersion` → `schemaFingerprint` (issue #2798)
The field that decides whether an existing index can be reused changed in
`.gitnexus/gitnexus.json` (and in each `branches/<slug>/gitnexus.json`):
`schemaVersion?: number` has been removed and `schemaFingerprint?: string`
added. The new value is a 12-character digest of the graph DDL this build
creates, so it *describes* the schema an index's tables were actually built
from rather than asserting a number about it.
An absent fingerprint is treated as a mismatch, and that is the whole
backward-compatibility story: every index written by an earlier GitNexus
carries no fingerprint, so it is rebuilt exactly once.
### Do I need to migrate?
**No.** There is nothing to run, edit, or pass. The first `analyze` after
upgrading logs one line —
```
index schema changed (built by an unidentified GitNexus build, this build is <fingerprint>); forcing a full re-analyze so the database is recreated from the current schema.
```
— and then performs that full re-analyze itself. The same run stamps the
fingerprint, and every run after it takes the normal incremental path again.
### What happens on re-index?
One automatic full re-analyze, once per index. Nothing else changes; the
resulting graph is what the current build would have produced anyway.
The scope of that one-time cost is worth knowing before you hit it. It is
per **index**, not per machine or per repository — branch-scoped index slots
(#2106) each keep their own `gitnexus.json`, so every slot pays for itself
the first time it is analyzed after the upgrade. On a very large repository
a full re-analyze is substantial, not a blip; plan the first post-upgrade
run accordingly.
### Why a digest instead of a version number?
`schemaVersion` was hand-incremented, and it had to predict something a
number cannot know: whether the DDL an on-disk database was created from
matches this build's. It collided with `main` eight times, twice *exactly* —
and an exact clash was the quiet failure. Two builds stamp the same number
over different DDL, the strict `===` reuse gate reads the index as current,
the `CREATE … TABLE` statements are skipped as "already exists", and edges
whose endpoint pair the live database cannot persist are dropped. A wrong
graph, with no error anywhere.
A derived digest cannot fail that way: two builds agree exactly when their
DDL agrees, so concurrent branches never need renumbering and a mismatch is
always a real mismatch. The retired ladder's per-version rationale (v2
`BasicBlock.callees` through v35's generated relation cross-product) now
lives only in git history:
`git show 561f913a3:gitnexus/src/storage/repo-manager.ts`.
### What about rollback?
Downgrading to an older GitNexus is safe. The older binary looks for
`schemaVersion`, does not find one, treats the index as pre-versioning, and
forces its own full rebuild — the same one-time cost in the other direction,
never a stale or mismatched graph.
### What if I alternate between an old and a new binary?
Every switch forces a rebuild. The end-of-run metadata is written as a fresh
object literal rather than merged over the previous file, so a new build's
write drops `schemaVersion` and an old build's write drops
`schemaFingerprint` — neither field survives the other's run, and each binary
then finds its own gate unsatisfied. This hits anyone running a pinned
`npx gitnexus@<version>` alongside a local build, or an editor hook still on
an older release. It is a cost, not a correctness problem: each run rebuilds
against its own schema, and the graph it serves is correct for the binary
that produced it. Pin one version per index to avoid the churn.

186
README.md
View file

@ -1,6 +1,4 @@
# GitNexus
**⚠️ Important Notice:** GitNexus has NO official cryptocurrency, token, or coin. Any token/coin using the GitNexus name on Pump.fun or any other platform is **not affiliated with, endorsed by, or created by** this project or its maintainers. Do not purchase any cryptocurrency claiming association with GitNexus.
# GitNexus (Akon Labs)
<div align="center">
@ -26,7 +24,7 @@
</a>
</p>
<p><strong>The nervous system for agent context.</strong></p>
<p><strong>The context engine for Enterprise Codebases</strong></p>
<p>
Indexes any codebase into a knowledge graph — every dependency, call chain, cluster, and execution flow —
@ -80,6 +78,32 @@ That's it. `analyze` indexes the codebase, installs agent skills, registers Clau
</details>
### Deploy to Render
Deploy GitNexus in one click:
[![Deploy to Render](https://render.com/images/deploy-to-render-button.svg)](https://render.com/deploy?repo=https://github.com/abhigyanpatwari/GitNexus)
The Blueprint creates two services. `gitnexus-server` runs `gitnexus serve` as a private service: no public URL, reachable only over Render's private network, with a persistent disk for indexes and cloned repos. `gitnexus-web` is the public one. It serves the UI and reverse-proxies `/api/*` to the server, so the browser talks to a single origin.
At the Blueprint's defaults this runs about **$35/month**: $25 for the server's `standard` instance, $7 for the web service's `starter` instance, and $2.50 for the 10 GB disk. See [Render's pricing](https://render.com/pricing) for other plans.
The deploy generates an access token, and the UI asks for it on first use:
1. Open the `gitnexus-web` service in your [Render dashboard](https://dashboard.render.com/).
2. Copy `GITNEXUS_SERVE_AUTH_TOKEN` from its **Environment** tab.
3. Load the site and paste the token into the prompt (or the settings panel).
Every `/api/*` request carries that token as a header, and the proxy answers `401` without it. The browser keeps it in `sessionStorage`, so a new tab asks again. To rotate it, edit the environment variable and redeploy.
The proxy strips `Origin` before forwarding, so the server's CSRF guard does nothing for proxied traffic; it passes `Origin`-less requests through by design. The token is the only control on this deploy, not a second layer behind the guard. Anyone holding it can read every indexed repo. See [SECURITY.md](SECURITY.md#hosted-deploys-on-render).
Indexing is memory-bound. If `gitnexus-server` runs out of memory on a large repo, raise its `plan`, which sets available RAM: `standard` is 2 GB, `pro` is 4 GB. Raise `sizeGB` only if the disk fills with clones and indexes.
### Deploy to RepoCloud
[![Deploy on RepoCloud](https://d16t0pc4846x52.cloudfront.net/deploylobe.svg)](https://repocloud.io/details/gitnexus/)
## Two Ways to Use GitNexus
| | **CLI + MCP** (recommended) | **Web UI** |
@ -157,7 +181,7 @@ flowchart TB
| `group_list` | List configured repository groups |
| `group_sync` | Rebuild a group's Contract Registry and cross-repo links |
> Per-repo tools take an optional `repo` parameter (omit it when only one repo is indexed) and an optional `branch` for indexes pinned with `gitnexus analyze --branch`. Omitting `branch` queries the workspace index, which follows your checked-out working tree — switching branches and re-running `gitnexus analyze` updates it incrementally. `explain` and `pdg_query` need an index built with `gitnexus analyze --pdg`.
> Per-repo read-only tools take an optional `repo` parameter. Omit it when only one repo is indexed, an MCP default is configured, or the GitNexus process cwd is inside a registered path without crossing into an unindexed nested Git checkout; otherwise pass it explicitly. Mutating tools require `repo` when multiple repos are indexed and no MCP default exists. Per-repo tools also take an optional `branch` for indexes pinned with `gitnexus analyze --branch`. Omitting `branch` queries the workspace index, which follows your checked-out working tree — switching branches and re-running `gitnexus analyze` updates it incrementally. `explain` and `pdg_query` need an index built with `gitnexus analyze --pdg`.
### Resources for instant context
@ -362,6 +386,7 @@ Everyday commands:
```bash
gitnexus setup # Configure MCP for detected editors (one-time; -c to select)
gitnexus analyze [path] # Index a repository (or update a stale index)
gitnexus analyze [path] --watch # Watch local files and serialize incremental refreshes
gitnexus mcp # Start MCP server (stdio) — serves all indexed repos
gitnexus serve # Start local HTTP server (multi-repo) for web UI connection
gitnexus eval-server # Start lightweight evaluation HTTP tools (loopback by default)
@ -374,6 +399,28 @@ gitnexus uninstall # Preview removal of GitNexus MCP/skills/hooks
You can also query the graph directly from the terminal — `gitnexus query`, `context`, `impact`, `trace`, `cypher`, `detect-changes`, and `check` mirror the MCP tools of the same names, and `gitnexus doctor` prints runtime platform capabilities.
`gitnexus analyze --watch` requires a Git repository. It runs one initial
analysis, then debounces scanner-admitted working-tree changes for 300 ms by
default and applies serialized incremental refreshes. Events arriving during a
refresh remain queued, and retryable failures retain the same batch with bounded
backoff. Invalid `.gitnexusrc` or ignore-file reloads pause ordinary refreshes
until the control file is fixed. Stop the watcher with Ctrl+C.
Watch mode accepts `--debounce`, `--workers`, `--worker-timeout`,
`--max-file-size`, `--branch`, `--pdg`, `--skip-fts`, `--name`, `--allow-duplicate-name`, and
`--verbose`. Explicit one-shot options such as `--force`, `--repair-fts`,
embedding flags, `--skills`, `--self-commit`, `--index-only`, and `--skip-git`
are rejected. Unsupported defaults from `.gitnexusrc` are ignored with a
warning rather than making an otherwise valid repository unwatchable.
POSIX requests clone-first copy-and-swap publication when the live index has no
orphan sidecars. Windows and sidecar fallback runs update in place: failures
known to occur before writes are retried, while a failure that may have mutated
the live index stops the watcher. Watch mode does not pull remotes. Running MCP
and `serve` processes reopen a newly published index automatically; MCP observes
the replacement on its next tool call, typically within five seconds, so no
restart is required.
<details>
<summary><strong>Authenticated <code>eval-server</code> binding</strong></summary>
@ -391,8 +438,10 @@ The token may be set in the shell, `.env.local`, or `.env` in the working direct
<summary><strong>All <code>analyze</code> flags</strong></summary>
```bash
gitnexus analyze --force # Full rebuild: re-parse + graph rebuild + FTS rebuild
gitnexus analyze --force # Full graph + FTS rebuild (reuses unchanged parser output)
gitnexus analyze --no-parse-cache # Full rebuild that re-parses every source file
gitnexus analyze --repair-fts # Fast path: rebuild/verify only FTS indexes on existing index data
gitnexus analyze --skip-fts # Index graph/embeddings without loading FTS or building keyword indexes
gitnexus analyze --skills # Generate repo-specific skill files from detected communities
gitnexus analyze --skip-embeddings # Skip embedding generation (faster)
gitnexus analyze --embeddings [limit] # Enable embedding generation (slower, better search)
@ -404,10 +453,22 @@ gitnexus analyze --verbose # Log skipped files when parsers are unavailabl
gitnexus analyze --worker-timeout 60 # Increase worker idle timeout for slow parses
gitnexus analyze --workers <n> # Parse worker pool size (>=1; default: cores-1, capped at 16,
# auto-sized to the repo). 0 is rejected — there is no sequential mode.
gitnexus analyze --spring-actuator ./actuator # Enrich with local Spring Boot Actuator JSON snapshots
gitnexus analyze --asyncapi-spec ./docs/asyncapi # Resolve broker addresses from AsyncAPI 3.x documents
gitnexus analyze --wal-checkpoint-threshold 67108864 # LadybugDB WAL auto-checkpoint threshold in bytes
# (default 67108864 = 64 MiB; -1 keeps Ladybug stock ~16 MiB)
```
`--skip-fts` (or `GITNEXUS_SKIP_FTS=1`) disables FTS extension loading and keyword-index construction for this analysis. Graph queries, communities, processes, and existing embeddings remain available. Status and search report "FTS disabled for this index". Remove both the flag and environment setting and run `analyze` again to restore keyword search, even at the same commit. Only the exact environment value `1` enables the opt-out; the flag takes precedence. It cannot be combined with `--repair-fts`. Disabling an existing FTS index may require one graph-store rebuild to avoid unsafe writes through native indexes.
`--spring-actuator` is explicitly opt-in and accepts either a JSON bundle keyed by `mappings`, `beans`, `conditions`, `configprops`, and/or `env`, or a directory containing endpoint-named JSON files. It confirms matching static nodes and adds conservative runtime-only routes, beans, and property keys. The configured input is excluded from source scanning; only normalized repository-relative exclusions are retained for future scans, never absolute paths. Env/configprops values, origins, condition messages, and source names are never persisted or printed. Because snapshots are external runtime state, an enabled run always rebuilds; the first later run without the option rebuilds once to remove runtime evidence. The same path can be set as `springActuator` in `.gitnexusrc`.
`--asyncapi-spec` is explicitly opt-in and accepts a directory of AsyncAPI documents or a single document; the path is resolved against the repository root, so a committed `docs/asyncapi` and an absolute cache written by something else both work. Each `operations[]` entry of an **AsyncAPI 3.x** document can contribute a `Destination` node keyed by broker and address, with `action: send` emitting `PUBLISHES_TO` and `action: receive` emitting `CONSUMES_FROM`, so a document and source code that name one address on one broker land on the same node. Edges start at the document, not at a callable — a document states that the service talks to an address, not which method does — and no address a document names is ever attached to an unresolved source site.
An operation must name a protocol, either through its own `bindings` or through the `servers[].protocol` of the servers its channel resolves to (a channel that lists no `servers` resolves to all of them); operations that name none are refused, as are operations whose two readings name different brokers, and channels that inherit a multi-protocol server set without choosing. HTTP and WebSocket documents are refused for destination minting: there the host rather than the address names the place, and an HTTP endpoint is already modelled as a `Route`. A parameterized address — a channel declaring `parameters`, or an address containing `{` — is refused rather than keyed: two services publishing `{env}.orders` share a pattern, not a queue. AsyncAPI **2.x is refused** under its own counted reason and never mapped, because its `publish`/`subscribe` are inverted relative to 3.x `send`/`receive` and a naive mapping would reverse the async graph while leaving it connected. Every refusal is counted, and a configured path that yields nothing is reported rather than passed over in silence.
Like Actuator snapshots, documents are external to git freshness — replacing one moves no commit and dirties no file — so an enabled run always rebuilds, and the first later run without the option rebuilds once to remove document-derived evidence. There is no glob-based auto-discovery, and the option is unsupported with `--watch`.
If `analyze` reports a worker parse timeout on a large or unusual repository, it keeps running and falls back safely. To give slow worker jobs more time, use `--worker-timeout 60` or set `GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS=60000`. For very large files, `GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES` controls the worker job byte budget.
**Embeddings node limit** — `gitnexus analyze --embeddings` generates semantic search vectors with a default 50,000-node safety cap to protect memory on large repositories:
@ -422,6 +483,46 @@ If embeddings are skipped on a large repository, the indexed graph likely exceed
</details>
<details>
<summary><strong>Keep remote repositories indexed with <code>gitnexus auto-sync</code></strong></summary>
`gitnexus auto-sync` clones or pulls configured repositories, analyzes new commits, and optionally syncs their group. It runs once immediately, then repeats on the configured interval. It runs in the foreground; use your process manager if it must survive a shell session. `gitnexus watch` is reserved and prints this split; it does not start auto-sync or local file watching.
```bash
# 1. Create the config once. It never overwrites an existing file.
gitnexus auto-sync init
# 2. Edit $GITNEXUS_HOME/watch_config.yml, then start it.
gitnexus auto-sync start # `gitnexus auto-sync` is equivalent
gitnexus auto-sync status
gitnexus auto-sync restart # Required after config changes
gitnexus auto-sync stop
gitnexus auto-sync reset # Clear failure state; leaves clones and indexes intact
```
`GITNEXUS_HOME` defaults to `~/.gitnexus`. A minimal configuration:
```yaml
sync_interval_minutes: 10
analyze_timeout: 5m
projects:
- local_path: /absolute/path/to/clones
branches: [main, master]
overwrite_local_changes: false
remote_urls:
- git@github.com:owner/repo.git
```
- `sync_interval_minutes` must be at least `5`; `local_path` must be an absolute path. Clones are stored below it as `host/namespace/repo`.
- Remote URLs must use SSH SCP form and are limited to GitHub, GitLab, or Gitee.
- `branches` are tried in order. The legacy `branch` field is supported, but do not set both.
- Analysis runs in an isolated worker; `analyze_timeout` defaults to, and cannot exceed, half of `sync_interval_minutes`. Timeout and `auto-sync stop` request safe cancellation; a worker in native work exits after reaching a JS-visible safe point. Until then, auto-sync reports `cancelling` or `stopping` and retains ownership so another auto-sync cannot take over, for up to 5 seconds — after that the parent stops waiting and leaves the worker to exit on its own rather than killing it mid-write. This behavior is the same on macOS and Windows. `overwrite_local_changes` defaults to `false`, so a dirty local clone is skipped rather than overwritten; setting it to `true` also deletes untracked files in the clone, while keeping ignored paths.
- Add `group_name` only after creating that group with `gitnexus group create <name>`. Partial clone output is isolated and removed after 14 days.
See the [full auto-sync configuration and runtime reference](gitnexus/README.md#gitnexus-auto-sync) for concurrency, timeouts, failure thresholds, and runtime files.
</details>
<details>
<summary><strong>Repository groups</strong> (multi-repo / monorepo service tracking)</summary>
@ -455,6 +556,7 @@ Commit a `.gitnexusrc` JSON file at the repo root to preconfigure recurring `ana
"skipContextFiles": true, // alias of skipAgentsMd: keep your own AGENTS.md/CLAUDE.md
"skipSkills": true, // don't install standard skill files under .claude/skills/ and .agents/skills/
"embeddings": true, // generate embeddings by default
"springActuator": "./actuator", // optional local runtime snapshot directory or bundle
"workerTimeout": 60,
}
```
@ -469,7 +571,7 @@ Notes:
- The default branch is resolved as: `--default-branch` > `.gitnexusrc` `defaultBranch`/`branch` > auto-detected `origin/HEAD` > `main`.
- `skipContextFiles` / `skipAiContext` are aliases for `skipAgentsMd` — they skip the `AGENTS.md` / `CLAUDE.md` block only. They do **not** imply `skipSkills`. `indexOnly` is the stronger option that skips all file injection.
- Supported keys: `defaultBranch` (`branch`), `skipAgentsMd` (`skipContextFiles`, `skipAiContext`), `skipSkills`, `indexOnly`, `stats`/`noStats`, `embeddings`, `dropEmbeddings`, `name`, `allowDuplicateName`, `maxFileSize`, `workerTimeout`, `walCheckpointThreshold`, `workers`, `embeddingThreads`, `embeddingBatchSize`, `embeddingSubBatchSize`, `embeddingDevice`.
- Supported keys: `defaultBranch` (`branch`), `skipAgentsMd` (`skipContextFiles`, `skipAiContext`), `skipSkills`, `indexOnly`, `stats`/`noStats`, `embeddings`, `dropEmbeddings`, `name`, `allowDuplicateName`, `maxFileSize`, `workerTimeout`, `walCheckpointThreshold`, `workers`, `springActuator`, `embeddingThreads`, `embeddingBatchSize`, `embeddingSubBatchSize`, `embeddingDevice`.
- The file is JSON only. Unknown keys and invalid values fail fast with an actionable error before analysis starts.
</details>
@ -479,35 +581,42 @@ Notes:
Most `analyze` knobs are also CLI flags (`--workers`, `--worker-timeout`, `--max-file-size`, `--verbose`). Use the env-var form when you'd otherwise repeat the same flag every run, or when invoking GitNexus from a long-running host (MCP server, eval-server, CI shell) that already manages its own environment. CLI flags take precedence over env vars; env vars take precedence over built-in defaults.
| Variable | Default | Effect | Tune when… |
| ----------------------------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GITNEXUS_WORKER_POOL_SIZE` | `cores - 1`, capped at 16 | Parse worker pool size (must be ≥ 1). Equivalent to `--workers <n>`. The worker pool is the sole parse path — there is no sequential parser, so `0` is rejected with an actionable error (the pool self-heals via quarantine + respawn). | Constrained containers (cgroup CPU limits) or CI runners with explicit quotas. To narrow down a worker crash set `1` for a single-worker pool — not `0`. |
| `GITNEXUS_PARSE_CHUNK_CONCURRENCY` | `2` | Number of chunks whose file contents may be read into memory in parallel while the pool dispatches the current chunk. Worker dispatch itself stays serial. | Repos large enough to chunk (multi-MB total source) where disk I/O is a measurable fraction of analyze wall-clock. |
| `GITNEXUS_VERBOSE` | unset | When `1`, enables verbose ingestion logs (skipped-file warnings, per-chunk throughput, parse-cache stats). Equivalent to `--verbose`. | Debugging an analyze that "completed" but seems to have missed files; tuning `--workers` / chunk concurrency against observable throughput. |
| `GITNEXUS_AUTH_TOKEN` | unset | Bearer token required when `eval-server` binds beyond loopback. May also be read from `.env.local` or `.env`; shell values take precedence. | Exposing the evaluation HTTP tools to a container, VM, or LAN. |
| `GITNEXUS_PROFILE_DEFERRED` | unset | When `1`, emits `[deferred-profile]` timing/progress logs for the post-chunk deferred resolution band (imports → heritage → buildHeritageMap → legacy call resolution). Implied by `GITNEXUS_VERBOSE`. | Diagnosing analyze stalls in "Resolving calls (all chunks)" on large Java/Kotlin repos (issue #1741) without the full verbose ingestion noise. |
| `GITNEXUS_PROFILE_DEFERRED_SLOW_MS` | `3000` (verbose) / `5000` | Per-file threshold in ms above which `processCallsFromExtracted` emits a `slow file …` log line. Parsed via `Number()`: accepts integers (`5000`), scientific notation (`2.5e3`), decimals (`.5`), and hex (`0x10`). Non-finite or non-positive values fall back to the default. | Hunting a few outlier files dominating the deferred call-resolution stage; lower to surface more, raise to focus only on the worst. |
| `PROF_LBUG_LOAD` | unset | When `1`, emits one `[lbug-load prof]` summary line per `loadGraphToLbug` call breaking the graph-DB persistence wall into stages (`csv-emit` / `copy-nodes` / `copy-rels` / `fallback` / `total`) plus node & edge counts. Zero-cost when unset. | Attributing large-repo analyze wall time across CSV generation vs. LadybugDB `COPY` (issue #2203) — the analyze "emit" timing is the scope-resolution bucket, not this DB-write path. |
| `GITNEXUS_MAX_FILE_SIZE` | `512` (KB) | Walker skip threshold in KB. Hard cap is `32768` (tree-sitter buffer ceiling). Equivalent to `--max-file-size <kb>`. | Indexing repos with intentionally-large source files (generated parsers, vendored bundles) that should still be parsed. |
| `GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS` | `30000` | Worker idle timeout in milliseconds before retry/fallback. Equivalent to `--worker-timeout <seconds>` × 1000. | Slow-parsing files (large minified JS, deeply-nested TS types) that legitimately need more than 30s. |
| `GITNEXUS_WORKER_READY_TIMEOUT_MS` | `5000` | Startup budget in milliseconds for a parse worker to load its grammar bindings and report `{type:'ready'}`. Slots that miss it are treated as startup crashes. | Slow or heavily loaded hosts where a full pool cold-starting concurrently needs more than 5s, and analyze aborts with "did not report ready within 5000ms". |
| `GITNEXUS_FTS_STEMMER` | `porter` | Stemmer used when rebuilding BM25/FTS indexes. Use `none` for CJK-heavy repositories, or a language stemmer such as `german`, `french`, or `spanish` for matching repository comments. Re-run `gitnexus analyze --repair-fts` after changing it. | Keyword search quality is poor for non-English comments or identifiers under English stemming. |
| `GITNEXUS_WAL_CHECKPOINT_THRESHOLD` | `67108864` (64 MiB) | LadybugDB WAL auto-checkpoint threshold in bytes. Equivalent to `--wal-checkpoint-threshold <bytes>`. `-1` keeps LadybugDB's stock threshold (~16 MiB). Larger thresholds reduce checkpoint frequency but increase the WAL size at rotation time — choose a smaller value on disk-constrained environments. | You need a larger or smaller WAL auto-checkpoint threshold for your analyze workload. |
| `GITNEXUS_LBUG_BUFFER_POOL_SIZE` | min(2 GiB, 80% RAM) | LadybugDB buffer-pool ceiling in bytes for every GitNexus database (analyze, MCP server, serve, group bridges). `0` restores LadybugDB's native unbounded default of 80% of system RAM; invalid values warn and fall back to the default (#2557). During `analyze` the pool is right-sized to the graph, scaled on non-4 KiB-page hosts by the page-size granule ratio up to min(2 GiB × pageSize/4 KiB, 80% RAM) (#2631); this env var overrides all of that as an absolute value. | A long-lived `gitnexus mcp` or a big incremental `analyze` uses too much memory, or a huge repo's working set genuinely needs a pool larger than 2 GiB. |
| `GITNEXUS_LBUG_MAX_DB_SIZE` | `17179869184` (16 GiB) | Maximum size in bytes of a single LadybugDB database file — an mmap/disk-address-space ceiling, not a memory limit (it does not constrain the buffer pool). Invalid values silently fall back to the default. | Indexing a genuinely huge monorepo whose on-disk graph index approaches 16 GiB. |
| `GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES` | `8388608` (8 MB) | Per-job byte budget the pool will send to a worker in one `postMessage`. | Very large individual files; mostly diagnostic — bumping past 8 MB risks structured-clone memory pressure. |
| `GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT` | `3` | Max replacement spawns per worker slot before the slot is dropped from the active rotation. Bounds respawn loops on a chronically-crashing slot. | Hosts where a flaky worker should retry more (raise) or fail-fast (lower) before the slot is dropped. |
| `GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS` | `5 × subBatchTimeoutMs` | Total retry wall-time budget per job before quarantining. Combined with `timeoutBackoffFactor`, prevents exponentially-growing retries from stalling for hours. | Slow files that legitimately need long total retry windows; lower to fail-fast on stalls. |
| `GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD` | `max(3, poolSize)` | Per-slot consecutive deaths before the pool's circuit breaker trips. After tripping, every subsequent dispatch rejects until a fresh pool is created. | Hosts where a SIGSEGV-prone native grammar should trip the breaker sooner; CI runners that should fail loudly. |
| `GITNEXUS_WORKER_SHUTDOWN_DRAIN_MS` | `30000` | Max wait at pool shutdown for a retired worker still inside native code. The worker is terminated at its next JS-safe point instead of mid-native-call (which aborts the whole process with `Napi::Error`, #2432); on expiry it is left running, unref'd, and terminated when it surfaces. | Shutdown latency matters more than draining a wedged worker (lower), or a legitimately-slow native grammar needs longer to surface (raise). |
| `GITNEXUS_CPP_CAPTURE_BUDGET_MS` | `20000` | Per-file wall-clock budget for C++ capture extraction. On breach the file keeps the captures accumulated so far and logs a warning — the worker returns to JS instead of stalling in native-heavy loops (#2432). `0` expires immediately. | Pathological generated C++ that still exceeds the budget after the indexed lookups; raise for completeness, lower to fail-fast. |
| `GITNEXUS_CHUNK_BYTE_BUDGET` | `2097152` (2 MB) | Chunk boundary used for cache-key composition and dispatch. Smaller = finer-grained cache hits but more dispatch overhead. | Tuning incremental-analyze cache behavior on monorepos. |
| `GITNEXUS_NO_GITIGNORE` | unset | When set, skips `.gitignore` parsing. `.gitnexusignore` is still honored. | Indexing a repo whose `.gitignore` excludes files you actually want indexed (e.g., generated code committed for cross-repo lookup). |
| `GITNEXUS_SKIP_OPTIONAL_GRAMMARS` | unset | When `=1` strictly, skips the vendored grammar materialize for `tree-sitter-dart`, `tree-sitter-proto`, `tree-sitter-swift`, and `tree-sitter-kotlin` at install time (and the Dart/Proto source builds). Those four won't be parsed; the install still succeeds. | Installing on a host without a C++ toolchain or where the vendored prebuilds don't match; willing to skip Dart/Proto/Swift/Kotlin parsing. |
| `GITNEXUS_MCP_READ_ONLY` | unset | Set to `1` to expose only proven single-repository read tools and resources; `0` disables the policy and any other value fails startup. | The MCP server runs in an environment where graph mutation, raw Cypher, and cross-repository group routing must be unavailable. |
| `GITNEXUS_MCP_ALLOWED_REPOS` | unset | Comma-separated allowlist of canonical indexed repository names or absolute paths. Invalid, ambiguous, or blank entries fail startup. | One MCP process must expose only a bounded subset of the repositories in the global registry. |
| `GITNEXUS_MCP_DEFAULT_REPO` | unset | Canonical indexed repository name or absolute path used when a tool or resource omits its repository. Must belong to the allowlist when one is set. | Several repositories are available but unqualified MCP calls should resolve deterministically. |
| `GITNEXUS_MCP_DEFAULT_MAX_TOKENS` | unset | Default positive-integer response budget for MCP `query`, `context`, and `impact`, estimated at four UTF-8 bytes per token. Explicit `maxTokens` wins. | Long MCP responses consume too much model context and callers cannot reliably add a per-request budget. |
| Variable | Default | Effect | Tune when… |
| ----------------------------------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GITNEXUS_WORKER_POOL_SIZE` | `cores - 1`, capped at 16 | Parse worker pool size (must be ≥ 1). Equivalent to `--workers <n>`. The worker pool is the sole parse path — there is no sequential parser, so `0` is rejected with an actionable error (the pool self-heals via quarantine + respawn). | Constrained containers (cgroup CPU limits) or CI runners with explicit quotas. To narrow down a worker crash set `1` for a single-worker pool — not `0`. |
| `GITNEXUS_PARSE_CHUNK_CONCURRENCY` | `2` | Number of chunks whose file contents may be read into memory in parallel while the pool dispatches the current chunk. Worker dispatch itself stays serial. | Repos large enough to chunk (multi-MB total source) where disk I/O is a measurable fraction of analyze wall-clock. |
| `GITNEXUS_VERBOSE` | unset | When `1`, enables verbose ingestion logs (skipped-file warnings, per-chunk throughput, parse-cache stats). Equivalent to `--verbose`. | Debugging an analyze that "completed" but seems to have missed files; tuning `--workers` / chunk concurrency against observable throughput. |
| `GITNEXUS_EMBEDDING_RETRY_TIMEOUTS` | unset | When truthy (`1`/`true`/`yes`), per-attempt HTTP embedding timeouts (`TimeoutError` on fetch or body read) go through the bounded `GITNEXUS_EMBEDDING_MAX_ATTEMPTS` retry loop instead of failing the job. Any other value leaves it off, so cloud/default timeouts remain terminal. | Local accelerators that drop a device lock when the client disconnects and succeed on the next request (observed with FastFlowLM on Ryzen AI). |
| `GITNEXUS_ANALYZER_IDENTITY_IN_PROCESS_GUARDS` | unset | When truthy (`1`/`true`/`yes`), forces in-process cache-guard validation once a batch has ≥128 requests. In-process mode also auto-selects when `packageRoot`/`buildRoot` fail `W_OK` with `EACCES`/`EROFS`. Otherwise those large batches use a Node subprocess probe. Batches under 128 always stay in-process. | Trusted or read-only installs where two identity subprocess spawns per analyze dominate wall time; leave unset to keep the default isolation path on writable trees. |
| `GITNEXUS_RESOLVE_DEF_GRAPH_ID_MEMO` | on (unset) | Memoizes `resolveDefGraphId` per `nodeLookup` instance (WeakMap). Enabled by default. Set to `0`/`false`/`off`/`no` to disable and recompute on every call (debug / bisect memo bugs). | Suspecting stale graph-id resolution after a lookup rebuild, or comparing memo vs uncached cost on a large index. |
| `GITNEXUS_AUTH_TOKEN` | unset | Bearer token required when `eval-server` binds beyond loopback. May also be read from `.env.local` or `.env`; shell values take precedence. | Exposing the evaluation HTTP tools to a container, VM, or LAN. |
| `GITNEXUS_MCP_AUTH_TOKEN` | unset | Bearer token for the dedicated `gitnexus mcp --http` server, for a **directly reachable** `gitnexus serve` `/api/mcp` route, and for the `docker-server` / web proxy in front of one. A non-loopback dedicated MCP bind requires it; `serve` enables protocol-layer MCP auth when it is set. Behind a proxy, set the **same** value on both services: the proxy spends the edge `GITNEXUS_SERVE_AUTH_TOKEN`, then replaces `Authorization` with this token on `/api/mcp` only. | Dedicated MCP, a `serve` the client can reach directly, or a proxied deploy (Render Blueprint) where the backend runs protocol-layer MCP auth — configure it on the proxy too. |
| `GITNEXUS_PROFILE_DEFERRED` | unset | When `1`, emits `[deferred-profile]` timing/progress logs for the post-chunk deferred resolution band (imports → heritage → buildHeritageMap → legacy call resolution). Implied by `GITNEXUS_VERBOSE`. | Diagnosing analyze stalls in "Resolving calls (all chunks)" on large Java/Kotlin repos (issue #1741) without the full verbose ingestion noise. |
| `GITNEXUS_PROFILE_DEFERRED_SLOW_MS` | `3000` (verbose) / `5000` | Per-file threshold in ms above which `processCallsFromExtracted` emits a `slow file …` log line. Parsed via `Number()`: accepts integers (`5000`), scientific notation (`2.5e3`), decimals (`.5`), and hex (`0x10`). Non-finite or non-positive values fall back to the default. | Hunting a few outlier files dominating the deferred call-resolution stage; lower to surface more, raise to focus only on the worst. |
| `PROF_LBUG_LOAD` | unset | When `1`, emits one `[lbug-load prof]` summary line per `loadGraphToLbug` call breaking the graph-DB persistence wall into stages (`csv-emit` / `copy-nodes` / `copy-rels` / `fallback` / `total`) plus node & edge counts. Zero-cost when unset. | Attributing large-repo analyze wall time across CSV generation vs. LadybugDB `COPY` (issue #2203) — the analyze "emit" timing is the scope-resolution bucket, not this DB-write path. |
| `GITNEXUS_MAX_FILE_SIZE` | `512` (KB) | Walker skip threshold in KB. Hard cap is `32768` (tree-sitter buffer ceiling). Equivalent to `--max-file-size <kb>`. | Indexing repos with intentionally-large source files (generated parsers, vendored bundles) that should still be parsed. |
| `GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS` | `30000` | Worker idle timeout in milliseconds before retry/fallback. Equivalent to `--worker-timeout <seconds>` × 1000. | Slow-parsing files (large minified JS, deeply-nested TS types) that legitimately need more than 30s. |
| `GITNEXUS_WORKER_READY_TIMEOUT_MS` | `5000` | Startup budget in milliseconds for a parse worker to load its grammar bindings and report `{type:'ready'}`. Slots that miss it are treated as startup crashes. | Slow or heavily loaded hosts where a full pool cold-starting concurrently needs more than 5s, and analyze aborts with "did not report ready within 5000ms". |
| `GITNEXUS_FTS_STEMMER` | `porter` | Stemmer used when rebuilding BM25/FTS indexes. Use `none` for CJK-heavy repositories, or a language stemmer such as `german`, `french`, or `spanish` for matching repository comments. Re-run `gitnexus analyze --repair-fts` after changing it. | Keyword search quality is poor for non-English comments or identifiers under English stemming. |
| `GITNEXUS_SKIP_FTS` | unset | When exactly `1`, skips FTS extension loading and keyword index creation during analyze. Equivalent to `--skip-fts`; a later analyze without either option restores FTS. | Graph-only consumers with their own retrieval, or short-lived indexes that do not need keyword search. |
| `GITNEXUS_WAL_CHECKPOINT_THRESHOLD` | `67108864` (64 MiB) | LadybugDB WAL auto-checkpoint threshold in bytes. Equivalent to `--wal-checkpoint-threshold <bytes>`. `-1` keeps LadybugDB's stock threshold (~16 MiB). Larger thresholds reduce checkpoint frequency but increase the WAL size at rotation time — choose a smaller value on disk-constrained environments. | You need a larger or smaller WAL auto-checkpoint threshold for your analyze workload. |
| `GITNEXUS_LBUG_BUFFER_POOL_SIZE` | min(2 GiB, 80% RAM) | LadybugDB buffer-pool ceiling in bytes for every GitNexus database (analyze, MCP server, serve, group bridges). `0` restores LadybugDB's native unbounded default of 80% of system RAM; invalid values warn and fall back to the default (#2557). During `analyze` the pool is right-sized to the graph, scaled on non-4 KiB-page hosts by the page-size granule ratio up to min(2 GiB × pageSize/4 KiB, 80% RAM) (#2631); this env var overrides all of that as an absolute value. | A long-lived `gitnexus mcp` or a big incremental `analyze` uses too much memory, or a huge repo's working set genuinely needs a pool larger than 2 GiB. |
| `GITNEXUS_LBUG_MAX_DB_SIZE` | `17179869184` (16 GiB) | Maximum size in bytes of a single LadybugDB database file — an mmap/disk-address-space ceiling, not a memory limit (it does not constrain the buffer pool). Invalid values silently fall back to the default. | Indexing a genuinely huge monorepo whose on-disk graph index approaches 16 GiB. |
| `GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES` | `8388608` (8 MB) | Per-job byte budget the pool will send to a worker in one `postMessage`. | Very large individual files; mostly diagnostic — bumping past 8 MB risks structured-clone memory pressure. |
| `GITNEXUS_WORKER_MAX_RESPAWNS_PER_SLOT` | `3` | Max replacement spawns per worker slot before the slot is dropped from the active rotation. Bounds respawn loops on a chronically-crashing slot. | Hosts where a flaky worker should retry more (raise) or fail-fast (lower) before the slot is dropped. |
| `GITNEXUS_WORKER_MAX_CUMULATIVE_TIMEOUT_MS` | `5 × subBatchTimeoutMs` | Total retry wall-time budget per job before quarantining. Combined with `timeoutBackoffFactor`, prevents exponentially-growing retries from stalling for hours. | Slow files that legitimately need long total retry windows; lower to fail-fast on stalls. |
| `GITNEXUS_WORKER_CONSECUTIVE_FAILURE_THRESHOLD` | `max(3, poolSize)` | Per-slot consecutive deaths before the pool's circuit breaker trips. After tripping, every subsequent dispatch rejects until a fresh pool is created. | Hosts where a SIGSEGV-prone native grammar should trip the breaker sooner; CI runners that should fail loudly. |
| `GITNEXUS_WORKER_SHUTDOWN_DRAIN_MS` | `30000` | Max wait at pool shutdown for a retired worker still inside native code. The worker is terminated at its next JS-safe point instead of mid-native-call (which aborts the whole process with `Napi::Error`, #2432); on expiry it is left running, unref'd, and terminated when it surfaces. | Shutdown latency matters more than draining a wedged worker (lower), or a legitimately-slow native grammar needs longer to surface (raise). |
| `GITNEXUS_CPP_CAPTURE_BUDGET_MS` | `20000` | Per-file wall-clock budget for C++ capture extraction. On breach the file keeps the captures accumulated so far and logs a warning — the worker returns to JS instead of stalling in native-heavy loops (#2432). `0` expires immediately. | Pathological generated C++ that still exceeds the budget after the indexed lookups; raise for completeness, lower to fail-fast. |
| `GITNEXUS_CHUNK_BYTE_BUDGET` | `2097152` (2 MB) | Per-bucket byte budget for parse-cache packing. Files are grouped by `(language, hash(path) mod 128)`; packs inside a bucket are cut at this limit. Smaller = finer-grained invalidation and more dispatch. Default is always 2 MiB and no longer scales with worker count. | Tuning incremental-analyze cache invalidation on monorepos without changing `--workers`. |
| `GITNEXUS_NO_GITIGNORE` | unset | When set, skips `.gitignore` parsing. `.gitnexusignore` is still honored. | Indexing a repo whose `.gitignore` excludes files you actually want indexed (e.g., generated code committed for cross-repo lookup). |
| `GITNEXUS_SKIP_OPTIONAL_GRAMMARS` | unset | When `=1` strictly, skips the vendored grammar materialize for `tree-sitter-dart`, `tree-sitter-proto`, `tree-sitter-swift`, and `tree-sitter-kotlin` at install time (and the Dart/Proto source builds). Those four won't be parsed; the install still succeeds. | Installing on a host without a C++ toolchain or where the vendored prebuilds don't match; willing to skip Dart/Proto/Swift/Kotlin parsing. |
| `GITNEXUS_MCP_READ_ONLY` | unset | Set to `1` to expose only proven single-repository read tools and resources; `0` disables the policy and any other value fails startup. | The MCP server runs in an environment where graph mutation, raw Cypher, and cross-repository group routing must be unavailable. |
| `GITNEXUS_MCP_ALLOWED_REPOS` | unset | Comma-separated allowlist of canonical indexed repository names or absolute paths. Invalid, ambiguous, or blank entries fail startup. | One MCP process must expose only a bounded subset of the repositories in the global registry. |
| `GITNEXUS_MCP_DEFAULT_REPO` | unset | Canonical indexed repository name or absolute path used when a tool or resource omits its repository. Must belong to the allowlist when one is set. | Several repositories are available but unqualified MCP calls should resolve deterministically. |
| `GITNEXUS_MCP_DEFAULT_MAX_TOKENS` | unset | Default positive-integer response budget for MCP `query`, `context`, and `impact`, estimated at four UTF-8 bytes per token. Explicit `maxTokens` wins. | Long MCP responses consume too much model context and callers cannot reliably add a per-request budget. |
| `GITNEXUS_PUBLIC_ORIGIN` | unset | The single browser origin `serve` is reached through, added to the CORS allowlist and to the write-route origin guard. A wildcard bind (`0.0.0.0`) has no host identity, so without this the server's own UI is refused. **Setting it currently refuses to start:** `serve` has no authentication, requests carrying no `Origin` header already reach `POST /api/analyze` and `DELETE /api/repo`, and this is the setting that would admit browser writes on top of that. Matching rules for when the gate lifts: the hostname must match exactly, and so must the scheme. A value with no scheme (`app.example.com`) means `https`, since a bare host comes from platform service discovery and those terminate TLS; spell out `http://app.example.com` for plain HTTP. An explicit port must match; with no port, any port on that hostname is accepted. Anything that is not one reachable host (a list, `*`, a bare port number, a `:0` port, a trailing dot) warns at startup and allows nothing. | `gitnexus serve` runs behind a reverse proxy or on a wildcard bind, and the UI's index/delete requests return `origin_not_allowed`. |
| `GITNEXUS_TRUST_PROXY` | `loopback, linklocal, uniquelocal` | Express `trust proxy` value — which upstream hops may set `X-Forwarded-*`, and so what the per-IP rate limiter reads as the client IP. Set it to the exact number of proxies you control. Every hop past that is one more entry of the chain the caller gets to write. `false`/`no`/`off` (and a `0` hop count) trust no hop; a proxy list Express can compile (`loopback`, `10.0.0.0/8, 127.0.0.1`) names them instead. `true`/`yes`/`on` is **rejected**: it reads the client-controlled leftmost `X-Forwarded-For` entry, so a spoofed chain earns a fresh rate-limit key per request, and express-rate-limit rejects it too (`ERR_ERL_PERMISSIVE_TRUST_PROXY`). Counts above `16` are rejected as well, as a sanity ceiling rather than a safety boundary. Any invalid value warns and falls back to the default. Bind non-loopback with this unset and `serve` warns: a load balancer outside the private ranges is untrusted, so every request keys to the balancer and the per-IP limit becomes one shared limit. | `serve` sits behind a load balancer outside the private ranges (AWS ALB, Cloudflare, CGNAT), where every request otherwise collapses to the proxy hop and rate limiting goes global. |
</details>
@ -555,7 +664,9 @@ GitNexus builds a complete knowledge graph of your codebase through a multi-phas
| Swift | — | — | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| C | — | — | ✓ | — | ✓ | ✓ | — | ✓ | ✓ |
| C++ | — | — | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| Objective-C | ✓ | — | ✓ | ✓ | ✓ | — | — | — | — |
| Dart | ✓ | — | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| Zig | ✓ | — | ✓ | — | ✓ | ✓ | ✓ | — | ✓ |
**Imports** — cross-file import resolution · **Named Bindings** — `import { X as Y }` / re-export tracking · **Exports** — public/exported symbol detection · **Heritage** — class inheritance, interfaces, mixins · **Type Annotations** — explicit type extraction for receiver resolution · **Constructor Inference** — infer receiver type from constructor calls (`self`/`this` resolution included for all languages) · **Config** — language toolchain config parsing (tsconfig, go.mod, etc.) · **Frameworks** — AST-based framework pattern detection · **Entry Points** — entry point scoring heuristics
@ -565,7 +676,7 @@ GitNexus builds a complete knowledge graph of your codebase through a multi-phas
GitNexus uses a **global registry** so one MCP server can serve multiple indexed repos. No per-project MCP config needed — set it up once and it works everywhere.
Each `gitnexus analyze` stores the index in `.gitnexus/` inside the repo (portable, gitignored) and registers a pointer in `~/.gitnexus/registry.json`. When an AI agent starts, the MCP server reads the registry and can serve any indexed repo. LadybugDB connections are opened lazily on first query and evicted after 5 minutes of inactivity (max 5 concurrent). If only one repo is indexed, the `repo` parameter is optional on all tools — agents don't need to change anything.
Each `gitnexus analyze` stores the index in `.gitnexus/` inside the repo (portable, gitignored) and registers a pointer in `~/.gitnexus/registry.json`. When an AI agent starts, the MCP server reads the registry and can serve any indexed repo. LadybugDB connections are opened lazily on first query and evicted after 5 minutes of inactivity (max 5 concurrent). Read-only tools can omit `repo` when only one repo is indexed, an MCP default is configured, or the GitNexus process cwd is inside a registered path without crossing into an unindexed nested Git checkout. Outside those paths—and for mutating tools with multiple indexed repos and no MCP default—pass `repo` explicitly.
<details>
<summary><strong>Architecture diagram</strong></summary>
@ -743,6 +854,7 @@ gitnexus wiki
# Use a custom model or provider (default model: minimax/minimax-m2.5)
gitnexus wiki --model gpt-4o
gitnexus wiki --base-url https://api.anthropic.com/v1
gitnexus wiki --provider grok # local Grok Build CLI (uses `grok login`, no API key)
# Force full regeneration
gitnexus wiki --force

83
RUNBOOK.md
View file

@ -46,6 +46,17 @@ npx gitnexus status
npx gitnexus list
```
**Scope extraction incomplete:** `npx gitnexus status` reports
`incompleteReasons: ["scope-extraction-failed"]` when one or more files still
lack scope captures after the worker and fallback passes. `impact` and `context`
then report a lower bound with `causes.scopeExtractionFiles` set to the affected
file count. Re-run `npx gitnexus analyze --force`; if the reason remains, inspect
the scope-extraction warnings for the unsupported or malformed source file.
Every pre-existing index remains unverified until it is analyzed once by a
version that writes the completeness receipt. An older index or unreadable completeness record reports
`incompleteReasons: ["scope-extraction-unverified"]`; re-analyze it before treating
empty impact results as exact.
---
## Embeddings
@ -56,7 +67,25 @@ npx gitnexus list
npx gitnexus analyze --embeddings
```
**Important:** If you already had embeddings, **always** pass `--embeddings` on later analyzes, or they can be dropped. See `stats.embeddings` in `.gitnexus/gitnexus.json` (or its legacy `meta.json` mirror; 0 means none).
**Important:** If you already had embeddings, a plain `npx gitnexus analyze` **preserves** them (Non-negotiable 5 in [GUARDRAILS.md](GUARDRAILS.md)) — pass `--embeddings` when you also want vectors generated for new or changed nodes, and `--drop-embeddings` only for a deliberate wipe. See `stats.embeddings` in `.gitnexus/gitnexus.json` (or its legacy `meta.json` mirror; 0 means none) — but that figure isn't always freshly measured: if a run's embedding-count query can't answer, it carries the previous run's number forward instead of writing a wrong zero. For a certified read, check `capabilities.vectorSearch.status` instead — it reads `unavailable` (never a stale count) whenever GitNexus can't vouch for the live vector index.
**Partial embedding index (analyze exits 0, but some nodes never got embedded):** A long run against a flaky embedding endpoint can finish successfully while a bounded number of sub-batches still fail. Affected nodes are dropped to zero rows (never left half-written) and recorded as a pending `embeddingCheckpoint`; `npx gitnexus status` then reports `incompleteReasons: ["embedding-checkpoint-pending"]`. Recovery is a plain:
```bash
npx gitnexus analyze
```
No `--embeddings` flag needed — a retained checkpoint forces embedding generation for the pending nodes regardless of flags, and clears once they succeed. `--drop-embeddings` abandons the pending nodes instead of retrying them; `--force` also discards the checkpoint (with a warning) and rebuilds without resuming it.
**Collapsed graph write (analyze exits NON-ZERO and says INCOMPLETE):** A run can finish writing metadata while only a fraction of the relationships it produced are readable back from the index — edges collapsing to a small share of what was built, or a `CodeRelation` table that never materialized (which reads as a persisted count of zero). Because the metadata IS written and the DB does hold rows, nothing looks broken: queries answer with missing edges rather than an error, which is a confident empty answer rather than a failure. `npx gitnexus status` reports `incompleteReasons: ["graph-write-collapsed"]`, the analyze summary prints `Repository indexed INCOMPLETELY` with the expected and persisted counts, and the CLI exits non-zero so automation is not told an unusable index is fine.
Recovery is a full rebuild:
```bash
npx gitnexus analyze --force
```
If it recurs, the cause is almost always environmental rather than a code defect: check free disk space on the volume holding `.gitnexus/`, make sure no second `analyze` is running against the same repo (both use `.gitnexus/csv` for staging), then run `npx gitnexus doctor`. The check compares in-memory relationship totals (including streamed rows) against what the DB hands back, and is deliberately skipped on incremental runs, where the two counts are not comparable.
**Large repos:** Analyze may skip or limit embedding work when node counts are very high; watch CLI output.
@ -158,6 +187,58 @@ If the error text is `"Only one write transaction at a time is allowed in the sy
---
## File acquisition/reclaim guard recovery
The portable file-lock backend uses `analyze.lock.guard` beside `analyze.lock`.
Every acquisition, including an empty slot, exclusively creates the guard before
inspecting, reclaiming, creating, and verifying the main lock. It removes the
guard before returning a workload handle or waiting on a live workload holder.
Linux abstract-socket and Windows named-pipe locking are unchanged.
A stalled or crashed guard owner blocks file acquisition even when its PID is
dead, its metadata is incomplete, or no main lock exists. **The guard is never
automatically stolen.** Guard contention times out after at most 30 seconds,
capped by the remaining acquisition timeout. This separate ceiling applies even
when `GITNEXUS_INDEX_LOCK_TIMEOUT_MS` is zero or negative (unbounded workload wait).
A guard-cleanup failure rejects acquisition; it must not start unprotected work.
Manual recovery is an outage procedure, not an age/PID-based cleanup:
1. Identify the exact lock directory named in the error. This shared primitive
also protects group sync and registry operations, not just repo analysis.
2. Stop **all relevant writers** and prevent restart: editor/agent hooks, watch
processes, scheduled jobs, services, and any containers sharing the directory.
Account for paused processes and every host with access. If quiescence cannot
be established, do not remove the guard. PID metadata is diagnostic only.
3. While restart remains disabled, inspect and preserve the guard/main records
for diagnosis, then remove only that directory's orphan `analyze.lock.guard`
and, if present, its orphan `analyze.lock`. Do not remove databases or sidecars
as part of lock recovery. Do not use a recursive or wildcard cleanup.
4. Ensure all participating writers use the guarded version and the same locking
backend/domain, then restart in a controlled fashion.
**Upgrade requires a coordinated stop/upgrade/restart.** Concurrent older
versions ignore the guard and can still displace live locks; mixed-version
mutual exclusion is not guaranteed. The file protocol assumes reliable atomic
local-filesystem `O_EXCL` creation and cooperating processes. Network/distributed
filesystems, external file replacement, and uncoordinated manual deletion are not
covered. A process crash while holding the short-lived guard trades automatic
recovery for fail-closed safety. Denied file creation returns a non-owning
`lockFree` handle only when neither workload lock nor acquisition guard exists;
unreadable paths fail closed. No staging sweep runs without ownership. Analysis,
registry transactions, group synchronization, and embeddings sync refuse
`lockFree` handles, including an otherwise up-to-date analysis on a file-backend
read-only mount.
The socket backend can still acquire ownership on a read-only index mount.
Heterogeneous permissions are not proof that another process cannot write.
If guard cleanup fails after this attempt created its workload record, acquisition
is refused and token-exact workload cleanup is attempted before returning the
error. Failed or unverifiable cleanup must be diagnosed under the same quiesced
recovery procedure above; never delete a possibly active successor's record.
---
## Where to dig deeper
- Architecture overview: [ARCHITECTURE.md](ARCHITECTURE.md)

18
SECURITY.md
View file

@ -51,6 +51,24 @@ If you fork GitNexus or self-host it, we recommend enabling the following in you
- **Secret scanning** and **Push protection** — blocks pushes that introduce known secret patterns. Defense-in-depth on top of the in-CI Gitleaks scan documented below.
- **Code scanning** — surfaces SARIF results from CodeQL, Trivy, Scorecard, and zizmor in one place.
### Hosted Deploys on Render
The `render.yaml` Blueprint (see the README's **Deploy to Render**) puts `gitnexus serve` on a **private service** with no public URL, and a public web service in front of it that reverse-proxies `/api/*`. What that does and does not protect:
- **The web service is public and its URL is discoverable.** `onrender.com` hostnames appear in certificate transparency logs. Treat the URL as known rather than secret.
- **The generated `GITNEXUS_SERVE_AUTH_TOKEN` is the only access control.** The proxy rejects any `/api/*` request without it with a `401` before forwarding. Rotate it by editing the environment variable on the `gitnexus-web` service and redeploying.
- **The CSRF guard is inert on this path.** The proxy strips `Origin` before forwarding, so the server's write-origin guard does nothing for proxied traffic — it passes `Origin`-less requests through by design. The token is not a second layer behind the guard.
- **Anyone holding the token can read every indexed repo's source.** These routes carry no origin guard, and the first three carry no rate limiter either: `GET /api/repos`, `GET /api/graph`, `POST /api/query`, `GET /api/file`, `GET /api/grep`. Whoever has the token can also index and delete repositories.
- **`POST /api/mcp` rides the same path.** When `GITNEXUS_MCP_AUTH_TOKEN` is set on the backend, `serve` protects `/api/mcp` with the same constant-time Bearer check as the dedicated HTTP MCP server, before parsing the request body. The Render Blueprint does not set a backend MCP token by default. To enable it behind the proxy, set the **same** `GITNEXUS_MCP_AUTH_TOKEN` on both the `gitnexus-web` proxy and the `gitnexus-server` backend: the proxy consumes the edge `GITNEXUS_SERVE_AUTH_TOKEN`, then replaces `Authorization` with the MCP token on `/api/mcp` (and its subpaths) only — the edge credential is never forwarded, and other `/api/*` routes stay stripped. Configuring it on the backend alone makes every proxied MCP request `401`.
- **A directly reachable `serve` still needs an explicit control.** If neither `GITNEXUS_MCP_AUTH_TOKEN` nor an authenticated edge/private-network boundary is present, `/api/mcp` is unauthenticated. Do not bind that topology to a LAN or public interface: MCP readers can access indexed source and graph context.
- **Rate limits bound cost, not access.** They cap what a token holder can spend; they do not decide who gets in.
Do not hand the URL out as a public demo. A token holder has read access to everything the deploy has indexed.
### `/api/grep` regex semantics and residual ReDoS exposure
`GET /api/grep` executes caller-supplied patterns as real regular expressions (with an optional path-substring `fileFilter` and `caseSensitive` flag) to honor the web chat's grep tool contract; `literal=1` restores the older escaped-substring mode. Mitigations: a 200-character pattern cap, line-by-line matching, a max-200 result cap, and a 5-second wall-clock budget. Matching runs in a `worker_threads` worker so a catastrophic pattern (e.g. `(a+)+$`) can be killed with `terminate()` when the budget expires — the parent event loop (other routes + SSE) stays responsive. A timed-out scan returns partial results with `timedOut: true`; the web grep tool surfaces that flag so an agent does not treat a cut-off scan as exhaustive. CodeQL still flags constructing a `RegExp` from the query string; that is the advertised contract, not accidental injection. Hosted deploys continue to gate the route behind the edge token.
## Automated Scans Running in CI
This repository runs the following scans automatically. Findings appear under the repository's **Security → Code scanning** tab.

460
docker-server.mjs
View file

@ -1,5 +1,8 @@
import { timingSafeEqual } from 'node:crypto';
import { writeSync } from 'node:fs';
import { open } from 'node:fs/promises';
import { createServer } from 'node:http';
import { createServer, request as httpRequest } from 'node:http';
import { request as httpsRequest } from 'node:https';
import { extname, isAbsolute, normalize, relative, resolve, sep } from 'node:path';
const host = '0.0.0.0';
@ -22,18 +25,444 @@ function jsonForScriptTag(obj) {
.replace(/&/g, '\\u0026');
}
const rawBackendUrl = process.env.GITNEXUS_BACKEND_URL ?? null;
if (rawBackendUrl && !isValidUrl(rawBackendUrl)) {
const safeRaw = rawBackendUrl.replace(/[\x00-\x1f\x7f]/g, ' ').slice(0, 200);
console.warn(
`[gitnexus-web] GITNEXUS_BACKEND_URL "${safeRaw}" is not a valid http/https URL -- ignoring.`,
// Warnings echo operator input back, so strip control characters (log forging)
// and cap the length first.
function sanitizeForLog(value) {
return (
String(value)
// The line-break strip is redundant with the range below, but CodeQL's
// js/log-injection recognizes only this shape as a sanitizer: a global
// replace of a literal \n with the empty string.
.replace(/\n/g, '')
.replace(/\r/g, '')
.replace(/[\x00-\x1f\x7f]/g, ' ')
.slice(0, 200)
);
}
const backendUrl = rawBackendUrl && isValidUrl(rawBackendUrl) ? rawBackendUrl : null;
// console.error is asynchronous when stderr is a pipe, so pairing it with
// process.exit can drop the one message explaining the refusal. writeSync isn't.
function exitWithRefusal(message) {
writeSync(2, `${message}\n`);
process.exit(1);
}
// `value` if it's a usable http/https URL, else null + a warning naming `label`.
// `rawForLog` lets a caller that normalized first echo back the operator's input.
function validHttpUrl(label, value, rawForLog = value) {
if (!value) return null;
if (isValidUrl(value)) return value;
const safeRaw = sanitizeForLog(rawForLog);
console.warn(`[gitnexus-web] ${label} "${safeRaw}" is not a valid http/https URL -- ignoring.`);
return null;
}
// Numeric env var. Every consumer below reads <= 0 as "disabled", so obeying a
// typo like -1 would switch a timeout off silently. Warn and use the default.
function numberFromEnv(label, fallback, min = 0) {
const raw = process.env[label];
if (raw === undefined || raw === '') return fallback;
const n = Number(raw);
if (!Number.isFinite(n)) {
console.warn(
`[gitnexus-web] ${label} "${sanitizeForLog(raw)}" is not a number -- using ${fallback}.`,
);
return fallback;
}
if (n < min) {
console.warn(
`[gitnexus-web] ${label} "${sanitizeForLog(raw)}" is below the minimum ${min} -- using ${fallback}.`,
);
return fallback;
}
return n;
}
// Falls back to RENDER_EXTERNAL_URL so a Render web service hands the browser
// its own public origin — same-origin API calls via the proxy below, no config.
const backendUrlVar =
process.env.GITNEXUS_BACKEND_URL !== undefined ? 'GITNEXUS_BACKEND_URL' : 'RENDER_EXTERNAL_URL';
const rawBackendUrl = process.env.GITNEXUS_BACKEND_URL ?? process.env.RENDER_EXTERNAL_URL ?? null;
const backendUrl = validHttpUrl(backendUrlVar, rawBackendUrl);
const configScript = backendUrl
? `<script>window.__GITNEXUS_CONFIG__=${jsonForScriptTag({ backendUrl })};</script>`
: '';
// Optional same-origin reverse proxy for the API server. On a split deploy
// (public web service, private API) the browser must reach the API without a
// cross-origin request, since its CORS allowlist and write-route guard only
// admit same-host origins. So the browser targets THIS origin and we forward
// /api/* to GITNEXUS_UPSTREAM_URL. Unset → no proxy (docker-compose default).
// A scheme-less host:port — what Render's `fromService: hostport` yields —
// gets http:// prepended.
const rawUpstream = process.env.GITNEXUS_UPSTREAM_URL;
const rawUpstreamUrl = rawUpstream
? /^https?:\/\//.test(rawUpstream)
? rawUpstream
: `http://${rawUpstream}`
: null;
const upstreamBase = validHttpUrl('GITNEXUS_UPSTREAM_URL', rawUpstreamUrl, rawUpstream);
// The one origin this proxy will ever connect to (see proxyToUpstream).
const upstreamOrigin = upstreamBase ? new URL(upstreamBase).origin : null;
// The Bearer token every /api/* request must carry. The private upstream has no
// auth of its own and loses its Origin guard one hop below (see
// proxyToUpstream), so the gate belongs here. The browser holds it — never
// inject it next to `backendUrl`. Blank-is-absent follows resolveAuthToken
// (gitnexus/src/mcp/http-transport.ts).
const authToken = process.env.GITNEXUS_SERVE_AUTH_TOKEN?.trim() || null;
// The protocol-layer credential the upstream `serve` expects on /api/mcp when it
// runs with MCP Bearer auth enabled. Set it to the SAME value on both services:
// the edge token is spent here and replaced with this one for MCP requests only
// (see proxyToUpstream). Unset — the default — means no injection, so a backend
// without MCP auth is unaffected. Blank-is-absent follows resolveAuthToken
// (gitnexus/src/mcp/http-transport.ts). Never logged.
const mcpAuthToken = process.env.GITNEXUS_MCP_AUTH_TOKEN?.trim() || null;
// Mirrors the non-loopback refusal in http-transport.ts (startMcpHttpServer),
// relocated because the trust boundary is here: an unguarded `serve` behind a
// private service is legitimate, an unguarded public proxy is not.
if (upstreamBase && !authToken) {
exitWithRefusal(
'[gitnexus-web] Refusing to start: GITNEXUS_UPSTREAM_URL is set without ' +
'GITNEXUS_SERVE_AUTH_TOKEN. The proxy would expose every indexed repo — ' +
'index, read source, and delete — to anyone with this URL. Set a token, ' +
'or unset GITNEXUS_UPSTREAM_URL to serve static assets only.',
);
}
// Rejected requests never reach the upstream limiter, so guesses are free. A
// throttle would add per-address state to a stateless proxy and a lockout an
// attacker can aim at a real user; a length floor makes guessing hopeless and
// only ever rejects a hand-picked token.
const MIN_AUTH_TOKEN_LENGTH = 32;
if (authToken && authToken.length < MIN_AUTH_TOKEN_LENGTH) {
exitWithRefusal(
`[gitnexus-web] Refusing to start: GITNEXUS_SERVE_AUTH_TOKEN is shorter than ` +
`${MIN_AUTH_TOKEN_LENGTH} characters. It is the only thing standing between the ` +
'public internet and every indexed repo, and a failed guess is not rate-limited. ' +
'Use a generated random value.',
);
}
// Whether an inbound X-Forwarded-For may be believed (see clientAddressFor).
// Default off, so a wrong deployment fails toward over-restriction rather than
// toward an address the caller picks. `true` is rejected as it is server-side
// (resolveTrustProxy, which also takes hop counts and so rejects `yes`/`on`
// too): it reads as "trust the whole chain".
function resolveTrustXff(raw) {
const value = raw?.trim();
if (!value) return false;
if (/^(1|yes|on)$/i.test(value)) return true;
if (/^(0|no|off|false)$/i.test(value)) return false;
console.warn(
`[gitnexus-web] GITNEXUS_PROXY_TRUST_XFF "${sanitizeForLog(value)}" is not a recognized ` +
'boolean -- ignoring the inbound X-Forwarded-For chain. Set 1 only when a load balancer ' +
'that appends the real peer sits in front of this service.',
);
return false;
}
const trustInboundXff = resolveTrustXff(process.env.GITNEXUS_PROXY_TRUST_XFF);
// Idle timeout for a proxied request → 504. Socket activity (SSE heartbeats)
// resets it, so long-lived streams are unaffected. 0 disables.
const proxyTimeoutMs = numberFromEnv('GITNEXUS_PROXY_TIMEOUT_MS', 120000);
// nginx's client_body_timeout equivalent: how long to wait for a replayable
// client body before 400. Defaults to the idle timeout; 0 disables.
const proxyClientBodyTimeoutMs = numberFromEnv(
'GITNEXUS_PROXY_CLIENT_BODY_TIMEOUT_MS',
proxyTimeoutMs,
);
// Bounded connection-retry, to ride out the few-second window where a
// single-instance upstream (private server + disk ⇒ no zero-downtime deploy)
// is restarting. Attempts of 1 disables it, and body buffering with it.
const proxyRetryAttempts = numberFromEnv('GITNEXUS_PROXY_RETRY_ATTEMPTS', 3, 1);
const proxyRetryEnabled = proxyRetryAttempts > 1;
const proxyRetryMaxBodyBytes = numberFromEnv('GITNEXUS_PROXY_RETRY_MAX_BODY_BYTES', 256 * 1024);
// Never connected ⇒ the upstream got nothing ⇒ safe to replay any method.
const preConnectRetryCodes = new Set(['ECONNREFUSED', 'ENOTFOUND', 'EAI_AGAIN']);
// Failed after connecting ⇒ the upstream may already be working on it, so
// replay only idempotent methods (RFC 7231 §4.2.2) to avoid double-execution.
const postConnectRetryCodes = new Set(['ECONNRESET', 'ETIMEDOUT']);
const idempotentMethods = new Set(['GET', 'HEAD', 'OPTIONS', 'PUT', 'DELETE', 'TRACE']);
// Buffer a request body, capped. Resolves null on overflow, client error, or
// timeout — one "unreadable body" contract, which the caller maps to 400.
// Listeners detach once settled so a later pipe of the same request is clean.
function readBodyCapped(req, cap, timeoutMs) {
return new Promise((resolvePromise) => {
const chunks = [];
let total = 0;
let settled = false;
let timer = null;
const cleanup = () => {
if (timer) clearTimeout(timer);
req.removeListener('data', onData);
req.removeListener('end', onEnd);
req.removeListener('error', onError);
};
const finish = (value) => {
if (settled) return;
settled = true;
cleanup();
resolvePromise(value);
};
const onData = (chunk) => {
total += chunk.length;
if (total > cap) {
finish(null);
return;
}
chunks.push(chunk);
};
const onEnd = () => finish(Buffer.concat(chunks));
const onError = () => finish(null);
req.on('data', onData);
req.on('end', onEnd);
req.on('error', onError);
// Hard cap regardless of idle activity; Node's requestTimeout is the outer
// backstop.
if (timeoutMs > 0) {
timer = setTimeout(() => {
console.warn(`[gitnexus-web] client body read timed out after ${timeoutMs}ms`);
finish(null);
}, timeoutMs);
}
});
}
// Constant-time Bearer check, mirroring createAuthMiddleware in
// gitnexus/src/mcp/http-transport.ts — dummy comparison included, so an absent
// or wrong-length header costs the same and the timing can't leak the length.
// Duplicated because this file is plain ESM and can't import from gitnexus/src.
function authorized(req) {
if (!authToken) return true; // static-only: no proxy, nothing to gate
const header = req.headers['authorization'];
const expected = Buffer.from(`Bearer ${authToken}`);
if (typeof header !== 'string') {
timingSafeEqual(Buffer.alloc(expected.length), expected);
return false;
}
const provided = Buffer.from(header);
if (provided.length !== expected.length) {
timingSafeEqual(Buffer.alloc(expected.length), expected);
return false;
}
return timingSafeEqual(provided, expected);
}
// WWW-Authenticate names the scheme; the stable `code` is what the web client
// dispatches on, not message text. The body must not distinguish "no token
// configured" from "wrong token". `Connection: close` because we answer before
// reading the body, which Node would otherwise drain (as with the 400 below).
function sendUnauthorized(res) {
const body = JSON.stringify({ error: 'unauthorized', code: 'unauthorized' });
res.writeHead(401, {
'Content-Type': 'application/json; charset=utf-8',
'Content-Length': Buffer.byteLength(body),
'WWW-Authenticate': 'Bearer',
Connection: 'close',
});
res.end(body);
}
// Fail a proxied request. Once headers are sent the body is partially written
// and can't be replaced, so the socket is all we can destroy.
function failGateway(res, status, message) {
if (res.headersSent) {
res.destroy();
} else {
res.writeHead(status, { 'Content-Type': 'text/plain; charset=utf-8' });
res.end(message);
}
}
// Hop-by-hop headers (RFC 7230 §6.1) describe one connection, so a proxy must
// not forward them in either direction; Node sets its own per hop.
const hopByHopHeaders = [
'connection',
'keep-alive',
'proxy-authenticate',
'proxy-authorization',
'te',
'trailer',
'transfer-encoding',
'upgrade',
];
function stripHopByHopHeaders(headers) {
// §6.1 also lets `Connection` name additional single-hop headers, which the
// fixed list below can't cover. Node lowercases header keys on both the
// server and client side, so a lowercased name indexes `headers` directly.
for (const listed of String(headers.connection ?? '').split(',')) {
const name = listed.trim().toLowerCase();
if (name) delete headers[name];
}
for (const name of hopByHopHeaders) delete headers[name];
return headers;
}
// The client address this proxy vouches for upstream. The API keys its rate
// limiter off req.ip, so forwarding a client-supplied X-Forwarded-For would let
// anyone rotate a fake address per request. Which entry is real depends on a
// deployment fact this process can't observe (is anything in front appending the
// peer?), so the operator asserts it via GITNEXUS_PROXY_TRUST_XFF; until then we
// forward the socket peer.
function clientAddressFor(req) {
if (!trustInboundXff) return req.socket.remoteAddress || null;
const forwarded = String(req.headers['x-forwarded-for'] ?? '')
.split(',')
.map((part) => part.trim())
.filter(Boolean)
.pop();
return forwarded || req.socket.remoteAddress || null;
}
// Forward an `/api/*` request upstream, streaming both bodies (SSE / chunked
// graph streams) untouched. Retries connect failures when the body is replayable.
async function proxyToUpstream(req, res) {
let upstream;
try {
upstream = new URL(req.url, upstreamBase);
} catch {
res.writeHead(400);
res.end('Bad request');
return;
}
// The `/api/` route guard keeps req.url host-relative, so resolution can't
// leave upstreamBase. Asserting it here means the SSRF boundary doesn't rest
// on that two-step argument: one legitimate destination, checked locally.
if (upstream.origin !== upstreamOrigin) {
console.error(`[gitnexus-web] refusing to proxy off-origin target ${upstream.origin}`);
res.writeHead(400);
res.end('Bad request');
return;
}
const isHttps = upstream.protocol === 'https:';
const requestFn = isHttps ? httpsRequest : httpRequest;
const headers = stripHopByHopHeaders({ ...req.headers });
// Terminate the browser origin: the API admits Origin-less requests as
// trusted server-to-server calls. Nothing is lost — the browser only ever
// talks to this same-origin web service.
delete headers.origin;
delete headers.referer;
// The edge token is spent here and must never be forwarded: copying
// Authorization would put a live credential into another service's logs. So
// drop it unconditionally first, then — for the MCP route alone, and only
// when a backend token is configured — replace it with that separate
// protocol credential. Unset GITNEXUS_MCP_AUTH_TOKEN (the default) leaves
// every request stripped, as before. The scope is the normalized pathname,
// so a query string can't widen it and /api/mcpfoo doesn't qualify.
delete headers.authorization;
const upstreamPath = upstream.pathname;
const isMcpRoute = upstreamPath === '/api/mcp' || upstreamPath.startsWith('/api/mcp/');
if (isMcpRoute && mcpAuthToken) headers.authorization = `Bearer ${mcpAuthToken}`;
headers.host = upstream.host;
// Replace, never forward, the inbound chain (see clientAddressFor).
const clientAddress = clientAddressFor(req);
if (clientAddress) headers['x-forwarded-for'] = clientAddress;
else delete headers['x-forwarded-for'];
// A retry replays the body, so buffer it up front — but only when small and
// of known length. Larger/unknown bodies (multipart uploads) stream once with
// no retry; an upload is never buffered.
const method = (req.method || 'GET').toUpperCase();
const isIdempotentMethod = idempotentMethods.has(method);
// A request has a body iff it frames one (RFC 7230 §3.3.3). Keying off the
// method sends a bodyless DELETE down the stream-once path and gives up a
// replay that costs nothing.
const hasBody =
req.headers['content-length'] !== undefined || req.headers['transfer-encoding'] !== undefined;
const len = Number(req.headers['content-length']);
const bufferable =
proxyRetryEnabled && Number.isFinite(len) && len >= 0 && len <= proxyRetryMaxBodyBytes;
let bodyBuf = hasBody ? null : Buffer.alloc(0);
if (hasBody && bufferable) {
bodyBuf = await readBodyCapped(req, proxyRetryMaxBodyBytes, proxyClientBodyTimeoutMs);
if (bodyBuf === null) {
// Overflow, client error, and timeout all collapse to 400 (not 413/408).
// `Connection: close` lets Node drop the socket after the 400 flushes,
// rather than half-open draining a stalled upload until requestTimeout.
if (!res.headersSent) {
res.writeHead(400, {
'Content-Type': 'text/plain; charset=utf-8',
Connection: 'close',
});
res.end('Bad request');
}
return;
}
}
// bodyBuf === null means "stream the live request once, no retry".
const retryEligible = bodyBuf !== null;
const attempt = (n) => {
let timedOut = false;
const upstreamReq = requestFn(
{
protocol: upstream.protocol,
hostname: upstream.hostname,
port: upstream.port || (isHttps ? 443 : 80),
method: req.method,
path: upstream.pathname + upstream.search,
headers,
},
(upstreamRes) => {
// Pipe rather than buffer, so SSE / chunked streams reach the browser
// incrementally. Node re-derives Transfer-Encoding for this hop.
const responseHeaders = stripHopByHopHeaders({ ...upstreamRes.headers });
res.writeHead(upstreamRes.statusCode || 502, responseHeaders);
upstreamRes.on('error', () => res.destroy());
upstreamRes.pipe(res);
},
);
upstreamReq.on('error', (err) => {
if (timedOut) return; // 504 already sent by the timeout handler below
// Only before any response byte reaches the browser — once headers are
// sent the body is partially written and can't be replayed.
const retryableError =
preConnectRetryCodes.has(err.code) ||
(isIdempotentMethod && postConnectRetryCodes.has(err.code));
if (retryEligible && !res.headersSent && n < proxyRetryAttempts && retryableError) {
const delay = 250 * 2 ** (n - 1); // 250ms, 500ms, ...
console.warn(
`[gitnexus-web] upstream ${sanitizeForLog(err.code)}; retry ${n}/${proxyRetryAttempts - 1} in ${delay}ms`,
);
setTimeout(() => {
// The client may have aborted during the backoff window; don't fire a
// fresh upstream request nobody is waiting for anymore.
if (res.writableEnded || res.destroyed) return;
attempt(n + 1);
}, delay);
return;
}
console.error('[gitnexus-web] upstream proxy error:', sanitizeForLog(err.message));
failGateway(res, 502, 'Bad gateway');
});
if (proxyTimeoutMs > 0) {
upstreamReq.setTimeout(proxyTimeoutMs, () => {
timedOut = true;
console.error(`[gitnexus-web] upstream proxy timeout after ${proxyTimeoutMs}ms`);
failGateway(res, 504, 'Gateway timeout');
upstreamReq.destroy();
});
}
if (bodyBuf !== null) {
// Replayable body already buffered; write it fresh on each attempt.
if (bodyBuf.length) upstreamReq.write(bodyBuf);
upstreamReq.end();
} else {
// Non-retryable: stream the live request once.
req.on('error', () => upstreamReq.destroy());
req.pipe(upstreamReq);
}
};
attempt(1);
}
const contentTypes = {
'.css': 'text/css; charset=utf-8',
'.html': 'text/html; charset=utf-8',
@ -68,6 +497,23 @@ const spaFallback = resolve(root, 'index.html');
const server = createServer(async (req, res) => {
const urlPath = req.url?.split('?')[0] || '/';
// Same-origin API proxy; everything else falls through to the SPA below.
if (upstreamBase && (urlPath === '/api' || urlPath.startsWith('/api/'))) {
// Before body buffering and the upstream socket, so an unauthenticated
// request costs nothing upstream. Static assets are never gated: the UI has
// to load in order to prompt for the token.
if (!authorized(req)) {
sendUnauthorized(res);
return;
}
// Fire-and-forget, so guard the boundary against unhandledRejection.
proxyToUpstream(req, res).catch((err) => {
console.error('[gitnexus-web] proxy handler crashed:', sanitizeForLog(err?.message ?? err));
failGateway(res, 502, 'Bad gateway');
});
return;
}
let decoded;
try {
decoded = decodeURIComponent(urlPath);

901
docker-server.test.mjs
View file

@ -1,4 +1,5 @@
import { mkdir, mkdtemp, rm, unlink, writeFile } from 'node:fs/promises';
import { connect } from 'node:net';
import http, { createServer } from 'node:http';
import { tmpdir } from 'node:os';
import { dirname, join } from 'node:path';
@ -263,3 +264,903 @@ it('does not inject config into static assets', async () => {
assert.equal(res.body, 'body{}');
});
});
// -- API reverse proxy (GITNEXUS_UPSTREAM_URL) -----------------------------
// Every proxy fixture below runs the server with this token: the proxy refuses
// to start without one, and refuses one under 32 characters.
const TEST_AUTH_TOKEN = 'proxy-test-token-0123456789abcdefghij';
const TEST_BEARER = `Bearer ${TEST_AUTH_TOKEN}`;
// The protocol token the upstream expects on /api/mcp. Deliberately unlike the
// edge token, so "injected the backend credential" and "forwarded the edge one"
// can never both satisfy an assertion.
const TEST_MCP_TOKEN = 'backend-mcp-token-0123456789abcdefghij';
const TEST_MCP_BEARER = `Bearer ${TEST_MCP_TOKEN}`;
// rawRequest never sends credentials; apiRequest does. In a file whose subject
// is who gets let through, no test should pass because a helper quietly
// authenticated for it.
function rawRequest(port, path, { method = 'GET', headers = {}, body } = {}) {
// Send an explicit Content-Length like a browser fetch() does — the proxy
// only buffers (and so only retries) bodies of known length.
const outHeaders = { ...headers };
if (
body !== undefined &&
!Object.keys(outHeaders).some((h) => h.toLowerCase() === 'content-length')
) {
outHeaders['content-length'] = String(Buffer.byteLength(body));
}
return new Promise((resolve, reject) => {
const req = http.request(
{ host: '127.0.0.1', port, path, method, headers: outHeaders },
(res) => {
let respBody = '';
res.setEncoding('utf8');
res.on('data', (chunk) => {
respBody += chunk;
});
res.on('end', () =>
resolve({ status: res.statusCode, headers: res.headers, body: respBody }),
);
},
);
req.on('error', reject);
if (body !== undefined) req.write(body);
req.end();
});
}
// An authenticated /api/* call. An explicit `authorization` header wins, so the
// auth tests can send a wrong one.
function apiRequest(port, path, { headers = {}, ...rest } = {}) {
const hasAuth = Object.keys(headers).some((h) => h.toLowerCase() === 'authorization');
return rawRequest(port, path, {
...rest,
headers: hasAuth ? headers : { ...headers, authorization: TEST_BEARER },
});
}
const respondOk = (_req, res) => {
res.writeHead(200, { 'Content-Type': 'application/json; charset=utf-8' });
res.end('{"ok":true}');
};
// Every proxy test needs the same four parts: a dist/ to serve, a fake upstream,
// a docker-server pointed at it, and teardown that leaks neither a process nor a
// temp dir. They differ only in how the upstream misbehaves.
//
// upstream request handler, replaceable mid-test via `ctx.handler`;
// null points the proxy at a port nothing ever listens on
// listenAfterMs bind the upstream this late, so the first attempt(s) hit
// ECONNREFUSED (a single-instance restart window)
// schemeless drop http:// from GITNEXUS_UPSTREAM_URL, the way Render's
// `fromService: { property: hostport }` yields it
// env extra environment for docker-server.mjs
//
// `ctx` collects what the upstream saw (calls, last request, last body) plus the
// proxy's stderr, so assertions read off one object.
async function withProxy(
{ upstream = respondOk, listenAfterMs = 0, schemeless = false, env = {} } = {},
fn,
) {
const dir = await mkdtemp(join(tmpdir(), 'gitnexus-proxy-'));
await mkdir(join(dir, 'dist'), { recursive: true });
await writeFile(join(dir, 'dist', 'index.html'), '<html><body>spa</body></html>');
const ctx = { calls: 0, received: null, body: null, stderr: '', handler: upstream };
// Read the forwarded request to completion before handing it to the handler,
// so no test has to repeat that plumbing to assert on headers or body.
const server = upstream
? createServer((req, res) => {
let body = '';
req.setEncoding('utf8');
req.on('data', (chunk) => {
body += chunk;
});
req.on('end', () => {
ctx.calls += 1;
ctx.body = body;
ctx.received = { method: req.method, url: req.url, headers: req.headers, body };
ctx.handler(req, res);
});
})
: null;
// A late (or never) bind needs its port reserved up front; otherwise let the
// OS assign one at listen time.
const upstreamPort =
server && listenAfterMs === 0
? await new Promise((r) => server.listen(0, '127.0.0.1', () => r(server.address().port)))
: await getFreePort();
const bindTimer =
server && listenAfterMs > 0
? setTimeout(() => server.listen(upstreamPort, '127.0.0.1'), listenAfterMs)
: null;
const port = await getFreePort();
const target = `127.0.0.1:${upstreamPort}`;
const proc = spawnServerWithEnv(dir, port, {
GITNEXUS_UPSTREAM_URL: schemeless ? target : `http://${target}`,
GITNEXUS_SERVE_AUTH_TOKEN: TEST_AUTH_TOKEN,
// An ambient GITNEXUS_MCP_AUTH_TOKEN in the developer's shell would make the
// proxy inject one on /api/mcp, so drop it: spawn omits undefined entries,
// which unsets the inherited value. A test that wants injection sets it via
// `env` below.
GITNEXUS_MCP_AUTH_TOKEN: undefined,
...env,
});
proc.stderr.setEncoding('utf8');
proc.stderr.on('data', (chunk) => {
ctx.stderr += chunk;
});
try {
await waitForServer(port);
await fn(port, ctx);
} finally {
if (bindTimer) clearTimeout(bindTimer);
await killAndWait(proc);
if (server?.listening) {
server.closeAllConnections?.();
await new Promise((r) => server.close(r));
}
await rm(dir, { recursive: true, force: true });
}
}
it('proxies /api/* requests to the upstream server', async () => {
await withProxy({}, async (port, ctx) => {
const res = await apiRequest(port, '/api/info?x=1');
assert.equal(res.status, 200);
assert.match(res.body, /"ok":true/);
assert.equal(ctx.received.url, '/api/info?x=1', 'path + query forwarded verbatim');
});
});
it('forwards the request method and body to the upstream', async () => {
await withProxy({}, async (port, ctx) => {
await apiRequest(port, '/api/query', {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: '{"q":"hello"}',
});
assert.equal(ctx.received.method, 'POST');
assert.equal(ctx.received.body, '{"q":"hello"}');
});
});
it('strips the browser Origin and Referer before forwarding to the API', async () => {
await withProxy({}, async (port, ctx) => {
await apiRequest(port, '/api/info', {
headers: { origin: 'https://gitnexus-web.onrender.com', referer: 'https://x/y' },
});
assert.equal(
ctx.received.headers.origin,
undefined,
'Origin must be stripped so the API treats it as a trusted server-to-server call',
);
assert.equal(ctx.received.headers.referer, undefined, 'Referer must be stripped');
});
});
it('strips hop-by-hop headers before forwarding to the API', async () => {
await withProxy({}, async (port, ctx) => {
await apiRequest(port, '/api/info', {
headers: {
'keep-alive': 'timeout=5',
upgrade: 'h2c',
'proxy-authorization': 'Basic abc',
te: 'trailers',
},
});
assert.equal(ctx.received.headers['keep-alive'], undefined);
assert.equal(ctx.received.headers.upgrade, undefined);
assert.equal(ctx.received.headers['proxy-authorization'], undefined);
assert.equal(ctx.received.headers.te, undefined);
});
});
it('strips request headers that Connection names as single-hop', async () => {
await withProxy({}, async (port, ctx) => {
// RFC 7230 §6.1 lets Connection name hop-by-hop headers beyond the
// well-known eight, and those must not be forwarded either. Against a fixed
// list alone, x-custom-hop reaches the upstream.
await apiRequest(port, '/api/info', {
headers: { connection: 'x-custom-hop', 'x-custom-hop': 'private' },
});
assert.equal(ctx.received.headers['x-custom-hop'], undefined);
// Connection itself is always re-derived by Node for the upstream hop, so
// assert the client's value didn't survive rather than that it's absent.
assert.notEqual(ctx.received.headers.connection, 'x-custom-hop');
});
});
it('collapses a spoofed X-Forwarded-For chain to the load balancer entry when XFF is trusted', async () => {
const env = { GITNEXUS_PROXY_TRUST_XFF: '1' };
await withProxy({ env }, async (port, ctx) => {
// With a load balancer in front, only the last entry is the LB's; the rest
// is client-supplied and would otherwise let a caller fake req.ip and evade
// the API's rate limits.
await apiRequest(port, '/api/info', {
headers: { 'x-forwarded-for': '10.0.0.1, 1.2.3.4, 203.0.113.9' },
});
assert.equal(ctx.received.headers['x-forwarded-for'], '203.0.113.9');
});
});
it('ignores an inbound X-Forwarded-For chain when GITNEXUS_PROXY_TRUST_XFF is unset', async () => {
await withProxy({}, async (port, ctx) => {
// With nothing in front of the proxy, the whole chain is the caller's to
// write, so popping it would forward an address they chose.
await apiRequest(port, '/api/info', {
headers: { 'x-forwarded-for': '10.0.0.1, 1.2.3.4, 203.0.113.9' },
});
assert.match(ctx.received.headers['x-forwarded-for'], /127\.0\.0\.1$/);
});
});
it('ignores an inbound X-Forwarded-For chain when GITNEXUS_PROXY_TRUST_XFF is off', async () => {
const env = { GITNEXUS_PROXY_TRUST_XFF: 'off' };
await withProxy({ env }, async (port, ctx) => {
await apiRequest(port, '/api/info', {
headers: { 'x-forwarded-for': '203.0.113.9' },
});
assert.match(ctx.received.headers['x-forwarded-for'], /127\.0\.0\.1$/);
});
});
it('warns and falls back to ignoring XFF when GITNEXUS_PROXY_TRUST_XFF is "true"', async () => {
// Rejected for the same reason resolveTrustProxy rejects it server-side: it
// reads as "trust everything", the configuration this knob exists to make
// deliberate.
const env = { GITNEXUS_PROXY_TRUST_XFF: 'true' };
await withProxy({ env }, async (port, ctx) => {
await apiRequest(port, '/api/info', {
headers: { 'x-forwarded-for': '203.0.113.9' },
});
assert.match(ctx.received.headers['x-forwarded-for'], /127\.0\.0\.1$/);
assert.match(
ctx.stderr,
/GITNEXUS_PROXY_TRUST_XFF "true" is not a recognized boolean/,
'an unrecognized value must warn rather than fail silently',
);
});
});
it('forwards the socket peer, not the rotating header, on every authenticated request', async () => {
// A caller rotating X-Forwarded-For per request earns a fresh limiter key
// upstream unless this proxy overwrites it. Hitting the API server directly
// would test its own trust-proxy handling instead of this hop.
await withProxy({}, async (port, ctx) => {
const forwarded = [];
for (const spoofed of ['1.1.1.1', '2.2.2.2', '3.3.3.3', '4.4.4.4']) {
await apiRequest(port, '/api/query', {
method: 'POST',
headers: { 'content-type': 'application/json', 'x-forwarded-for': spoofed },
body: '{"q":"hi"}',
});
forwarded.push(ctx.received.headers['x-forwarded-for']);
}
assert.equal(ctx.calls, 4);
for (const address of forwarded) {
assert.match(
address,
/127\.0\.0\.1$/,
'every request must key off the socket peer, not the value the client rotated',
);
}
});
});
it('sets X-Forwarded-For from the socket peer when the client sends none', async () => {
await withProxy({}, async (port, ctx) => {
await apiRequest(port, '/api/info');
assert.match(
ctx.received.headers['x-forwarded-for'],
/127\.0\.0\.1$/,
'the API must always see a proxy-derived client address',
);
});
});
it('strips hop-by-hop headers from the upstream response', async () => {
const upstream = (_req, res) => {
res.writeHead(200, { 'Content-Type': 'text/plain', Trailer: 'X-Late' });
res.end('ok');
};
await withProxy({ upstream }, async (port) => {
const res = await apiRequest(port, '/api/info');
assert.equal(res.status, 200);
assert.equal(res.headers.trailer, undefined, 'Trailer describes the upstream hop only');
assert.equal(res.body, 'ok');
});
});
it('strips response headers that Connection names as single-hop', async () => {
const upstream = (_req, res) => {
res.writeHead(200, {
'Content-Type': 'text/plain',
Connection: 'x-upstream-hop',
'x-upstream-hop': 'internal',
});
res.end('ok');
};
await withProxy({ upstream }, async (port) => {
const res = await apiRequest(port, '/api/info');
assert.equal(res.status, 200);
assert.equal(res.headers['x-upstream-hop'], undefined, 'named on the upstream hop only');
});
});
it('does NOT proxy non-/api routes (still serves the SPA)', async () => {
await withProxy({}, async (port, ctx) => {
const res = await rawRequest(port, '/some/app/route');
assert.equal(res.status, 200);
assert.match(res.body, /spa/);
assert.equal(ctx.calls, 0, 'non-/api requests must not reach the upstream');
});
});
it('streams a chunked upstream response through to the client', async () => {
const upstream = (_req, res) => {
res.writeHead(200, { 'Content-Type': 'text/event-stream' });
res.write('data: one\n\n');
setTimeout(() => {
res.write('data: two\n\n');
res.end();
}, 20);
};
await withProxy({ upstream }, async (port) => {
const res = await apiRequest(port, '/api/stream');
assert.equal(res.status, 200);
assert.equal(res.headers['content-type'], 'text/event-stream');
assert.match(res.body, /data: one/);
assert.match(res.body, /data: two/);
});
});
it('accepts a scheme-less host:port upstream (Render fromService hostport)', async () => {
await withProxy({ schemeless: true }, async (port, ctx) => {
const res = await apiRequest(port, '/api/info');
assert.equal(res.status, 200);
assert.equal(ctx.received.url, '/api/info', 'scheme-less upstream should still be proxied');
});
});
it('serves RENDER_EXTERNAL_URL as the backend origin when GITNEXUS_BACKEND_URL is unset', async () => {
await withInjectionServer(
{ RENDER_EXTERNAL_URL: 'https://gitnexus-web.onrender.com' },
async (port) => {
const res = await rawGet(port, '/');
assert.equal(res.status, 200);
// Assert on the parsed value, not a substring of the page: a bare
// includes() would also pass if the URL appeared in a comment.
const injected = /window\.__GITNEXUS_CONFIG__=(\{.*?\});/.exec(res.body)?.[1];
assert.ok(injected, 'Expected __GITNEXUS_CONFIG__ in response body');
assert.equal(JSON.parse(injected).backendUrl, 'https://gitnexus-web.onrender.com');
},
);
});
it('returns 504 when the upstream does not respond within the timeout', async () => {
// Upstream accepts the connection but never responds — an idle hang.
const env = { GITNEXUS_PROXY_TIMEOUT_MS: '300' };
await withProxy({ upstream: () => {}, env }, async (port) => {
const res = await apiRequest(port, '/api/info');
assert.equal(res.status, 504);
});
});
it('returns 502 when the upstream is unreachable', async () => {
// Retry disabled so this fails fast (the unreachable-upstream contract).
const env = { GITNEXUS_PROXY_RETRY_ATTEMPTS: '1' };
await withProxy({ upstream: null, env }, async (port) => {
const res = await apiRequest(port, '/api/info');
assert.equal(res.status, 502);
});
});
// -- Connection-retry across an upstream restart window ---------------------
//
// `listenAfterMs: 400` binds the upstream late, so the first attempt hits
// ECONNREFUSED and must be retried — a single-instance restart. The default 3
// attempts (backoff 250ms, 500ms) span ~750ms, so a retry lands after the bind.
it('retries a connection-refused POST and succeeds once the upstream is up', async () => {
await withProxy({ listenAfterMs: 400 }, async (port, ctx) => {
const res = await apiRequest(port, '/api/analyze', {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: '{"repo":"x"}',
});
assert.equal(res.status, 200, 'first attempt should ride out the restart gap');
assert.match(res.body, /"ok":true/);
assert.equal(ctx.calls, 1, 'upstream must run the job exactly once (no double-execute)');
assert.equal(ctx.body, '{"repo":"x"}', 'buffered body replayed intact');
});
});
it('retries a bodyless DELETE, which frames no body to replay', async () => {
// Retry eligibility follows RFC 7230 §3.3.3 framing. A DELETE with neither
// Content-Length nor Transfer-Encoding has nothing to buffer, so it replays
// safely even though it isn't a GET.
await withProxy({ listenAfterMs: 400 }, async (port, ctx) => {
const res = await apiRequest(port, '/api/repo', { method: 'DELETE' });
assert.equal(res.status, 200, 'a bodyless DELETE must ride out the restart gap');
assert.equal(ctx.calls, 1);
});
});
it('falls back to the default retry budget when the knob is out of range', async () => {
// A negative attempt count is a typo. Obeying it would turn every restart
// window into a 502, silently.
const env = { GITNEXUS_PROXY_RETRY_ATTEMPTS: '-1' };
await withProxy({ listenAfterMs: 400, env }, async (port, ctx) => {
const res = await apiRequest(port, '/api/info');
assert.equal(res.status, 200);
assert.equal(ctx.calls, 1);
});
});
it('warns and keeps the default when a timeout knob is negative', async () => {
const env = { GITNEXUS_PROXY_TIMEOUT_MS: '-1' };
await withProxy({ upstream: null, env }, async (_port, ctx) => {
// Every consumer reads <= 0 as "disabled", so an unvalidated -1 removes the
// idle timeout and lets a proxied request hang forever.
assert.match(
ctx.stderr,
/GITNEXUS_PROXY_TIMEOUT_MS "-1" is below the minimum 0 -- using 120000/,
);
});
});
it('does NOT retry after the client aborts during the backoff window', async () => {
// The client aborts (~100ms) while a retry is pending, before the upstream
// binds (~400ms). The backoff guard must cancel it — otherwise the retry
// lands after the bind and runs a job nobody is waiting on.
await withProxy({ listenAfterMs: 400 }, async (port, ctx) => {
await new Promise((resolve) => {
const req = http.request({
host: '127.0.0.1',
port,
path: '/api/analyze',
method: 'POST',
headers: {
'content-type': 'application/json',
'content-length': '12',
authorization: TEST_BEARER,
},
});
req.on('error', () => {}); // aborting surfaces a local socket error; ignore
req.write('{"repo":"x"}');
req.end();
// Abort after the first attempt has failed-and-scheduled (ECONNREFUSED is
// near-instant) but well before the upstream binds at ~400ms.
setTimeout(() => {
req.destroy();
resolve();
}, 100);
});
// Wait past the upstream bind + full retry budget (~750ms) so a leaked retry
// would already have landed.
await new Promise((r) => setTimeout(r, 900));
assert.equal(ctx.calls, 0, 'aborted request must not be retried against the upstream');
});
});
it('returns 502 after exhausting the retry budget when the upstream stays down', async () => {
const env = { GITNEXUS_PROXY_RETRY_ATTEMPTS: '3' };
await withProxy({ upstream: null, env }, async (port) => {
const res = await apiRequest(port, '/api/analyze', {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: '{"repo":"x"}',
});
assert.equal(res.status, 502, 'genuinely-down upstream still returns 502 after the budget');
});
});
it('does NOT retry a POST that connects then resets before responding', async () => {
// The upstream accepts the connection, reads the whole request, then dies
// before sending any response byte — an instance that received the job and
// crashed/restarted mid-flight. Because the reset arrives AFTER connecting and
// POST is non-idempotent, replaying could run the job twice, so the proxy must
// NOT retry: the upstream sees exactly one call and the browser gets 502.
const upstream = (_req, res) => res.socket.destroy();
const env = { GITNEXUS_PROXY_RETRY_ATTEMPTS: '3' };
await withProxy({ upstream, env }, async (port, ctx) => {
const res = await apiRequest(port, '/api/analyze', {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: '{"repo":"x"}',
});
assert.equal(res.status, 502, 'post-connection reset on a POST fails fast, no retry');
// Give any (erroneous) retry a chance to fire before asserting.
await new Promise((r) => setTimeout(r, 300));
assert.equal(
ctx.calls,
1,
'non-idempotent POST must not be replayed after the upstream got it',
);
});
});
it('does NOT retry after the upstream starts streaming, then drops mid-body', async () => {
// Send headers + a partial body, then abruptly destroy the socket.
const upstream = (_req, res) => {
res.writeHead(200, { 'Content-Type': 'application/json' });
res.write('{"partial":');
setTimeout(() => res.socket.destroy(), 20);
};
const env = { GITNEXUS_PROXY_RETRY_ATTEMPTS: '3' };
await withProxy({ upstream, env }, async (port, ctx) => {
// Settle on end OR on the mid-body abort/error, so the dropped connection
// can't hang the test. What matters is that the proxy did NOT replay the
// request (no duplicate job): the upstream must see exactly 1 call.
await new Promise((resolve) => {
const req = http.request(
{
host: '127.0.0.1',
port,
path: '/api/analyze',
method: 'POST',
headers: {
'content-type': 'application/json',
'content-length': '12',
authorization: TEST_BEARER,
},
},
(res) => {
res.on('data', () => {});
res.on('end', resolve);
res.on('aborted', resolve);
res.on('error', resolve);
},
);
req.on('error', resolve);
req.write('{"repo":"x"}');
req.end();
});
// Give any (erroneous) retry a chance to fire before asserting.
await new Promise((r) => setTimeout(r, 300));
assert.equal(ctx.calls, 1, 'must not replay once the response body has started');
});
});
it('does NOT buffer or retry a body larger than the retry cap', async () => {
// Tiny cap so a modest body exceeds it and is streamed, not buffered.
const env = { GITNEXUS_PROXY_RETRY_MAX_BODY_BYTES: '16' };
const bigBody = 'x'.repeat(1024);
await withProxy({ env }, async (port, ctx) => {
const res = await apiRequest(port, '/api/analyze/upload', {
method: 'POST',
headers: { 'content-type': 'application/octet-stream' },
body: bigBody,
});
assert.equal(res.status, 200, 'over-cap body is streamed straight through');
assert.equal(ctx.body.length, bigBody.length, 'full body reaches upstream (not capped)');
});
});
it('returns 400 when the client declares a body but never finishes sending it', async () => {
// A live upstream, so a failure to reach it can't be mistaken for the body
// timeout. It must see zero requests: the proxy never connects because the
// buffering read times out first. The dedicated knob is set (leaving the
// upstream idle timeout at its default) to prove the two tune independently.
const env = { GITNEXUS_PROXY_CLIENT_BODY_TIMEOUT_MS: '300' };
await withProxy({ env }, async (port, ctx) => {
// Raw socket (not http.request, which would auto-finish the body): send a
// Content-Length: 100 request but only 10 bytes, then hold the socket open.
// We never close our side — the proxy must close it for us once the body
// read times out (via `Connection: close`), rather than holding the
// half-open connection until the server requestTimeout reaps it.
const { status, serverClosed, raw } = await new Promise((resolve) => {
const sock = connect(port, '127.0.0.1', () => {
sock.write(
'POST /api/analyze HTTP/1.1\r\n' +
'Host: 127.0.0.1\r\n' +
'Content-Type: application/json\r\n' +
`Authorization: ${TEST_BEARER}\r\n` +
'Content-Length: 100\r\n' +
'\r\n' +
'x'.repeat(10), // fewer than 100 bytes, then stall
);
});
let buf = '';
let status = null;
// Fail-safe: if the proxy never closes on its own, report serverClosed
// false (so the assertion fails cleanly) instead of hanging the test.
const guard = setTimeout(() => {
sock.destroy();
resolve({ status, serverClosed: false, raw: buf });
}, 2000);
sock.setEncoding('utf8');
sock.on('data', (chunk) => {
buf += chunk;
if (status === null) {
const m = buf.split('\r\n', 1)[0].match(/^HTTP\/\d\.\d (\d{3})/);
if (m) status = Number(m[1]);
}
});
// The server closing its side (Connection: close) ends our socket; treat
// any teardown initiated by the server as "closed promptly".
sock.on('error', () => {}); // a reset may precede 'close'; swallow it
sock.on('close', () => {
clearTimeout(guard);
resolve({ status, serverClosed: true, raw: buf });
});
});
assert.equal(status, 400, 'stalled body read must be bounded and return 400, not hang');
assert.ok(
serverClosed,
'proxy must close the half-open connection promptly, not hold it until requestTimeout',
);
assert.match(
raw.toLowerCase(),
/connection: close/,
'the 400 for a stalled body must advertise Connection: close',
);
assert.equal(ctx.calls, 0, 'proxy must not connect upstream when the body never arrives');
});
});
// -- Token gate at the public edge (GITNEXUS_SERVE_AUTH_TOKEN) --------------
//
// The proxy terminates the browser Origin, so the API's own write guard can't
// see a cross-site request coming. The token replaces it, checked on the way in.
it('answers an /api/* request with no Authorization header with a well-formed 401', async () => {
await withProxy({}, async (port, ctx) => {
const res = await rawRequest(port, '/api/health');
assert.equal(res.status, 401);
assert.equal(res.headers['www-authenticate'], 'Bearer');
assert.match(res.headers['content-type'], /application\/json/);
// The UI dispatches on the stable code, not on message text.
assert.deepEqual(JSON.parse(res.body), { error: 'unauthorized', code: 'unauthorized' });
assert.equal(ctx.calls, 0, 'an unauthenticated request must cost nothing upstream');
});
});
it('closes the connection on a rejected request rather than draining its body', async () => {
// The 401 is answered before the body is read, so without Connection: close
// Node drains up to 64KB of an unauthenticated upload to keep the socket
// reusable. Same reasoning as the stalled-body 400 above.
await withProxy({}, async (port, ctx) => {
const res = await rawRequest(port, '/api/analyze', {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ path: '/etc' }),
});
assert.equal(res.status, 401);
assert.equal(res.headers.connection, 'close');
assert.equal(ctx.calls, 0);
});
});
it('rejects a wrong token of the same length', async () => {
await withProxy({}, async (port, ctx) => {
const wrong = 'x'.repeat(TEST_AUTH_TOKEN.length);
const res = await apiRequest(port, '/api/health', {
headers: { authorization: `Bearer ${wrong}` },
});
assert.equal(res.status, 401);
assert.equal(ctx.calls, 0);
});
});
it('rejects a wrong token of a different length', async () => {
// The unequal-length branch takes a different path through the comparison
// (dummy compare, no timingSafeEqual on the real buffers) and still must 401.
await withProxy({}, async (port, ctx) => {
const res = await apiRequest(port, '/api/health', {
headers: { authorization: 'Bearer short' },
});
assert.equal(res.status, 401);
assert.equal(ctx.calls, 0);
});
});
it('rejects the raw token without the Bearer prefix', async () => {
await withProxy({}, async (port, ctx) => {
const res = await apiRequest(port, '/api/health', {
headers: { authorization: TEST_AUTH_TOKEN },
});
assert.equal(res.status, 401);
assert.equal(ctx.calls, 0);
});
});
it('forwards an /api/* request that carries the correct token', async () => {
await withProxy({}, async (port, ctx) => {
const res = await apiRequest(port, '/api/health', {
headers: { authorization: TEST_BEARER },
});
assert.equal(res.status, 200);
assert.equal(ctx.calls, 1);
});
});
it('strips the Authorization header instead of forwarding the edge token', async () => {
// The edge credential is spent and stripped at this hop. Forwarding it
// would copy a live credential into another service's logs. With no
// GITNEXUS_MCP_AUTH_TOKEN configured — the default — nothing replaces it.
await withProxy({}, async (port, ctx) => {
const res = await apiRequest(port, '/api/mcp', { method: 'POST', body: '{}' });
assert.equal(res.status, 200, 'the request itself must still be proxied');
assert.equal(ctx.received.headers.authorization, undefined);
});
});
// -- Upstream MCP token injection (GITNEXUS_MCP_AUTH_TOKEN) -----------------
//
// A backend running protocol-layer MCP auth expects its own Bearer on
// /api/mcp, and the edge credential can't serve as one. Both services are
// configured with the same GITNEXUS_MCP_AUTH_TOKEN; this hop spends the edge
// token and substitutes the backend one, for that route only.
// Stands in for a `serve` with MCP Bearer auth enabled: only the exact backend
// credential gets through, so a passing two-hop request proves what was sent.
const mcpBackend = (req, res) => {
if (req.headers.authorization !== TEST_MCP_BEARER) {
res.writeHead(401, { 'Content-Type': 'application/json; charset=utf-8' });
res.end('{"error":"unauthorized"}');
return;
}
res.writeHead(200, { 'Content-Type': 'application/json; charset=utf-8' });
res.end('{"ok":true}');
};
it('treats a blank GITNEXUS_MCP_AUTH_TOKEN as unset and still strips', async () => {
const env = { GITNEXUS_MCP_AUTH_TOKEN: ' ' };
await withProxy({ env }, async (port, ctx) => {
const res = await apiRequest(port, '/api/mcp', { method: 'POST', body: '{}' });
assert.equal(res.status, 200);
assert.equal(ctx.received.headers.authorization, undefined);
});
});
it('replaces the edge credential with the upstream MCP token on /api/mcp', async () => {
const env = { GITNEXUS_MCP_AUTH_TOKEN: TEST_MCP_TOKEN };
await withProxy({ upstream: mcpBackend, env }, async (port, ctx) => {
const res = await apiRequest(port, '/api/mcp', { method: 'POST', body: '{}' });
assert.equal(res.status, 200, 'a backend that demands the MCP token must accept this hop');
assert.equal(ctx.received.headers.authorization, TEST_MCP_BEARER);
assert.notEqual(
ctx.received.headers.authorization,
TEST_BEARER,
'the edge credential must never be forwarded',
);
});
});
it('injects the upstream MCP token on /api/mcp subpaths and ignores the query string', async () => {
const env = { GITNEXUS_MCP_AUTH_TOKEN: TEST_MCP_TOKEN };
await withProxy({ upstream: mcpBackend, env }, async (port, ctx) => {
for (const path of ['/api/mcp/messages', '/api/mcp?session=abc']) {
const res = await apiRequest(port, path, { method: 'POST', body: '{}' });
assert.equal(res.status, 200, `${path} must reach the MCP backend authenticated`);
assert.equal(ctx.received.headers.authorization, TEST_MCP_BEARER, path);
}
});
});
it('leaves non-MCP routes stripped when an upstream MCP token is configured', async () => {
// /api/mcpfoo shares a prefix with the MCP route but is not it, and a plain
// API route never carries a protocol credential.
const env = { GITNEXUS_MCP_AUTH_TOKEN: TEST_MCP_TOKEN };
await withProxy({ env }, async (port, ctx) => {
for (const path of ['/api/mcpfoo', '/api/health']) {
const res = await apiRequest(port, path);
assert.equal(res.status, 200);
assert.equal(ctx.received.headers.authorization, undefined, path);
}
});
});
it('never gates static assets behind the token', async () => {
// The UI has to load before it can prompt for a token.
await withProxy({}, async (port, ctx) => {
for (const path of ['/', '/index.html', '/some/app/route']) {
const res = await rawRequest(port, path);
assert.equal(res.status, 200, `${path} must be served without a token`);
assert.match(res.body, /spa/);
}
assert.equal(ctx.calls, 0);
});
});
// Run docker-server.mjs to completion and report how it exited. Used for the
// boot-time refusal, which never reaches a listening state.
function runUntilExit(cwd, env) {
return new Promise((resolve, reject) => {
const proc = spawn(process.execPath, [serverScript], {
cwd,
env: { ...process.env, ...env },
stdio: 'pipe',
});
let stderr = '';
proc.stderr.setEncoding('utf8');
proc.stderr.on('data', (chunk) => {
stderr += chunk;
});
proc.on('error', reject);
proc.on('exit', (code) => resolve({ code, stderr }));
// A server that starts instead of refusing never exits, so name that failure
// here rather than letting it surface as a timeout or a null exit code.
setTimeout(() => {
proc.kill();
reject(new Error('docker-server.mjs kept running; it was expected to refuse and exit'));
}, 5000).unref();
});
}
async function withDistDir(fn) {
const dir = await mkdtemp(join(tmpdir(), 'gitnexus-boot-'));
await mkdir(join(dir, 'dist'), { recursive: true });
await writeFile(join(dir, 'dist', 'index.html'), '<html><body>spa</body></html>');
try {
await fn(dir);
} finally {
await rm(dir, { recursive: true, force: true });
}
}
it('refuses to start when the proxy is enabled without a token', async () => {
await withDistDir(async (dir) => {
const port = await getFreePort();
const { code, stderr } = await runUntilExit(dir, {
PORT: String(port),
GITNEXUS_UPSTREAM_URL: '127.0.0.1:4747',
GITNEXUS_SERVE_AUTH_TOKEN: undefined,
});
assert.equal(code, 1, 'an unauthenticated public proxy must fail closed at boot');
assert.match(stderr, /Refusing to start/);
assert.match(stderr, /GITNEXUS_SERVE_AUTH_TOKEN/);
});
});
it('refuses to start when the token is short enough to guess', async () => {
// Nothing rate-limits a failed token, so a weak one is guessable at network
// speed. The floor is what makes the missing limiter safe.
await withDistDir(async (dir) => {
const port = await getFreePort();
const { code, stderr } = await runUntilExit(dir, {
PORT: String(port),
GITNEXUS_UPSTREAM_URL: '127.0.0.1:4747',
GITNEXUS_SERVE_AUTH_TOKEN: 'hunter2',
});
assert.equal(code, 1);
assert.match(stderr, /shorter than 32 characters/);
assert.ok(!stderr.includes('hunter2'), 'the refusal must never echo the token');
});
});
it('treats a whitespace-only token as absent rather than as a short one', async () => {
// ' ' trims to empty, so this must hit the missing-token refusal, not the
// length one.
await withDistDir(async (dir) => {
const port = await getFreePort();
const { code, stderr } = await runUntilExit(dir, {
PORT: String(port),
GITNEXUS_UPSTREAM_URL: '127.0.0.1:4747',
GITNEXUS_SERVE_AUTH_TOKEN: ' ',
});
assert.equal(code, 1);
assert.match(stderr, /is set without GITNEXUS_SERVE_AUTH_TOKEN/);
});
});
it('starts normally with neither the proxy nor a token configured', async () => {
// docker-compose's default: static assets only, nothing to gate, no refusal.
await withDistDir(async (dir) => {
const port = await getFreePort();
const proc = spawnServerWithEnv(dir, port, { GITNEXUS_SERVE_AUTH_TOKEN: undefined });
try {
await waitForServer(port);
const res = await rawRequest(port, '/');
assert.equal(res.status, 200);
assert.match(res.body, /spa/);
} finally {
await killAndWait(proc);
}
});
});

109
docs/languages/objective-c-provider.md Normal file
View file

@ -0,0 +1,109 @@
# Objective-C Language Provider
Status: implemented
The deterministic provider is covered by focused unit and integration tests. The parser-loader ABI
smoke runs in the published multi-OS test matrix, and the native prebuild workflow owns
Objective-C together with all six vendored grammar targets. This status describes the implemented
MVP; it does not promise full Objective-C runtime dispatch.
## Goal
Add deterministic, symbol-level Objective-C analysis to GitNexus. The first release must support high-confidence code navigation and direct static dependency analysis for `.m`, `.mm`, and Objective-C `.h` files. It must not imply that Objective-C runtime dispatch is fully resolved.
The provider belongs in the existing language-provider and scope-resolution extension points. Shared ingestion code must remain language-agnostic.
## Compatibility contract
- Existing language detection and parsing must remain unchanged.
- A `.h` file must be classified from its content or surrounding context; it cannot be unconditionally claimed by Objective-C because C and C++ also use that extension.
- If Objective-C grammar loading fails, the error must clearly name the missing provider/grammar and cannot corrupt a previously valid index.
- Provider and grammar versions must be stored in index metadata. A version change that can alter node identity or edges requires a full rebuild.
- No LLM participates in parsing, name resolution, or edge creation. Analysis is Tree-sitter plus deterministic static resolution.
## MVP model
The provider must extract and connect:
- Classes, superclasses, protocols, categories, class extensions, properties, ivars, C functions, imports, declarations, and implementations.
- Instance and class methods, preserving their complete multi-part selector.
- Inheritance, protocol conformance, import, declaration/implementation, host-class/category, and statically resolved call relationships.
Stable identity must include enough ownership to distinguish same-named methods. Recommended forms are:
```text
objc:class:<ClassName>
objc:protocol:<ProtocolName>
objc:category:<HostClass>:<CategoryName>
objc:method:<Owner>:-:<selector>
objc:method:<Owner>:+:<selector>
objc:function:<qualified-or-file-scoped-name>
```
For example, `-loadData:completion:` and `+loadData:completion:` are different symbols. A category method remains linked to both its category and host class; querying the host class must expose distributed implementations.
## Resolution policy
Resolution must be conservative. A missing or dynamic target is evidence of uncertainty, not proof that no target exists.
| Receiver case | Required result |
| --------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Explicit class name, `self`, or `super` | Resolve when the owner is statically known. |
| Local, parameter, property, or ivar with known static type | Resolve to matching owner and selector. |
| Protocol-typed receiver | Link the protocol method and identify possible implementations as candidates. |
| Multiple host-class/category implementations of one selector | Record all static candidates as evidence; do not emit a certain call edge because runtime image-load order is unknown. |
| `id`, `Class`, macros, reflection, `performSelector:`, `NSInvocation`, runtime injection, or unknown type | Store selector/location with `resolution=unresolved`; do not emit a certain call edge. |
The provider should first collect file-local declarations, imports, and types, then resolve across the repository. It must use structured Tree-sitter captures or AST traversal, not regular expressions over source text. Multi-part selectors, block arguments, nullability annotations, generics, macros, and multiline declarations make a regex-only extractor unsafe.
## Imports and incremental correctness
- Resolve quoted project imports against the current directory, configured include roots, and indexed headers. Model framework imports as external-module evidence without downloading SDK source.
- Merge `@interface`, `@implementation`, categories, and extensions across files.
- A changed header, protocol, class declaration, or category invalidates importing and affected implementation/call-resolution state. Incremental output after such a change must match a full rebuild.
- Index metadata must record provider version, grammar version, include/exclude configuration, and parsing options used for resolution.
## Implementation sequence
1. Add and package a pinned Objective-C Tree-sitter grammar; verify macOS arm64 and the production Linux runner can load it.
2. Add language detection for `.m`, `.mm`, and content-classified `.h` files.
3. Implement AST extraction and stable IDs for declarations and definitions.
4. Implement repository-level merge, imports, inheritance, protocol, and category relationships.
5. Add conservative message-send resolution and explicit unresolved evidence.
6. Integrate invalidation, metadata comparison, MCP/CLI output, and fixtures.
## Fixtures and acceptance
Create a minimal Objective-C fixture containing a class, protocol, category, extension, superclass, properties, ivars, C function, imports, multi-part selector, block parameter, `self`, `super`, protocol receiver, and `id` receiver. Use `symodulebridge` as a real integration fixture after the minimal suite is stable.
The acceptance bar is:
- `query "SYModuleCaller"` yields class/method semantic nodes, not only file nodes.
- `context "SYModuleCaller" --file <path>` yields declaration, implementation, imports, and known references.
- Known statically typed message sends create call edges; dynamic sends are marked unresolved.
- Same selector on multiple classes, a category override, and `+` versus `-` methods remain distinct.
- A `.m`, `.h`, protocol, or category edit produces results equivalent to a clean rebuild.
- Generated documentation, dependency directories, and build output are excluded through explicit indexing configuration.
## Non-goals
The MVP does not promise exact runtime type inference for `id` or `instancetype`, reflection, swizzling, arbitrary category replacement, dynamic selector construction, or complete impact analysis across every runtime dispatch path. Tool results must surface confidence and unresolved evidence rather than presenting guesses as certain graph facts.
## Current implementation coverage
Implemented capabilities:
- Vendored `tree-sitter-objc` grammar, registered through the existing Tree-sitter loader.
- `.m` and `.mm` language mapping plus content-based `.h` classification so plain C/C++ headers are not unconditionally claimed.
- LanguageProvider extraction for classes, protocols, categories, extensions, methods, properties, ivars, C functions, imports, unresolved message evidence, stable Objective-C qualified names, and provider/grammar metadata.
- Length-preserving preprocessing of bare, file-scope all-caps macro markers before Tree-sitter parsing. This recovers declarations after wrappers such as `RCT_EXTERN_C_BEGIN` / `RCT_EXTERN_C_END` without expanding macros or adding framework-specific rules.
- ScopeResolver edges for imports, inheritance, protocol conformance, category host membership, implementation evidence, and conservative static message sends.
- Persisted query/context support for Objective-C class and method nodes, including implementation evidence via `DECLARES`.
- Regression tests for grammar loading, `.h` classification, stable identities, conservative calls, metadata feature mismatch, persisted query/context behavior, and incremental-vs-force parity for Objective-C fixture edits.
Known limits of this MVP:
- The first version does not perform full Objective-C runtime dispatch, swizzling, dynamic selector construction, macro expansion, or `id` flow inference. Bare file-scope marker macros are elided only to preserve parser recovery; their expansion semantics are not interpreted.
- Protocol receiver handling records the protocol method and candidate implementation evidence, but candidate implementations are not emitted as certain call edges.
- When a host class and one or more named categories define the same selector, the provider records candidate evidence rather than choosing a runtime winner or emitting multiple certain call edges.
- Objective-C++ `.mm` files are parsed with the Objective-C grammar path for this MVP; deep C++ semantic extraction inside Objective-C++ bodies remains outside this provider.

473
docs/plans/2026-08-04-gitnexus-plan-python-dotted-namespace-receiver.md Normal file
View file

@ -0,0 +1,473 @@
# GitNexus Engineering Plan
> Task: Emit the missing `CALLS` edge for Python's unaliased multi-segment namespace import (`import pkg.db` + `pkg.db.session_scope()`), issue #2826.
> Evidence verified at commit b2cd1c2ad637657125248c0dd2046de71ceea965; GitNexus index 13 commits behind HEAD, refresh skipped: every cited path is byte-identical between the index commit (1ef6447e) and the pinned commit — verified by blob-id comparison, so no graph claim here rests on drifted content. PDG layer absent from this index (`MATCH ()-[r:CodeRelation {type:'CDG'}]->() RETURN count(r)` → 0); `--pdg` upgrade skipped, source reads substitute at higher evidence strength.
> Evidence provenance schema 2; global dirty digest 0912a3ee3219cb75c82aefbf9f010e8dbe313150d6553768fd55d22af87a135c; cited-path manifest 13 sorted entries; exact generated plan path excluded.
## 1. Objective
`import pkg.db` followed by `pkg.db.session_scope()` must emit a `CALLS` edge from the caller to `session_scope`, matching the three sibling import spellings that already resolve (`from pkg.db import session_scope`, `import pkg.db as pdb`, `from pkg import db`). Two same-package imports in one file (`import pkg.a` + `import pkg.b`) must not cross-resolve, and no shared file under `gitnexus/src/core/ingestion/` may name a language (AGENTS.md §42).
## 2. Current Behaviour
The failure is a **key/lookup mismatch inside one map**, not a missing resolution path.
For `import pkg.db`, `splitImportStmt` emits one match with `@import.source` = the whole `dotted_name` text `"pkg.db"` `[verified]` (`gitnexus/src/core/ingestion/languages/python/import-decomposer.ts:46-54`). `interpretPythonImport`'s `'plain'` arm then splits it `[verified]` (`gitnexus/src/core/ingestion/languages/python/interpret.ts:33-42`):
```ts
case 'plain': {
// `import numpy`
if (sourceCap === undefined) return null;
return {
kind: 'namespace',
localName: sourceCap.text.split('.')[0]!, // `import a.b.c` exposes `a`
importedName: sourceCap.text,
targetRaw: sourceCap.text,
};
}
```
`finalizeImportEdges` carries both halves onto the edge: `localName` verbatim, and `targetExportedName = parsed.importedName` for `kind === 'namespace'` `[verified]` (`gitnexus-shared/src/scope-resolution/finalize-algorithm.ts:398-406, 434-447`). So the finalized `ImportEdge` is `{ localName: 'pkg', targetExportedName: 'pkg.db', targetFile: 'pkg/db.py', kind: 'namespace' }`.
`collectNamespaceTargets` keys **only on `localName`** `[verified]` (`gitnexus/src/core/ingestion/scope-resolution/scope/namespace-targets.ts:44-57`), producing `{'pkg' → ['pkg/db.py']}`.
At the call site, Python's query binds the attribute's `object` field with a wildcard — `object: (_) @reference.receiver` `[verified]` (`gitnexus/src/core/ingestion/languages/python/query.ts:267-270`) — so for `pkg.db.session_scope()` the receiver node is the inner `attribute`, and `extractExplicitReceiver` takes its raw text `[verified]` (`gitnexus/src/core/ingestion/scope-extractor.ts:1235-1239`): `receiverName === 'pkg.db'`.
`emitReceiverBoundCalls` then walks its cases `[verified]` (`gitnexus/src/core/ingestion/scope-resolution/passes/receiver-bound-calls.ts:404-421, 546-655, 831-848`):
- **Case 0 (compound receiver)** fires because `receiverName.includes('.')` (line 563-567). It asks `resolveCompoundReceiverClass` for a **class**; `pkg.db` names a module, so it returns `undefined`, sets `compoundReceiverUnresolved = true`, and — critically — does **not** `handledSites.add`, so control falls through (lines 577, 622-655).
- **Case 1 (namespace receiver)** runs `namespaceTargets.get('pkg.db')` (line 832). The map holds `'pkg'`. Miss.
- **Case 1.5** needs `provider.resolveQualifiedReceiverMember`, implemented only by the C++ provider `[verified]` (`gitnexus/src/core/ingestion/languages/cpp/scope-resolver.ts:399-406`; `context` on that symbol shows one outgoing call to `resolveCppQualifiedNamespaceMember` and no other implementer). Python leaves it undefined, so the case is skipped.
No later case types a module receiver, so the site drops. Reproduced on both `origin/main` and PR #2810's head; PR #2810 changes Python receiver *typing* (`languages/python/receiver-binding.ts`) and does not touch this path `[verified]` by running the repro against both trees.
The three sibling spellings resolve because each binds a **single-segment** local name: `pdb` (alias arm), `session_scope` (named binding, not a receiver at all), and `db` (reclassified to `kind: 'namespace'` by #2770's `isNamespaceImport` hook, keying the map on `'db'`).
## 3. Relevant Architecture
`collectNamespaceTargets` is the shared, language-neutral bridge between finalized import edges and receiver resolution. Its contract note already states that `ImportEdge.kind === 'namespace'` is authoritative and that providers may reclassify into it — that reclassification hook (`isNamespaceImport`) is #2770's extension point `[verified]` (`gitnexus-shared/src/scope-resolution/finalize-algorithm.ts:99-107`).
Its output feeds three consumers, all per-file (`fileCompoundOpts`, `receiver-bound-calls.ts:405-406`):
1. `emitReceiverBoundCalls` Case 1 — namespace-receiver member calls (`receiver-bound-calls.ts:832`);
2. `resolveConstructionExpressionClass` — namespace-qualified construction `pkg.db.Model()` (`compound-receiver.ts:245-260`);
3. `resolveCompoundReceiverClass`'s namespace-qualified-constructor disambiguation `options.namespaceTargets?.has(objExpr)` (`compound-receiver.ts:759-766`).
AGENTS.md line 42 is the binding constraint: *"Shared code in `gitnexus/src/core/ingestion/` must not name languages — plug language behavior in via `LanguageProvider` / `ScopeResolver` hooks."* `[verified]`
## 4. GitNexus Findings
- `context({name: 'collectNamespaceTargets', repo: 'GitNexus'})` — `epistemic: "exact"`; incoming calls are exactly two: `emitReceiverBoundCalls` (`.../passes/receiver-bound-calls.ts`) and a test-local `build` in `test/unit/scope-resolution/python/python-module-namespace-construction.test.ts`. `[graph]` These are the d=1 dependents; the two `compound-receiver.ts` consumers reach the map by parameter rather than by call, so they do not appear here and were found by source grep `[verified]`.
- `context({name: 'resolveQualifiedReceiverMember', repo: 'GitNexus'})` — resolves to a single definition at `languages/cpp/scope-resolver.ts:399`, `outgoing.calls: [resolveCppQualifiedNamespaceMember]`, no incoming. `[graph]` Confirms the Case-1.5 hook is C++-only, matching the issue reporter's read of the published bundle.
- `cypher({statement: "MATCH ()-[r:CodeRelation {type: 'CDG'}]->() RETURN count(r)"})` — `| cdg_rows | 0 |`. `[graph]` The index carries no PDG layer; §5 is therefore empty by fact, not by omission.
- Related tests located by directory listing `[verified]`: `test/fixtures/lang-resolution/` already holds `python-module-import`, `python-bare-import`, `python-plain-import-alias`, `python-multi-segment-ancestor-import`, `python-function-local-namespace-import`, `python-class-body-namespace-import`, and #2770's `python-from-module-alias`. `test/integration/resolvers/python.test.ts` is the convention-matching home for the new assertions (#2770 added its coverage there, +38 lines).
## 5. Statement-Level PDG Findings
Empty by fact: the current index has zero `CDG` rows, so no statement-level slice exists to build. A `--pdg` re-index was deliberately not run — it is the largest fixed cost available to this session, the analyzer holds no writer lock against a live MCP server (#2658), and every constraint the slice would supply (which case gates the namespace lookup, whether Case 0's failure falls through) was read directly from source at higher evidence strength in §2.
## 6. Proposed Changes
### 6.1 `collectNamespaceTargets` — also key on the dotted access path
- **File:** `gitnexus/src/core/ingestion/scope-resolution/scope/namespace-targets.ts`
- **Symbol:** `collectNamespaceTargets` (source-verified)
- **Responsibility:** map every receiver spelling that names an imported module to that module's file(s).
- **Change:** inside the existing edge loop, after recording `edge.localName`, also record `edge.targetExportedName` **when it contains a dot and its first dot-separated segment equals `edge.localName`**. Same array-dedupe as the existing key.
- **Why this is language-neutral (AGENTS.md §42):** the condition names no language. It encodes one structural fact — *a namespace binding whose exported module name is a dotted path rooted at the local name is also reachable under that whole path.* Verified against every other namespace-emitting provider at the pinned commit `[verified]`:
- TypeScript `import * as X from './y'` → `localName 'X'`, `importedName './y'`; first segment `''` ≠ `'X'` → no key (`languages/typescript/interpret.ts:77-81, 118-122`).
- C# `using System.Collections.Generic` → `localName 'Generic'` (last segment), `importedName 'System.Collections.Generic'`; first segment `'System'` ≠ `'Generic'` → no key (`languages/csharp/interpret.ts:33-37, 62-66`).
- Go / Rust / Ruby → `localName === importedName`, no dot → no key (`languages/{go,rust,ruby}/interpret.ts`).
- Python `import pkg.db` → `'pkg' === 'pkg.db'.split('.')[0]` → key `'pkg.db'` added. This is the only provider the predicate admits today.
- **Constraint:** additive only. The existing `localName` key must keep its current value and ordering so no currently-resolving site changes target.
- **Two-package safety:** `import pkg.a` + `import pkg.b` in one file yields `{'pkg' → ['pkg/a.py','pkg/b.py'], 'pkg.a' → ['pkg/a.py'], 'pkg.b' → ['pkg/b.py']}`. Receiver `pkg.a` hits exactly one file; the ambiguous `'pkg'` bucket is only reachable by a receiver literally spelled `pkg`, which is unchanged from today. `[inferred]` — pinned by a test in §8.
### 6.2 `isNamespaceNameShadowed` — test the root segment, not the dotted path
- **File:** `gitnexus/src/core/ingestion/scope-resolution/passes/compound-receiver.ts`
- **Symbol:** `isNamespaceNameShadowed` (source-verified, lines 152-183) and its one call site at line 250.
- **Defect this fix activates:** the guard walks the scope chain looking for a binding, type binding, lexical name, or owned def **named exactly `namespaceName`**. With 6.1 in place, `namespaceName` can be `'pkg.db'`, but Python binds only `pkg` — so a local `pkg = something` that genuinely shadows the import would fail to suppress the namespace interpretation, and the "verified namespace is authoritative" branch (line 249-259) would return a wrong class instead of declining.
- **Change:** shadow-test the first dot-separated segment of `namespaceName` (identical behaviour for the single-segment names it sees today, since root === whole name).
- **Not scope creep:** 6.1 is what first routes a dotted name into this guard; shipping 6.1 without it introduces the false positive.
### 6.3 No change required in `receiver-bound-calls.ts`
Case 1's lookup already uses the full dotted `receiverName` and Case 0's failure already falls through to it (`receiver-bound-calls.ts:577, 622-655, 832`) `[verified]`. Recorded here so the executor does not "fix" a path that is already correct.
## 7. Implementation Sequence
1. **Add the failing fixture and assertions first.** Create `gitnexus/test/fixtures/lang-resolution/python-dotted-namespace-import/` (files in §8) and a `describe` block in `gitnexus/test/integration/resolvers/python.test.ts` following the file's existing `writeFixtureRepo` + `mkdtempSync` convention. Confirm the dotted row fails and all three control rows pass. Delete the scratch `gitnexus/test/integration/resolvers/repro-2826-python-dotted-import.test.ts` in this step — its content is superseded by the fixture-backed tests.
2. **Implement 6.1** in `namespace-targets.ts`, and update its header contract note to state that a namespace edge may be keyed both by its local name and by a dotted access path rooted at that name. Re-run the step-1 tests: the dotted row must flip to passing with the controls still green.
3. **Implement 6.2** in `compound-receiver.ts` with the shadowing test from §8 (a local `pkg = Decoy()` must suppress, not misresolve).
4. **Run the regression surface**: full resolver + scope-resolution integration suites, both packages' `tsc --noEmit`.
5. **Regenerate recorded baselines once, last.** Run each `--check` gate; regenerate only the baselines that actually moved (`bench/receiver-resolution/baseline.json` is the expected one — this change adds resolved edges). Per plan-template §7, this is deliberately the final step so intermediate commits do not churn and re-drift the artifacts.
## 8. Test Strategy
**New fixture** `gitnexus/test/fixtures/lang-resolution/python-dotted-namespace-import/`:
| file | contents |
| --- | --- |
| `pkg/__init__.py` | empty |
| `pkg/db.py` | `def session_scope(): ...` |
| `pkg/cache.py` | `def session_scope(): ...` — the decoy that makes cross-resolution detectable |
| `caller_dotted.py` | `import pkg.db` + `def uses_dotted(): return pkg.db.session_scope()` |
| `caller_from.py`, `caller_alias.py`, `caller_frommod.py` | the three sibling controls from the issue |
| `caller_two_pkgs.py` | `import pkg.db` **and** `import pkg.cache`, one function calling each |
| `caller_deep.py` | `import pkg.sub.deep` + `pkg.sub.deep.f()` (3-segment) |
| `caller_shadowed.py` | module-level `import pkg.db`, then a function with a local `pkg = Decoy()` before `pkg.db.session_scope()` |
**Scenarios** (input → action → expected):
1. `caller_dotted.py` → run pipeline → `CALLS` edge `uses_dotted` → `pkg/db.py:session_scope`, `reason: 'import-resolved'`. **This is the issue's acceptance row.**
2. The three sibling callers → same run → all three still resolve to `pkg/db.py:session_scope`. Regression control: a run where the controls also broke would prove nothing about row 1.
3. `caller_two_pkgs.py` → `pkg.db.session_scope()` resolves **only** to `pkg/db.py` and `pkg.cache.session_scope()` **only** to `pkg/cache.py`; assert the absence of the crossed pair explicitly, not just the presence of the right one.
4. `caller_deep.py` → 3-segment receiver resolves — proves the predicate is not hard-coded to two segments.
5. `caller_shadowed.py` → **no** edge from the shadowed function to `pkg/db.py` (6.2's guard). Fails loudly if 6.2 regresses.
6. Cross-language non-regression: the existing TypeScript / C# / Go namespace-import resolver tests must stay green unchanged — that is the executable proof the new key is not minted for them.
**Tests to update:** `gitnexus/test/integration/resolvers/python.test.ts` (add the describe block). `gitnexus/test/unit/scope-resolution/python/python-module-namespace-construction.test.ts` is a direct `collectNamespaceTargets` caller — re-run it; extend it only if its expectations enumerate map keys exhaustively.
**Verification commands** (each verified to exist in `gitnexus/package.json` / `.github/workflows/ci-tests.yml` at the pinned commit):
```bash
# from gitnexus/ — pretest:integration runs scripts/build.js, so the parse worker exists
GITNEXUS_WORKER_READY_TIMEOUT_MS=60000 npm run test:integration -- test/integration/resolvers/python.test.ts
GITNEXUS_WORKER_READY_TIMEOUT_MS=60000 npm run test:integration -- test/integration/resolvers
npm run test:unit -- test/unit/scope-resolution
npx tsc --noEmit # and the same in ../gitnexus-shared
node --import tsx bench/receiver-resolution/measure.mjs --check
node --import tsx bench/python-scope/measure.mjs --check
node --import tsx bench/python-scope/import-target-fingerprint.mjs --check
node --import tsx bench/scope-capture/measure.mjs --check
```
`GITNEXUS_WORKER_READY_TIMEOUT_MS=60000` is required on this host: the default 5000 ms worker-ready deadline fails as a crash-loop here (observed while reproducing the issue), which is environmental, not a code fault.
## 9. Risk and Impact Analysis
Accounting for every direct (d=1) dependent of the changed map:
| d=1 dependent | risk | mitigation |
| --- | --- | --- |
| `emitReceiverBoundCalls` Case 1 (`receiver-bound-calls.ts:832`) | New keys make previously-dropped sites resolve. A wrong target would be a *new* false edge. | The predicate admits only Python's `import a.b` shape; each new key maps to exactly one file per import statement. §8 scenario 3 pins non-crossing. |
| `resolveConstructionExpressionClass` (`compound-receiver.ts:245-260`) | `pkg.db.Model()` now takes the "verified namespace is authoritative" branch, which deliberately does **not** fall through on a miss or ambiguity — so a wrong key would convert a working heuristic resolution into a silent decline. | The branch requires `namespaceFiles.length > 0`, i.e. the import genuinely resolved. Ambiguity still returns `undefined` (`namespaceMatches.length === 1` guard). Shadowing is fixed by 6.2. |
| `resolveCompoundReceiverClass` namespace-constructor disambiguation (`compound-receiver.ts:759-766`) | `namespaceTargets.has(objExpr)` now true for dotted namespaces, routing `pkg.db.Model(x).run()` into the construction interpretation. | Correct by intent — that branch exists precisely to make a namespace-qualified bare constructor safe. Behaviour change, so §8 should include a construction row if the fixture's cost is low. |
| `test/unit/scope-resolution/python/python-module-namespace-construction.test.ts:build` | May assert exact map contents. | Re-run in step 4; extend rather than weaken if it enumerates keys. |
| C++ provider | Case 1 is skipped entirely for C++ (`provider.resolveQualifiedReceiverMember !== undefined`), but the two `compound-receiver.ts` consumers are **not** provider-gated. | C++ `#include` does not produce a `kind: 'namespace'` edge with a dotted `targetExportedName` rooted at its local name; the predicate declines. Covered by the existing C++ suites plus `bench/cpp-qualified-ns/measure.mjs --check`. |
**Recorded-artifact risk:** `bench/receiver-resolution/measure.mjs --check` gates both a shape matrix and a drop-count arm; new resolved edges are expected to move the count arm and the gate fails on drift. Regenerating in step 5 only (per §7) keeps intermediate commits clean. `bench/python-scope/*` and `bench/scope-capture/*` fingerprint captures and import-target resolution — neither is touched by this change, so a movement there is a signal to stop and investigate, not to regenerate.
**Performance:** one extra `Map.set` per multi-segment namespace import per file; the loop is already O(module import edges). No new traversal.
**No schema/version impact:** this changes what the resolver produces, not how it is stored. Existing indexes need a re-analyze to show the new edges — matching the note PR #2810 carried for the same reason.
## 10. Files Expected to Change
| File | Symbols | Reason |
| ---- | ------- | ------ |
| `gitnexus/src/core/ingestion/scope-resolution/scope/namespace-targets.ts` | `collectNamespaceTargets` | Add the dotted-access-path key (§6.1) and update the contract note |
| `gitnexus/src/core/ingestion/scope-resolution/passes/compound-receiver.ts` | `isNamespaceNameShadowed` | Shadow-test the root segment (§6.2) |
| `gitnexus/test/integration/resolvers/python.test.ts` | new `describe` block | Issue acceptance row + controls + regression rows |
| `gitnexus/test/fixtures/lang-resolution/python-dotted-namespace-import/**` | — | New fixture (§8) |
| `gitnexus/test/integration/resolvers/repro-2826-python-dotted-import.test.ts` | — | Delete; superseded by the fixture-backed tests |
| `gitnexus/bench/receiver-resolution/baseline.json` | — | Regenerate once, final step, only if `--check` moves |
## 11. Reusable Implementation Context
```yaml
implementation_context:
task_summary: >
Python `import pkg.db` + `pkg.db.session_scope()` emits no CALLS edge (#2826).
Root cause: collectNamespaceTargets keys its map only on ImportEdge.localName
('pkg'), while the receiver text is the full dotted path ('pkg.db'). Fix by
additionally keying on ImportEdge.targetExportedName when it is dotted and
rooted at localName — a predicate no other provider satisfies — plus a
root-segment fix to the shadow guard the new key first exposes.
acceptance_criteria:
- 'CALLS edge uses_dotted -> pkg/db.py:session_scope with reason import-resolved'
- 'The three sibling spellings (from-import, alias, from-module-attr) still resolve'
- 'import pkg.a + import pkg.b in one file do not cross-resolve'
- 'A local binding shadowing the package root suppresses the namespace interpretation'
- 'No shared file under gitnexus/src/core/ingestion/ names a language (AGENTS.md §42)'
evidence_provenance:
schema_version: 2
head_commit: 'b2cd1c2ad637657125248c0dd2046de71ceea965'
generated_plan_path: 'docs/plans/2026-08-04-gitnexus-plan-python-dotted-namespace-receiver.md'
global_dirty_digest:
algorithm: 'sha256'
canonicalization: 'gitnexus-evidence-provenance-v2 NUL-framed UTF-8 records'
value: '0912a3ee3219cb75c82aefbf9f010e8dbe313150d6553768fd55d22af87a135c'
cited_path_manifest:
- path: '.github/workflows/ci-tests.yml'
object_kind: { head: regular, index: regular, worktree: regular, untracked: absent }
state: 'clean'
rename_from: null
rename_to: null
head_digest: 'sha256:0f1fba71be1e2b026d1ca2d35934ffe197b26bd4d31d5e5025d1e797e89754ff'
index_digest: 'sha256:0f1fba71be1e2b026d1ca2d35934ffe197b26bd4d31d5e5025d1e797e89754ff'
worktree_digest: 'sha256:0f1fba71be1e2b026d1ca2d35934ffe197b26bd4d31d5e5025d1e797e89754ff'
untracked_digest: 'absent'
- path: 'AGENTS.md'
object_kind: { head: regular, index: regular, worktree: regular, untracked: absent }
state: 'clean'
rename_from: null
rename_to: null
head_digest: 'sha256:797b9d58a9c3dbed5af048904b3d3ba55ba6a2256a442fd15d35eb8b568cd1dd'
index_digest: 'sha256:797b9d58a9c3dbed5af048904b3d3ba55ba6a2256a442fd15d35eb8b568cd1dd'
worktree_digest: 'sha256:797b9d58a9c3dbed5af048904b3d3ba55ba6a2256a442fd15d35eb8b568cd1dd'
untracked_digest: 'absent'
- path: 'gitnexus-shared/src/scope-resolution/finalize-algorithm.ts'
object_kind: { head: regular, index: regular, worktree: regular, untracked: absent }
state: 'clean'
rename_from: null
rename_to: null
head_digest: 'sha256:9c3656484d8b5bd49394918446ab91c73db722e3fe2314fc08c9c284541c415b'
index_digest: 'sha256:9c3656484d8b5bd49394918446ab91c73db722e3fe2314fc08c9c284541c415b'
worktree_digest: 'sha256:9c3656484d8b5bd49394918446ab91c73db722e3fe2314fc08c9c284541c415b'
untracked_digest: 'absent'
- path: 'gitnexus-shared/src/scope-resolution/types.ts'
object_kind: { head: regular, index: regular, worktree: regular, untracked: absent }
state: 'clean'
rename_from: null
rename_to: null
head_digest: 'sha256:d9b0e9e0d47c10a71392ad8d0de31b08327c6268915488f1153c04cdc39fbdfc'
index_digest: 'sha256:d9b0e9e0d47c10a71392ad8d0de31b08327c6268915488f1153c04cdc39fbdfc'
worktree_digest: 'sha256:d9b0e9e0d47c10a71392ad8d0de31b08327c6268915488f1153c04cdc39fbdfc'
untracked_digest: 'absent'
- path: 'gitnexus/src/core/ingestion/languages/python/import-decomposer.ts'
object_kind: { head: regular, index: regular, worktree: regular, untracked: absent }
state: 'clean'
rename_from: null
rename_to: null
head_digest: 'sha256:97e28381e7d3f6040e5368d043d086ab2d3df24aad5e2bcb0c3da866a455a23e'
index_digest: 'sha256:97e28381e7d3f6040e5368d043d086ab2d3df24aad5e2bcb0c3da866a455a23e'
worktree_digest: 'sha256:97e28381e7d3f6040e5368d043d086ab2d3df24aad5e2bcb0c3da866a455a23e'
untracked_digest: 'absent'
- path: 'gitnexus/src/core/ingestion/languages/python/interpret.ts'
object_kind: { head: regular, index: regular, worktree: regular, untracked: absent }
state: 'clean'
rename_from: null
rename_to: null
head_digest: 'sha256:65ca96b207b89a86f44772f8f8ff8030acf06774214ddee67ef031db3d770419'
index_digest: 'sha256:65ca96b207b89a86f44772f8f8ff8030acf06774214ddee67ef031db3d770419'
worktree_digest: 'sha256:65ca96b207b89a86f44772f8f8ff8030acf06774214ddee67ef031db3d770419'
untracked_digest: 'absent'
- path: 'gitnexus/src/core/ingestion/languages/python/query.ts'
object_kind: { head: regular, index: regular, worktree: regular, untracked: absent }
state: 'clean'
rename_from: null
rename_to: null
head_digest: 'sha256:f9e145114aba978e34525c1ccb553ba37feea4152f882dc0e105dc8b21230d78'
index_digest: 'sha256:f9e145114aba978e34525c1ccb553ba37feea4152f882dc0e105dc8b21230d78'
worktree_digest: 'sha256:f9e145114aba978e34525c1ccb553ba37feea4152f882dc0e105dc8b21230d78'
untracked_digest: 'absent'
- path: 'gitnexus/src/core/ingestion/scope-extractor.ts'
object_kind: { head: regular, index: regular, worktree: regular, untracked: absent }
state: 'clean'
rename_from: null
rename_to: null
head_digest: 'sha256:34089a212075f16d8c270240c64985b0a666864547ed414449a59747e4922d80'
index_digest: 'sha256:34089a212075f16d8c270240c64985b0a666864547ed414449a59747e4922d80'
worktree_digest: 'sha256:34089a212075f16d8c270240c64985b0a666864547ed414449a59747e4922d80'
untracked_digest: 'absent'
- path: 'gitnexus/src/core/ingestion/scope-resolution/passes/compound-receiver.ts'
object_kind: { head: regular, index: regular, worktree: regular, untracked: absent }
state: 'clean'
rename_from: null
rename_to: null
head_digest: 'sha256:88a083a625449187fe770e992c580ec84d70ddb9f395f54949c1b85a29838f97'
index_digest: 'sha256:88a083a625449187fe770e992c580ec84d70ddb9f395f54949c1b85a29838f97'
worktree_digest: 'sha256:88a083a625449187fe770e992c580ec84d70ddb9f395f54949c1b85a29838f97'
untracked_digest: 'absent'
- path: 'gitnexus/src/core/ingestion/scope-resolution/passes/receiver-bound-calls.ts'
object_kind: { head: regular, index: regular, worktree: regular, untracked: absent }
state: 'clean'
rename_from: null
rename_to: null
head_digest: 'sha256:1873a19be4235b6882aab63422a0bc632192ac30407e60e7d5648aa70e5759c3'
index_digest: 'sha256:1873a19be4235b6882aab63422a0bc632192ac30407e60e7d5648aa70e5759c3'
worktree_digest: 'sha256:1873a19be4235b6882aab63422a0bc632192ac30407e60e7d5648aa70e5759c3'
untracked_digest: 'absent'
- path: 'gitnexus/src/core/ingestion/scope-resolution/scope/namespace-targets.ts'
object_kind: { head: regular, index: regular, worktree: regular, untracked: absent }
state: 'clean'
rename_from: null
rename_to: null
head_digest: 'sha256:54062a70276ec1761a94b6548499d265c91fd3b648422a8567a21e06d800e09d'
index_digest: 'sha256:54062a70276ec1761a94b6548499d265c91fd3b648422a8567a21e06d800e09d'
worktree_digest: 'sha256:54062a70276ec1761a94b6548499d265c91fd3b648422a8567a21e06d800e09d'
untracked_digest: 'absent'
- path: 'gitnexus/test/integration/resolvers/python.test.ts'
object_kind: { head: regular, index: regular, worktree: regular, untracked: absent }
state: 'clean'
rename_from: null
rename_to: null
head_digest: 'sha256:4c0f55a923f51d736476b5bcb276d293637d90fecc10ab5e08624a4b541fe999'
index_digest: 'sha256:4c0f55a923f51d736476b5bcb276d293637d90fecc10ab5e08624a4b541fe999'
worktree_digest: 'sha256:4c0f55a923f51d736476b5bcb276d293637d90fecc10ab5e08624a4b541fe999'
untracked_digest: 'absent'
- path: 'gitnexus/test/integration/resolvers/repro-2826-python-dotted-import.test.ts'
object_kind: { head: absent, index: absent, worktree: absent, untracked: regular }
state: 'untracked'
rename_from: null
rename_to: null
head_digest: 'absent'
index_digest: 'absent'
worktree_digest: 'absent'
untracked_digest: 'sha256:6fe3a74a69db12a1a0aeceef2b32eb0c04d5e4880fc93b7b840348118e70078c'
primary_symbols:
- symbol: 'collectNamespaceTargets'
file: 'gitnexus/src/core/ingestion/scope-resolution/scope/namespace-targets.ts'
lines: '39-57'
role: 'The defect site — builds the receiver-name → target-file map keyed only on localName'
- symbol: 'interpretPythonImport'
file: 'gitnexus/src/core/ingestion/languages/python/interpret.ts'
lines: '33-42'
role: 'Splits `import a.b` into localName "a" / importedName "a.b"; source of both halves'
- symbol: 'emitReceiverBoundCalls'
file: 'gitnexus/src/core/ingestion/scope-resolution/passes/receiver-bound-calls.ts'
lines: '404-421, 546-655, 831-848'
role: 'Case 0 declines on a module receiver and falls through; Case 1 does the failing map lookup'
- symbol: 'isNamespaceNameShadowed'
file: 'gitnexus/src/core/ingestion/scope-resolution/passes/compound-receiver.ts'
lines: '152-183'
role: 'Shadow guard that must test the root segment once dotted keys exist'
- symbol: 'finalizeImportEdges'
file: 'gitnexus-shared/src/scope-resolution/finalize-algorithm.ts'
lines: '398-406, 434-447'
role: 'Carries importedName onto ImportEdge.targetExportedName for namespace edges'
related_symbols:
- symbol: 'resolveQualifiedReceiverMember'
relationship: 'ScopeResolver hook, C++-only implementer'
relevance: 'Case 1.5 — deliberately NOT the fix path; implementing it for Python would duplicate what Case 1 already does'
- symbol: 'resolveConstructionExpressionClass'
relationship: 'consumes namespaceTargets by parameter'
relevance: 'Second consumer of the map; gains correct pkg.db.Model() resolution'
- symbol: 'resolveCompoundReceiverClass'
relationship: 'consumes namespaceTargets by parameter (compound-receiver.ts:759-766)'
relevance: 'Third consumer; has() now true for dotted namespaces'
- symbol: 'isNamespaceImport'
relationship: 'finalize hook added by #2770'
relevance: 'Prior art — how the from-pkg-import-db sibling was made to resolve'
- symbol: 'build'
relationship: 'test-of collectNamespaceTargets'
relevance: 'test/unit/scope-resolution/python/python-module-namespace-construction.test.ts — re-run after the change'
execution_path:
- 'splitImportStatement emits one match per imported name; @import.source = full dotted_name text'
- 'interpretPythonImport plain arm → ParsedImport{kind:namespace, localName:first-segment, importedName:full-dotted}'
- 'finalizeImportEdges → ImportEdge{localName, targetExportedName=importedName, targetFile, kind:namespace}'
- 'collectNamespaceTargets builds Map keyed on localName only ← DEFECT'
- 'scope-extractor extractExplicitReceiver takes raw text of the attribute object → "pkg.db"'
- 'emitReceiverBoundCalls Case 0 declines (module, not class), falls through without marking handled'
- 'Case 1 map lookup on "pkg.db" misses; Case 1.5 skipped (no Python hook); site drops silently'
pdg_constraints: [] # index has zero CDG rows; no --pdg layer to slice
architectural_patterns:
- pattern: 'Provider reclassification at finalize instead of shared-code special-casing'
example_location: 'gitnexus-shared/src/scope-resolution/finalize-algorithm.ts:99-107 (isNamespaceImport, #2770)'
usage_guidance: 'Considered and rejected here: the information needed is already on the finalized edge, so no new hook is warranted'
- pattern: 'Verified namespace is authoritative — do not fall through to workspace-wide simple-name heuristics'
example_location: 'gitnexus/src/core/ingestion/scope-resolution/passes/compound-receiver.ts:245-259'
usage_guidance: 'Because that branch declines rather than guessing, a wrong key costs a lost edge, not a wrong one — but the shadow guard must be right'
- pattern: 'Fixture + assertions in test/integration/resolvers/python.test.ts'
example_location: 'gitnexus/test/integration/resolvers/python.test.ts:562-600 (vendored-django guard)'
usage_guidance: 'mkdtempSync + writeFixtureRepo + afterAll rmSync; assert both presence of the right edge and absence of the wrong one'
files_to_modify:
- file: 'gitnexus/src/core/ingestion/scope-resolution/scope/namespace-targets.ts'
symbols: ['collectNamespaceTargets']
intended_change: 'Additionally key the map on edge.targetExportedName when it contains a dot and its first segment equals edge.localName; keep the existing localName key unchanged; update the header contract note'
- file: 'gitnexus/src/core/ingestion/scope-resolution/passes/compound-receiver.ts'
symbols: ['isNamespaceNameShadowed']
intended_change: 'Shadow-test the first dot-separated segment of namespaceName (no-op for single-segment names)'
- file: 'gitnexus/test/integration/resolvers/python.test.ts'
symbols: []
intended_change: 'Add a describe block covering the six §8 scenarios'
- file: 'gitnexus/test/fixtures/lang-resolution/python-dotted-namespace-import/'
symbols: []
intended_change: 'New fixture per the §8 table'
- file: 'gitnexus/test/integration/resolvers/repro-2826-python-dotted-import.test.ts'
symbols: []
intended_change: 'Delete — superseded by the fixture-backed tests'
tests:
- file: 'gitnexus/test/integration/resolvers/python.test.ts'
scenarios:
- 'import pkg.db + pkg.db.session_scope() → run pipeline → CALLS uses_dotted → pkg/db.py:session_scope, reason import-resolved'
- 'three sibling spellings in the same repo → run pipeline → all still resolve to pkg/db.py:session_scope (control)'
- 'import pkg.db AND import pkg.cache in one file, both defining session_scope → each call resolves only to its own module; assert the crossed pair is ABSENT'
- 'import pkg.sub.deep + pkg.sub.deep.f() → 3-segment receiver resolves'
- 'module-level import pkg.db shadowed by a function-local pkg = Decoy() → NO edge to pkg/db.py'
- 'existing TypeScript/C#/Go namespace-import resolver tests → unchanged green (no key minted for them)'
- file: 'gitnexus/test/unit/scope-resolution/python/python-module-namespace-construction.test.ts'
scenarios:
- 'Re-run unchanged; extend only if it enumerates map keys exhaustively'
verification_commands:
- 'cd gitnexus && GITNEXUS_WORKER_READY_TIMEOUT_MS=60000 npm run test:integration -- test/integration/resolvers/python.test.ts'
- 'cd gitnexus && GITNEXUS_WORKER_READY_TIMEOUT_MS=60000 npm run test:integration -- test/integration/resolvers'
- 'cd gitnexus && npm run test:unit -- test/unit/scope-resolution'
- 'cd gitnexus && npx tsc --noEmit'
- 'cd gitnexus-shared && npx tsc --noEmit'
- 'cd gitnexus && node --import tsx bench/receiver-resolution/measure.mjs --check'
- 'cd gitnexus && node --import tsx bench/python-scope/measure.mjs --check'
- 'cd gitnexus && node --import tsx bench/python-scope/import-target-fingerprint.mjs --check'
- 'cd gitnexus && node --import tsx bench/scope-capture/measure.mjs --check'
risks:
- 'New map keys reach three consumers, two of them by parameter rather than by call — the graph d=1 list alone under-reports them'
- 'compound-receiver treats a verified namespace as authoritative and declines instead of falling through, so a bad key loses edges silently'
- 'bench/receiver-resolution/baseline.json is expected to move; regenerate ONCE in the final step'
- 'Default 5000 ms worker-ready timeout crash-loops on this host; export GITNEXUS_WORKER_READY_TIMEOUT_MS=60000'
assumptions:
- 'Every non-Python provider fails the dotted-rooted-at-localName predicate. CHECK: grep "kind: .namespace." across gitnexus/src/core/ingestion/languages/*/interpret.ts and confirm localName is never the first segment of a dotted importedName. Verified at b2cd1c2ad for typescript, csharp, go, rust, ruby.'
- 'python.test.ts is no longer gated behind REGISTRY_PRIMARY_PYTHON. CHECK: grep REGISTRY_PRIMARY in that file — zero hits at b2cd1c2ad, so it runs unconditionally.'
- 'The GitNexus index is 13 commits behind but byte-identical on every cited path. CHECK: git rev-parse 1ef6447e:<path> vs b2cd1c2ad:<path>.'
open_questions:
- 'Should the misleading localName-only key for a dotted import (pkg → pkg/db.py) be removed? It can produce a false positive today: pkg.helper() resolves into pkg/db.py if db.py happens to define helper. Deferred — a separate behaviour change needing its own regression pass.'
- 'Python`s `import a.b.c` also makes `a.b` reachable. The proposed predicate keys only the exact imported path, so `a.b.f()` under `import a.b.c` alone stays unresolved. Deferred as a narrower follow-up.'
- 'C# `using System.Collections.Generic` + `System.Collections.Generic.List` is the same class of gap and is deliberately NOT addressed here (its localName is the last segment, so the predicate declines). Worth its own issue.'
avoid:
- 'Do not repeat full repository discovery'
- 'Do not replace established patterns without evidence'
- 'Do not implement resolveQualifiedReceiverMember for Python — Case 1 already does this job; a second path would double-resolve'
- 'Do not change ImportEdge.localName for dotted imports (interpret.ts:38) — it is the deliberate `import a.b.c exposes a` semantics and other consumers depend on it'
- 'Do not name a language in gitnexus/src/core/ingestion/ shared code (AGENTS.md §42)'
- 'Do not regenerate bench baselines per step — only once, in the final step'
- 'Do not weaken an existing test to accommodate the new keys; extend it instead'
```
## 12. Assumptions and Open Questions
**Assumptions** (each re-checkable cheaply by the executor):
1. Every non-Python namespace-emitting provider fails the `dotted && first segment === localName` predicate. Verified at `b2cd1c2ad` for TypeScript, C#, Go, Rust and Ruby by reading each `interpret.ts`; JavaScript, Java and PHP emit no `kind: 'namespace'` import there. **Re-check:** grep `kind: 'namespace'` across `gitnexus/src/core/ingestion/languages/*/interpret.ts`.
2. `python.test.ts` runs unconditionally — no `REGISTRY_PRIMARY_PYTHON` gate remains at the pinned commit (zero grep hits). An older parity-leg convention no longer applies.
3. The index's 13-commit lag is harmless here because every cited path is byte-identical at the index commit and the pinned commit.
**Open questions / explicitly deferred:**
- **The bogus first-segment key.** For `import pkg.db`, the map still holds `'pkg' → ['pkg/db.py']`, so `pkg.helper()` would resolve into `pkg/db.py` if that file happens to define `helper` — a pre-existing false positive this plan does **not** fix. Removing it is a separate behaviour change with its own regression surface (`python-multi-segment-ancestor-import`, `python-bare-import`). Worth pinning the current behaviour in a test so it is visible rather than silent.
- **`import a.b.c` also binds `a.b`.** Python makes intermediate packages reachable; the proposed predicate keys only the exact imported path, so `a.b.f()` under `import a.b.c` alone stays unresolved. Narrower follow-up.
- **C# has the mirror-image gap.** `using System.Collections.Generic` + `System.Collections.Generic.List` fails the predicate because C# sets `localName` to the *last* segment. Deliberately out of scope; deserves its own issue.
- **Construction coverage.** §8 does not currently include a `pkg.db.Model()` row. Add one if the fixture cost is trivial — that path (`compound-receiver.ts:245-260`) changes behaviour and is otherwise untested by this plan.
## 13. Definition of Done
1. `CALLS` edge `uses_dotted` → `pkg/db.py:session_scope` (`reason: 'import-resolved'`) is emitted, asserted by a fixture-backed test in `python.test.ts`.
2. All three sibling control rows still resolve in the same run.
3. `import pkg.a` + `import pkg.b` in one file resolve only to their own modules; the crossed pair is asserted **absent**.
4. A 3-segment receiver resolves; a package root shadowed by a local binding does **not**.
5. The scratch `repro-2826-python-dotted-import.test.ts` is deleted.
6. No file under `gitnexus/src/core/ingestion/` names a language.
7. `npm run test:integration -- test/integration/resolvers` and `npm run test:unit -- test/unit/scope-resolution` pass; `tsc --noEmit` clean in both packages.
8. Every bench `--check` in §8 passes, with `bench/receiver-resolution/baseline.json` regenerated exactly once in the final commit if and only if it moved — and any movement in `python-scope`/`scope-capture` investigated rather than regenerated.

312
docs/plans/2026-08-28-gitnexus-plan-impact-file-risk.md Normal file
View file

@ -0,0 +1,312 @@
# GitNexus Engineering Plan
> Task: Fix #3075 — File `impact` risk is not comparable to Function/Method risk.
> Evidence verified at commit `6bff33d14cbfe1e7b4f04bca51507e9f64ef579c` (`feat/kotlin-const-resolver`); GitNexus index 129 commits behind, refresh skipped: full-repo `--index-only --pdg` rebuild is impractical this session. Scorer and schema claims are `[verified]` from source; live inversion numbers are `[graph]` on the stale index.
## 1. Objective
Make File vs symbol `impact.risk` honest for consumers: either they can tell the scales differ, or they can compare on a shared two-axis score. Do **not** DEFINES-bridge processes/modules onto File targets (issue reporter sampled 8/10 one-importer files jumping to HIGH/CRITICAL). Do **not** retune Function HIGH/CRITICAL thresholds (agent warn-before-edit).
Acceptance:
- A File with a wider blast radius than a Function in the same file no longer looks “safer” when a consumer only reads `risk`, **or** the result states that `risk` is not comparable across kinds and offers `riskSharedAxes` for comparison.
- File targets still cannot trip HIGH/CRITICAL via `processes_affected` / `modules_affected` unless those axes become real in the index (they are not today).
- Existing Function/Method labels under the current four-axis ladder stay the same for the same inputs.
- MCP `riskNote` remains UNKNOWN-only (`tools.ts` contract).
## 2. Current Behaviour
Callgraph `impact` ends in `LocalBackend._runImpactBFS` (`gitnexus/src/mcp/local/local-backend.ts`). After BFS it enriches impacted ids with `STEP_IN_PROCESS` and `MEMBER_OF`, then scores:
```7720:7738:gitnexus/src/mcp/local/local-backend.ts
} else if (
directCount >= 30 ||
processCount >= 5 ||
moduleCount >= 5 ||
impacted.length >= 200
) {
risk = 'CRITICAL';
} else if (
directCount >= 15 ||
processCount >= 3 ||
moduleCount >= 3 ||
impacted.length >= 100
) {
risk = 'HIGH';
} else if (directCount >= 5 || impacted.length >= 30) {
risk = 'MEDIUM';
} else {
risk = 'LOW';
}
```
Empty upstream → `UNKNOWN` + `riskNote`. Downstream empty stays LOW. `skipEnrichment` (ambiguous probes) already scores on direct+total only. PDG mode forces `risk: UNKNOWN` (`composeUnifiedPdgImpactResult`) — out of scope.
File BFS walk is mostly File←IMPORTS File. Enrichment queries those File ids. Processes are CALLS traces (`process-processor.ts`); communities admit only Function/Class/Method/Interface (`isCommunitySymbol` in `community-processor.ts:412-416`). File is not in that set. `enrichCandidateLabels` UNION also **omits File**, so File `target.type` is often `""`; detect File via `id` prefix `File:`.
Web Graph RAG (`gitnexus-web/src/core/llm/tools.ts` ~1331–1346) duplicates the same ladder.
## 3. Relevant Architecture
| Layer | Role |
|---|---|
| Index | File never sources `STEP_IN_PROCESS` / `MEMBER_OF` by construction |
| MCP `_runImpactBFS` | Blast radius + four-axis `risk` |
| Ambiguous probes | `skipEnrichment` → 2-axis `risk` already |
| `mergeRisk` | Group overlay; monotone in crossings; does not know target kind |
| CLI `formatImpactResult` | Prints counts; **does not print `risk`** on the resolved callgraph path; JSON `impactCommand` still ships `risk` |
| `ai-context.ts` / `tools.ts` | Agent contract: warn on HIGH/CRITICAL; `riskNote` UNKNOWN-only |
| Web LLM `impact` | Same formula, prose `RISK:` line |
Modules: Local (MCP), Cli (format/docs), Group (`mergeRisk`), gitnexus-web LLM tools. Shared package `gitnexus-shared` is already a dependency of both CLI and web.
## 4. GitNexus Findings
- Primary: `_runImpactBFS` — d=1 `[graph]` `impact(target:_runImpactBFS, maxDepth:1, includeTests:true)`: `_impactImpl`, `impactByUid`. Production chain `[verified]`: `impact` → `_impactImpl` → `_runImpactBFS`; `impactByUid` skips per-symbol process lists but **not** aggregation (`skipPerSymbolEnrichment` only).
- `LocalBackend.impact` d=1 `[graph]` `context`: `callTool`.
- Duplicate scorer `[verified]` grep: `gitnexus-web/src/core/llm/tools.ts`.
- `mergeRisk` `[verified]` callers in `src/`: only `runGroupImpact` (`cross-impact.ts:907`). Graph d=1 listed a test File (`impact-pdg-shape.test.ts`) and missed `runGroupImpact` — trust source.
- Schema `[verified]`: `isCommunitySymbol` excludes File; `schema.ts` documents MEMBER_OF as Function/Class/Method/Interface only.
- Live inversion `[graph]` stale index, `impact summaryOnly` on GitNexus:
| target | kind | impacted | direct | processes | modules | risk |
|---|---|---|---|---|---|---|
| `lbug-config.ts` | File | 54 | 12 | 0 | 0 | MEDIUM |
| `openLbugConnection` | Function | 16 | 9 | 3 | 2 | HIGH |
| `local-backend.ts` | File | 12 | 10 | 0 | 0 | MEDIUM |
| `refreshRepos` | Method | 50 | 5 | 4 | 7 | CRITICAL |
- Clusters/processes resources `[graph]`: Local/Cli/Group sit in the impact path; process traces are function-stepped, not File-stepped.
- Related tests `[verified]`: `test/unit/impact-pagination.test.ts` (CRITICAL from `direct=400`); `test/integration/impact-zero-caller-risk.test.ts` (`withTestLbugDB` seed — pattern to extend); `test/unit/eval-formatters.test.ts` (`formatImpactResult`); group `mergeRisk` tests.
## 5. Statement-Level PDG Findings
PDG unavailable (`pdg_query` on `_runImpactBFS`: “no PDG layer”). Recommend `node .gitnexus/run.cjs analyze --index-only --pdg` before any future statement-slice work. Control flow of the scorer is a straight if/else after enrichment; no hidden guards. `skipEnrichment` is the only branch that structurally zeros process/module counts besides File ids.
## 6. Proposed Changes
### 6.1 Extract `scoreImpactRisk` — `gitnexus-shared/src/impact-risk.ts` (new)
- **Responsibility:** Pure function: `{ direction, directCount, processCount, moduleCount, impactedCount, unusedAxes }` → `{ risk, riskSharedAxes, riskScale }`.
- **Behaviour:** Existing UNKNOWN/CRITICAL/HIGH/MEDIUM/LOW thresholds unchanged when `unusedAxes` is empty. `riskSharedAxes` always scores as if `processCount=0` and `moduleCount=0` (UNKNOWN rule still applies). `riskScale.comparableAcrossKinds` is false iff `unusedAxes` is non-empty. `riskScale.unusedAxes` lists `{ axis, reason }`.
- **Constraints:** Zero deps. Export from `gitnexus-shared/src/index.ts`. Do not put MCP types here.
- **File detection:** caller passes unused axes; helper does not parse UIDs.
### 6.2 Wire MCP — `_runImpactBFS` in `local-backend.ts`
- After computing `processCount`/`moduleCount`, set `unusedAxes`:
- target `id` starts with `File:` **or** `symType === 'File'` → processes + modules, reason `file-nodes-have-no-process-or-community-membership`;
- `skipEnrichment` → same axes, reason `enrichment-skipped` (ambiguous probes).
- Replace inline ladder with `scoreImpactRisk`.
- Spread `riskScale` and `riskSharedAxes` on the result next to `risk`. Do **not** set `riskNote` for File.
- Ambiguous candidate summaries: forward the new fields (probes already skip enrichment).
- `target.type` for File: if still `""`, prefer `'File'` when `id` starts with `File:` (display-only; helps CLI).
### 6.3 Web duplicate — `gitnexus-web/src/core/llm/tools.ts`
- Import `scoreImpactRisk` from `gitnexus-shared`. Print `RISK:` from `risk`; if `!comparableAcrossKinds`, one extra line: not comparable to Function risk; shared-axes label is `riskSharedAxes`.
### 6.4 Agent/MCP contract copy
- `gitnexus/src/mcp/tools.ts` impact description: document `riskScale` / `riskSharedAxes`; keep `riskNote` UNKNOWN-only; say File `risk` is not comparable to symbol `risk`.
- `gitnexus/src/cli/ai-context.ts`: HIGH/CRITICAL warning still applies; add: do not rank a File `MEDIUM` below a contained Function `HIGH` without `riskSharedAxes`.
- `formatImpactResult`: on resolved callgraph results with `risk`, print `Risk: {risk}` and, when incomparable, `Shared-axes risk: {riskSharedAxes} (File/process axes unused)`.
### 6.5 Explicitly not changing
- DEFINES-bridge, community/process indexers, `mergeRisk` formula, PDG `UNKNOWN`, `detectChanges` `risk_level`, Function thresholds.
## 7. Implementation Sequence
1. Add `gitnexus-shared` helper + unit table (issue-shaped inputs + UNKNOWN + skipEnrichment). Shared package tests if present; otherwise `gitnexus/test/unit/impact-risk.test.ts` importing the helper.
2. Switch `_runImpactBFS` + candidate probe payload. Tree still coherent: old `risk` values identical for Function fixtures.
3. Integration seed in `impact-zero-caller-risk.test.ts` **or** new `impact-file-risk-scale.test.ts`: File with ≥5 File IMPORTS (MEDIUM on direct) vs Function with 3 process-member callers (HIGH); assert File `riskScale.comparableAcrossKinds === false`, Function true, File `riskSharedAxes === risk`, Function `riskSharedAxes` is LOW/MEDIUM while `risk` is HIGH.
4. CLI formatter + `eval-formatters.test.ts`.
5. `tools.ts` + `ai-context.ts` wording.
6. Web import + a unit assertion on the printed RISK block if a test already covers that tool.
7. `npx tsc --noEmit` in `gitnexus/` and `gitnexus-web/`; `cd gitnexus && npm run test:unit -- test/unit/impact-risk.test.ts test/unit/eval-formatters.test.ts`; integration file from step 3.
## 8. Test Strategy
| File | Scenarios |
|---|---|
| `gitnexus/test/unit/impact-risk.test.ts` (new) | Issue table: File(25,13,0,0)→MEDIUM; Function(15,2,4,2)→HIGH; shared-axes File MEDIUM vs Function LOW; empty upstream UNKNOWN; downstream empty LOW; skipEnrichment unused axes; CRITICAL via direct≥30 still works with unused process axes |
| `gitnexus/test/integration/impact-file-risk-scale.test.ts` (new) | `withTestLbugDB` seed: `File:src/crypto.ts` ← 13 File IMPORTS, no File STEP_IN_PROCESS; `getEncryptionKey` with 2 CALLS from functions that have STEP_IN_PROCESS to 4 distinct Process nodes — reproduce inversion; assert new fields |
| `gitnexus/test/integration/impact-zero-caller-risk.test.ts` | Unchanged UNKNOWN/`riskNote`; candidates may grow `riskScale` — assert still present only when UNKNOWN for `riskNote` |
| `gitnexus/test/unit/impact-pagination.test.ts` | Hub CRITICAL unchanged |
| `gitnexus/test/unit/eval-formatters.test.ts` | Resolved result prints Risk + shared-axes line for File-shaped `riskScale` |
| Web | Only if an existing Graph RAG impact test snapshots `RISK:` |
Commands (exist in `gitnexus/package.json`): `npm run test:unit`, `npm test` (full vitest), `npx tsc --noEmit`. Web: `npm test`, `npx tsc -b --noEmit`. Integration needs `pretest:integration` / `npm run test:integration` (runs `scripts/build.js`).
## 9. Risk and Impact Analysis
Direct dependents of `_runImpactBFS` `[graph]`: `_impactImpl`, `impactByUid`. `_impactImpl` is the only d=1 of `impact` besides the method’s own class. Any JSON consumer of `impact` (MCP, CLI `output(result)`, group local leg) sees additive fields — compatible if they ignore unknowns.
- **HIGH workflow:** Function HIGH/CRITICAL unchanged. File still cannot reach HIGH via processes; a File with `direct≥15` or `total≥100` still can. Agents that compare File MEDIUM vs Function HIGH must start using `riskSharedAxes` or `riskScale`.
- **Ambiguous `maxRisk`:** probes skip enrichment, so File vs Function candidates are already 2-axis there — inversion is weaker on that path.
- **Group `mergeRisk`:** still compares incomparable File local `risk` to crossing count. Do not retune this PR; if a group File target is common, follow-up.
- **Web:** browser bundle picks up `gitnexus-shared` export — confirm `gitnexus-shared` build/exports include the new file.
- **Performance:** none (pure arithmetic after existing enrichment).
- **Ladybug empty labels:** File detection must not rely on `symType` alone.
## 10. Files Expected to Change
| File | Symbols | Reason |
|---|---|---|
| `gitnexus-shared/src/impact-risk.ts` | `scoreImpactRisk` | New shared scorer |
| `gitnexus-shared/src/index.ts` | exports | Public helper |
| `gitnexus/src/mcp/local/local-backend.ts` | `_runImpactBFS`, ambiguous candidate map | Wire scorer + File unused axes |
| `gitnexus/src/mcp/tools.ts` | `impact` description | Contract |
| `gitnexus/src/cli/ai-context.ts` | generated Always Do | Agent warning |
| `gitnexus/src/cli/eval-server.ts` | `formatImpactResult` | Print scale |
| `gitnexus-web/src/core/llm/tools.ts` | web `impact` | Same formula |
| `gitnexus/test/unit/impact-risk.test.ts` | — | Table tests |
| `gitnexus/test/integration/impact-file-risk-scale.test.ts` | — | Seeded inversion |
| `gitnexus/test/unit/eval-formatters.test.ts` | `formatImpactResult` | Formatter |
## 11. Reusable Implementation Context
```yaml
implementation_context:
task_summary: "Fix #3075: File impact.risk is a 2-axis score silently labelled on a 4-axis scale. Extract scoreImpactRisk; mark File/skipEnrichment axes unused; add riskScale + riskSharedAxes; do not DEFINES-bridge or retune Function thresholds."
acceptance_criteria:
- "File vs Function comparison is either labelled incomparable (riskScale) or done via riskSharedAxes"
- "Function/Method risk for identical four-axis inputs unchanged"
- "riskNote still UNKNOWN-only"
- "Integration seed reproduces crypto.ts-style inversion and asserts the new fields"
primary_symbols:
- symbol: "_runImpactBFS"
file: "gitnexus/src/mcp/local/local-backend.ts"
lines: "6991-7888"
role: "BFS + enrichment + inline risk ladder (replace ladder only)"
- symbol: "scoreImpactRisk"
file: "gitnexus-shared/src/impact-risk.ts"
lines: "new"
role: "Pure scorer + shared-axes + riskScale"
- symbol: "formatImpactResult"
file: "gitnexus/src/cli/eval-server.ts"
lines: "305-641"
role: "Human/LLM text surface for impact JSON"
related_symbols:
- symbol: "_impactImpl"
relationship: "CALLS"
relevance: "Resolves target, PDG vs callgraph, ambiguous skipEnrichment probes"
- symbol: "impactByUid"
relationship: "CALLS"
relevance: "Group fan-out; keep skipPerSymbolEnrichment; still run aggregation"
- symbol: "mergeRisk"
relationship: "consumes risk string"
relevance: "Do not change this PR"
- symbol: "isCommunitySymbol"
relationship: "index gate"
relevance: "Why File modules_affected is always 0"
- symbol: "composeUnifiedPdgImpactResult"
relationship: "separate path"
relevance: "PDG risk stays UNKNOWN"
execution_path:
- "impact / callTool → _impactImpl (resolve symbol, File id prefix File:)"
- "_runImpactBFS: IMPORTS-heavy walk for File; CALLS walk for Function"
- "Enrich STEP_IN_PROCESS / MEMBER_OF on impacted ids (empty for File ids)"
- "scoreImpactRisk with unusedAxes for File or skipEnrichment"
- "JSON to MCP/CLI; formatImpactResult for eval text; web LLM tools parallel path"
pdg_constraints:
- description: "No PDG layer on the planning index; scorer is post-enrichment arithmetic"
affected_statements: []
implementation_consequence: "Do not wait on PDG; do not change pdg impact risk"
architectural_patterns:
- pattern: "Additive optional JSON fields on impact (riskNote, epistemic, partial)"
example_location: "gitnexus/src/mcp/local/local-backend.ts _runImpactBFS base object ~7754"
usage_guidance: "Add riskScale/riskSharedAxes the same way; never overload riskNote"
- pattern: "withTestLbugDB CREATE seed for impact contract"
example_location: "gitnexus/test/integration/impact-zero-caller-risk.test.ts"
usage_guidance: "Seed File IMPORTS + Function CALLS + Process membership separately"
files_to_modify:
- file: "gitnexus-shared/src/impact-risk.ts"
symbols: ["scoreImpactRisk"]
intended_change: "new pure scorer"
- file: "gitnexus-shared/src/index.ts"
symbols: []
intended_change: "re-export"
- file: "gitnexus/src/mcp/local/local-backend.ts"
symbols: ["_runImpactBFS"]
intended_change: "unusedAxes + helper; File type display"
- file: "gitnexus/src/mcp/tools.ts"
symbols: []
intended_change: "document fields"
- file: "gitnexus/src/cli/ai-context.ts"
symbols: []
intended_change: "agent comparability note"
- file: "gitnexus/src/cli/eval-server.ts"
symbols: ["formatImpactResult"]
intended_change: "print risk + shared-axes when incomparable"
- file: "gitnexus-web/src/core/llm/tools.ts"
symbols: []
intended_change: "import helper; extra prose line"
tests:
- file: "gitnexus/test/unit/impact-risk.test.ts"
scenarios:
- "File(25,13,0,0)+unused process/module → risk MEDIUM, comparableAcrossKinds false, riskSharedAxes MEDIUM"
- "Function(15,2,4,2) → HIGH, riskSharedAxes LOW (direct 2, total 15)"
- "upstream impactedCount 0 → UNKNOWN both fields"
- "direct 400 → CRITICAL even with unused process axes"
- file: "gitnexus/test/integration/impact-file-risk-scale.test.ts"
scenarios:
- "Seed File crypto.ts with 13 File importers vs getEncryptionKey with process-rich callers → inversion on risk, File incomparable, Function comparable"
- file: "gitnexus/test/unit/eval-formatters.test.ts"
scenarios:
- "formatImpactResult includes Shared-axes risk when riskScale.comparableAcrossKinds is false"
verification_commands:
- "cd gitnexus && npx tsc --noEmit"
- "cd gitnexus && npm run test:unit -- test/unit/impact-risk.test.ts test/unit/eval-formatters.test.ts test/unit/impact-pagination.test.ts"
- "cd gitnexus && npm run test:integration -- test/integration/impact-file-risk-scale.test.ts test/integration/impact-zero-caller-risk.test.ts"
- "cd gitnexus-web && npx tsc -b --noEmit"
risks:
- "Consumers that only read risk still see the inversion unless they adopt riskScale/riskSharedAxes — that is the chosen (explicit-scale) fix"
- "File type often empty; must key unusedAxes off File: id prefix"
- "gitnexus-shared export must reach the web bundle"
assumptions:
- "WHAT: File nodes never gain STEP_IN_PROCESS/MEMBER_OF without an indexer change. HOW: keep isCommunitySymbol and process traces as-is; tests seed File with zero such edges"
- "WHAT: Additive JSON fields are backward compatible. HOW: existing tests that exact-match the full impact object may need to allow extra keys — grep expect(res).toEqual on impact results before landing"
- "WHAT: HEAD 6bff33d is the pin; scorer line numbers ~7720. HOW: re-read the ladder if that hunk moved"
open_questions:
- "Whether GroupImpactResult should copy riskScale from local File targets (deferred unless tests already snapshot the full group object)"
avoid:
- "Do not DEFINES-bridge File→symbol processes/modules"
- "Do not lower Function process/module HIGH/CRITICAL thresholds"
- "Do not reuse riskNote for File incomparability"
- "Do not change PDG impact risk or detectChanges risk_level"
- "Do not treat labels(n)[0] or empty target.type as proof the node is not a File"
- "Do not repeat full repository discovery"
```
## 12. Assumptions and Open Questions
**Assumptions**
- Indexer will not start attaching File→Process/Community in this change (`isCommunitySymbol` stays). `[verified]` source; `[assumed]` future indexers.
- Ignoring unknown JSON keys is safe for MCP clients; any `toEqual` goldens in-repo must be updated. `[assumed]` — grep during implement.
- Stale-index inversion (`lbug-config.ts` vs `openLbugConnection`) is illustrative; the integration seed is the regression lock. `[graph]` vs `[verified]` seed.
**Open questions**
- Group `mergeRisk` + File local risk: copy `riskScale` onto `GroupImpactResult`? Default **no** unless a test breaks.
- Class/Interface STEP_IN_PROCESS sparsity: out of scope (#3075 is File).
- Printing `risk` on CLI formatted output is new (JSON already has it). Keep the extra lines short.
**Deferred**
- Recalibrated File-only HIGH thresholds.
- Indexing File community membership.
- DEFINES-bridge after a threshold RFC.
- Related #2975 (docs vs scorer wording) except as touched by `tools.ts`.
## 13. Definition of Done
- [ ] `scoreImpactRisk` is the only callgraph ladder in MCP and web.
- [ ] File (and skipEnrichment) results include `riskScale.comparableAcrossKinds === false` and `riskSharedAxes`.
- [ ] Function four-axis HIGH/CRITICAL cases in unit tests still pass with the same labels.
- [ ] Integration seed proves wider File blast + lower `risk` than a contained Function, and `riskSharedAxes` orders them without pretending processes existed on the File.
- [ ] `riskNote` still absent unless `risk === 'UNKNOWN'`.
- [ ] `tools.ts` + `ai-context.ts` state that File `risk` is not comparable to symbol `risk`.
- [ ] `cd gitnexus && npx tsc --noEmit` and the named unit/integration commands pass; web typecheck passes.

6
eslint-rules/require-safe-parse.mjs
View file

@ -19,14 +19,14 @@
*
* False-positive suppression:
* - Skips calls whose receiver is a known non-tree-sitter library (`JSON`,
* `URL`, `marked`, `Number`).
* `URL`, `marked`, `Number`, `path`).
* - Skips calls whose first argument is a string-literal (grammar-load smoke
* tests like `_testParser.parse('service X { rpc Y (R) returns (R); }')`).
* - Skips test files (`.test.ts`/`.test.tsx`/`.spec.ts`).
* - Skips the `safe-parse.ts` helper itself.
*/
const SKIPPED_RECEIVERS = new Set(['JSON', 'URL', 'marked', 'Number', 'Math']);
const SKIPPED_RECEIVERS = new Set(['JSON', 'URL', 'marked', 'Number', 'Math', 'path']);
export default {
meta: {
@ -74,7 +74,7 @@ export default {
// Receiver-text-shape skip: anything matching well-known JS APIs that
// happen to have a `.parse(<expr>)` shape but aren't tree-sitter.
if (
/^(JSON|URL|marked|Number|Math|Date|globalThis\.JSON)\b/.test(receiverText) ||
/^(JSON|URL|marked|Number|Math|Date|path|globalThis\.JSON)\b/.test(receiverText) ||
/\bjson\.parse\b/i.test(receiverText)
) {
return;

1
eval/.gitignore vendored
View file

@ -14,3 +14,4 @@ build/
# Environment
.env
.venv/
.venv

1
eval/analysis/analyze_results.py
View file

@ -17,7 +17,6 @@ Usage:
import json
import logging
import os
import subprocess
import sys
from pathlib import Path

1
eval/bridge/mcp_bridge.py
View file

@ -14,7 +14,6 @@ import subprocess
import sys
import threading
import time
from pathlib import Path
from typing import Any
from constants import (

5
eval/pyproject.toml
View file

@ -6,7 +6,10 @@ readme = "README.md"
requires-python = ">=3.11"
dependencies = [
"mini-swe-agent>=2.0.0",
"litellm!=1.82.7,!=1.82.8,>=1.83.7",
"litellm[proxy]!=1.82.7,!=1.82.8,>=1.99.0",
"cryptography>=50.0.0",
"python-multipart>=0.0.30",
"restrictedpython>=8.3",
"datasets>=3.0.0",
"typer>=0.12.0",
"rich>=13.0.0",

2
eval/run_eval.py
View file

@ -27,12 +27,10 @@ import threading
import time
from itertools import product
from pathlib import Path
from typing import Any
import typer
import yaml
from rich.console import Console
from rich.live import Live
from rich.table import Table
from utils.errors import is_debug_enabled, log_safe_exception

75
eval/tests/bench_fixtures.py Normal file
View file

@ -0,0 +1,75 @@
"""Shared row shapes for the sweep tests.
Building the finalization tests turned up what a real scored review row must
carry: the report renders the whole review metric set, so an incomplete row
fails in string formatting rather than in the logic under test. That is a
property of the fixture, not of production - the shape lives here once so each
test does not rediscover it.
"""
from __future__ import annotations
from typing import Any
def scored_review_row(**overrides: Any) -> dict[str, Any]:
"""One admissible review cell, with zero-valued metrics written out."""
row: dict[str, Any] = {
"ok": True,
"error_kind": None,
"error_detail": None,
"resolved": True,
"review_evidence_valid": True,
"review_score": {"weighted_f1": 0.5},
"review_weighted_f1": 0.5,
"review_true_positives": 1,
"review_false_positives": 0,
"review_false_negatives": 0,
"review_precision": 0.5,
"review_recall": 0.5,
"review_f1": 0.5,
"review_weighted_precision": 0.5,
"review_weighted_recall": 0.5,
"review_blocker_recall": 1.0,
"review_severity_accuracy": 1.0,
"review_category_accuracy": 1.0,
"review_grounded_evidence": 1.0,
"review_verdict_correct": True,
"review_clean_control": True,
"review_clean_pass": True,
"transcript_missing": False,
"transcript_artifacts": [],
"num_turns": 3,
"duration_s": 1.0,
"cost_usd": 0.5,
"input_tokens": 1,
"output_tokens": 1,
"cache_creation_input_tokens": 0,
"cache_read_input_tokens": 0,
"diff_files": 0,
"diff_insertions": 0,
"diff_deletions": 0,
}
row.update(overrides)
return row
def unusable_review_row(**overrides: Any) -> dict[str, Any]:
"""A cell that ran but produced evidence nothing can be scored from."""
# Merged into one mapping rather than passed as explicit keywords beside
# **overrides: Python rejects a duplicate keyword in the call expression
# itself, so unusable_review_row(error_kind=...) raised TypeError before
# scored_review_row could apply the override this helper advertises.
return scored_review_row(
**{
"ok": False,
"resolved": False,
"review_evidence_valid": False,
"error_kind": "review-evidence-invalid",
"review_score": None,
"review_weighted_f1": None,
**overrides,
}
)

175
eval/tests/fixtures/fake_claude.py vendored Executable file
View file

@ -0,0 +1,175 @@
#!/usr/bin/env python3
"""A stand-in for the Claude Code CLI: real HTTP, real tool execution, real stream-json.
Not a mock of the harness's own code. It does what the CLI does at the two
boundaries the harness depends on - it calls ANTHROPIC_BASE_URL for a turn, it
EXECUTES the tool blocks that come back, and it prints the stream-json event
sequence the parent parses. Only Write really executes - it is what produces the
review artifact, so the artifact path has to be genuine end to end. Skill is
MODELLED: it validates the request and returns a synthetic result, because the
parent's evidence gate keys on the request/result pair rather than on a skill
having loaded, and a fixture cannot load a real one. Bash is stubbed outright:
arbitrary shell from a scripted reply buys no fidelity for the paths this
exercises and plenty of ways to damage the host. Everything between
those boundaries (the sandbox,
the artifact capture, the scoring, the row) stays real, which is the whole
point: those are the layers that shipped bugs no unit test could see.
Reads the prompt from stdin, as the real CLI does under "-p --input-format text".
"""
from __future__ import annotations
import json
import os
import pathlib
import sys
import urllib.request
def _turn(base_url: str, prompt: str) -> dict:
request = urllib.request.Request(
base_url.rstrip("/") + "/v1/messages",
data=json.dumps({"model": os.environ.get("ANTHROPIC_MODEL", "mock"), "max_tokens": 1024,
"messages": [{"role": "user", "content": prompt}]}).encode(),
headers={"Content-Type": "application/json",
"x-api-key": os.environ.get("ANTHROPIC_API_KEY", ""),
"anthropic-version": "2023-06-01"},
)
with urllib.request.urlopen(request, timeout=30) as response:
return json.load(response)
def _run_tool(name: str, params: dict) -> str:
"""Write executes for real - it is what produces the review artifact.
Skill and Bash do not: see the module docstring for which is modelled and
which is stubbed, and why neither can be genuine here.
"""
if name == "Write":
target = pathlib.Path(params["file_path"])
target.parent.mkdir(parents=True, exist_ok=True)
# Atomic, exactly as the real Write tool does it: temp file beside the
# target, then rename. This is the operation the read-only workspace
# boundary has to permit for the artifact directory and refuse for the
# workspace, so a stand-in that wrote in place would prove nothing.
staging = target.with_name(target.name + ".tmp.fake")
staging.write_text(params.get("content", ""))
os.replace(staging, target)
return f"wrote {target}"
if name == "Skill":
# Modelled explicitly rather than falling through to a generic success.
# The parent's evidence gate keys on a Skill request with a non-error
# result, so leaving this unimplemented let an unexecuted skill satisfy
# the gate - the gate would have been measuring the fixture, not a skill.
skill = params.get("skill") or params.get("command") or params.get("name")
if not skill:
raise NotImplementedError("Skill request carried no skill name")
return f"loaded skill {skill}"
if name == "Bash":
return "(bash suppressed in the stand-in)"
# An unsupported tool is a FAILED tool run, not a quiet success. Returning a
# plain string here made the parent's evidence gate read an unexecuted Skill
# request as a successful invocation.
raise NotImplementedError(f"unsupported tool {name}")
def main() -> int:
# stdin, because that is where the real CLI takes it under
# "-p --input-format text": the parent pipes prompt bytes in. Scanning argv
# for a non-flag token picks up a flag's VALUE instead ("text"), which is
# exactly what the prompt-fidelity test caught.
prompt = sys.stdin.read()
base_url = os.environ.get("ANTHROPIC_BASE_URL")
if not base_url:
print(json.dumps({"type": "result", "subtype": "error", "is_error": True,
"session_id": "fake-session", "num_turns": 0}), flush=True)
return 1
emit = lambda event: print(json.dumps(event), flush=True) # noqa: E731
emit({"type": "system", "subtype": "init", "session_id": "fake-session"})
try:
message = _turn(base_url, prompt)
except (OSError, ValueError) as exc:
# A provider failure is a failed SESSION, not a crashed process: dying
# here leaves no terminal result event, so the parent reports a generic
# stream error instead of the upstream failure it actually saw.
emit({"type": "result", "subtype": "error", "is_error": True,
"session_id": "fake-session", "num_turns": 0,
"error": f"provider request failed: {type(exc).__name__}: {exc}"})
return 1
blocks = message.get("content", [])
emit({"type": "assistant", "message": {"role": "assistant", "content": blocks}})
tool_results = []
for block in blocks:
if block.get("type") == "tool_use":
# A refused write is a tool ERROR the session reports and carries
# on from, not a crash. Letting it kill the process would lose the
# result event and misreport a working boundary as a broken run.
failed = False
try:
output = _run_tool(block["name"], block.get("input", {}))
except (OSError, NotImplementedError) as exc:
output, failed = f"error: {type(exc).__name__}: {exc}", True
# is_error is load-bearing: the parent treats an ABSENT is_error as
# success, so a refused or unsupported tool would otherwise be
# scored as a completed one.
tool_results.append({
"type": "tool_result", "tool_use_id": block["id"],
"content": output, "is_error": failed,
})
if tool_results:
emit({"type": "user", "message": {"role": "user", "content": tool_results}})
# Unknown is not zero. A reply carrying no usage used to become four
# zero-valued fields plus a fabricated cost, which the harness then treats
# as a real measurement - the exact confusion the accounting this fixture
# feeds exists to prevent.
usage = message.get("usage")
# Every field that gets forwarded is validated, not just the required two.
# The parent's well_formed check tests only that the four keys are PRESENT,
# so an unvalidated cache value rides into a success result and is recorded
# as a real measurement. A field good enough to report is good enough to
# check.
countable = lambda v: isinstance(v, int) and not isinstance(v, bool) and v >= 0 # noqa: E731
if not isinstance(usage, dict) or not all(
countable(usage.get(f)) for f in ("input_tokens", "output_tokens")
) or not all(
countable(usage[f])
for f in ("cache_read_input_tokens", "cache_creation_input_tokens")
if f in usage
):
emit({"type": "result", "subtype": "error", "is_error": True,
"session_id": "fake-session", "num_turns": 1,
"error": "provider reply carried no usable usage; refusing to report a measured run"})
return 1
emit({
"type": "result",
"subtype": "success",
"is_error": False,
"session_id": "fake-session",
"num_turns": 1,
"duration_ms": 1200,
# A measured zero is not the same as unmeasured; the parent rejects a
# collapsed cost, so report a real one.
"total_cost_usd": 0.42,
# Forward exactly the fields the provider reported. Defaulting the
# absent ones to 0 fabricated a complete measurement out of an
# incomplete reply - and worse, it made the parent's own completeness
# check (runner_sessions.USAGE_FIELDS / well_formed) unfirable from any
# offline test, because the stand-in always satisfied it.
"usage": {
field: usage[field]
for field in ("input_tokens", "output_tokens",
"cache_read_input_tokens", "cache_creation_input_tokens")
if field in usage
},
})
return 0
if __name__ == "__main__":
raise SystemExit(main())

467
eval/tests/test_comparator_reuse.py Normal file
View file

@ -0,0 +1,467 @@
"""Comparator-row reuse: skip unchanged incumbent/CE cells, never candidates."""
from __future__ import annotations
import hashlib
import os
from datetime import UTC, datetime, timedelta
from pathlib import Path
import pytest
from workflow_bench import comparator_reuse
from workflow_bench.comparator_reuse import (
ComparatorReuseExpectation,
TaskReuseBinding,
materialize_reused_row,
row_is_reusable_comparator,
select_reusable_comparator_rows,
)
from workflow_bench.proposer_sandbox import SandboxError
from workflow_bench.runner_sessions import PARENT_EVENT_STREAM_SOURCE
requires_openat = pytest.mark.skipif(
os.open not in os.supports_dir_fd,
reason="comparator reuse resolves every artifact against a pinned directory descriptor",
)
def _digest(text: str = "blob") -> str:
return hashlib.sha256(text.encode()).hexdigest()
def _artifact(name: str = "session-1.jsonl", payload: bytes = b'{"type":"ok"}\n') -> dict:
return {
"path": f"transcripts/{name}",
"sha256": hashlib.sha256(payload).hexdigest(),
"bytes": len(payload),
"source": PARENT_EVENT_STREAM_SOURCE,
}
def _row(**overrides) -> dict:
base = {
"task": "review-pr-2718-defect",
"arm": "review",
"run": 0,
"ok": True,
"error_kind": None,
"model": "gpt-5.6-sol",
"benchmark_model": "gpt-5.6-sol",
"effort": "xhigh",
"sandbox_backend": "bwrap",
"task_base_sha": "a" * 40,
"task_prompt_digest": _digest("prompt"),
"oracle_digest": _digest("oracle"),
"oracle_command_digest": _digest("oracle-cmd"),
"oracle_manifest_digest": _digest("oracle-man"),
"skill_digest": _digest("skill"),
"candidate_overlay_digest": None,
"review_evidence_valid": True,
# Production sets this whenever the review source exists, which is the
# normal path for a valid review; the fixture predated the requirement.
"review_artifact": "review-pr-2718-defect-review-run0.review.json",
"review_score": {"weighted_f1": 0.4},
"review_weighted_f1": 0.4,
"transcript_missing": False,
"transcript_artifacts": [_artifact()],
"recorded_at": datetime.now(UTC).isoformat(),
"runtime_digest": _digest("cli"),
"task_asset_manifest_digest": _digest("assets"),
"sandbox_dependency_manifest_digest": _digest("deps"),
}
base.update(overrides)
return base
def _expected(**overrides) -> ComparatorReuseExpectation:
now = datetime.now(UTC)
values = dict(
model="gpt-5.6-sol",
effort="xhigh",
sandbox_backend="bwrap",
runtime_digest=_digest("cli"),
now=now,
max_age=timedelta(days=90),
tasks={
"review-pr-2718-defect": TaskReuseBinding(
task_base_sha="a" * 40,
task_prompt_digest=_digest("prompt"),
oracle_digest=_digest("oracle"),
oracle_command_digest=_digest("oracle-cmd"),
oracle_manifest_digest=_digest("oracle-man"),
task_asset_manifest_digest=_digest("assets"),
sandbox_dependency_manifest_digest=_digest("deps"),
)
},
skill_digests={"review": _digest("skill"), "ce_review": None},
ce_plugin_version="3.24.0",
ce_plugin_manifest_digest=_digest("ce"),
)
values.update(overrides)
return ComparatorReuseExpectation(**values)
def test_matching_incumbent_review_row_is_reusable() -> None:
assert row_is_reusable_comparator(_row(), _expected()) is True
def test_candidate_rows_are_never_reusable() -> None:
assert row_is_reusable_comparator(_row(arm="candidate_review"), _expected()) is False
def test_skill_digest_drift_rejects_reuse() -> None:
assert row_is_reusable_comparator(_row(), _expected(skill_digests={"review": _digest("other")})) is False
def test_excluded_or_failed_rows_are_not_reusable() -> None:
expected = _expected()
assert row_is_reusable_comparator(_row(error_kind="session-error", ok=False), expected) is False
assert row_is_reusable_comparator(_row(ok=False), expected) is False
assert row_is_reusable_comparator(_row(review_evidence_valid=False), expected) is False
assert row_is_reusable_comparator(_row(recorded_at=(datetime.now(UTC) - timedelta(days=91)).isoformat()), expected) is False
def test_runtime_digest_mismatch_rejects_when_both_sides_are_bound() -> None:
row = _row(runtime_digest=_digest("old-cli"))
assert row_is_reusable_comparator(row, _expected(runtime_digest=_digest("new-cli"))) is False
assert row_is_reusable_comparator(row, _expected(runtime_digest=_digest("old-cli"))) is True
# A row with no runtime_digest was measured by a harness that recorded none,
# which is the drift this lock exists to catch - not evidence of agreement.
assert row_is_reusable_comparator(_row(runtime_digest=None), _expected()) is False
# And a sweep that cannot determine its own digest must not reuse either.
assert row_is_reusable_comparator(_row(), _expected(runtime_digest=None)) is False
def test_ce_review_matches_plugin_digest_not_repo_skill() -> None:
row = _row(
arm="ce_review",
skill_digest=None,
ce_plugin_version="3.24.0",
ce_plugin_manifest_digest=_digest("ce"),
)
assert row_is_reusable_comparator(row, _expected()) is True
assert (
row_is_reusable_comparator(row, _expected(ce_plugin_manifest_digest=_digest("other")))
is False
)
def test_select_drops_conflicting_duplicates() -> None:
first = _row(review_weighted_f1=0.4)
second = _row(review_weighted_f1=0.9, recorded_at=datetime.now(UTC).isoformat())
selected = select_reusable_comparator_rows([first, second], expected=_expected())
assert selected == {}
same = select_reusable_comparator_rows([first, dict(first)], expected=_expected())
assert ("review-pr-2718-defect", "review", 0) in same
@requires_openat
def test_materialize_copies_transcript_and_review_artifacts(tmp_path: Path) -> None:
payload = b'{"type":"result"}\n'
source = tmp_path / "prior"
dest = tmp_path / "fresh"
(source / "transcripts").mkdir(parents=True)
dest.mkdir()
transcript = source / "transcripts" / "session-1.jsonl"
transcript.write_bytes(payload)
transcript.chmod(0o600)
review = source / "review-pr-2718-defect-review-run0.review.json"
review.write_text('{"verdict":"comment"}\n')
patch = source / "review-pr-2718-defect-review-run0.patch"
patch.write_text("diff\n")
row = _row(
review_artifact=review.name,
transcript_artifacts=[_artifact(payload=payload)],
)
copied = materialize_reused_row(row, source_dir=source, dest_dir=dest)
assert copied["reused"] is True
assert copied["reused_from_recorded_at"] == row["recorded_at"]
assert (dest / "transcripts" / "session-1.jsonl").read_bytes() == payload
assert (dest / review.name).read_text() == review.read_text()
assert (dest / patch.name).read_text() == "diff\n"
assert copied["transcript_artifacts"][0]["sha256"] == hashlib.sha256(payload).hexdigest()
@pytest.mark.skipif(os.name == "nt", reason="symlink creation may require elevated Windows privileges")
@requires_openat
def test_a_reused_artifact_is_copied_from_the_inode_that_was_checked(tmp_path: Path) -> None:
"""The reuse source is a directory another sweep wrote and may still write.
Validating a path and then re-opening it hands a concurrent writer the gap:
replace the checked file with a symlink and the copy follows it out of the
results directory. Swapping the path while the descriptor is held is that
same substitution, made deterministic.
"""
(tmp_path / "transcript.jsonl").write_bytes(b"verified\n")
decoy = tmp_path / "decoy.jsonl"
decoy.write_bytes(b"substituted\n")
with comparator_reuse._open_real_directory(tmp_path, label="reuse source") as dir_fd:
with comparator_reuse._open_regular("transcript.jsonl", dir_fd=dir_fd, label="transcript") as descriptor:
(tmp_path / "transcript.jsonl").unlink()
(tmp_path / "transcript.jsonl").symlink_to(decoy)
comparator_reuse._copy_owner_only(descriptor, "copy.jsonl", dir_fd=dir_fd)
assert (tmp_path / "copy.jsonl").read_bytes() == b"verified\n"
with pytest.raises(SandboxError, match="regular non-symlink"):
with comparator_reuse._open_regular("transcript.jsonl", dir_fd=dir_fd, label="transcript"):
pass
@pytest.mark.skipif(os.name == "nt", reason="symlink creation may require elevated Windows privileges")
@requires_openat
def test_a_symlinked_transcripts_directory_is_refused_on_both_sides(tmp_path: Path) -> None:
"""`O_NOFOLLOW` refuses the leaf, not the directory above it.
A `transcripts` symlink on the source side makes reuse read a file outside
the results directory; one on the destination side writes the copy outside
this sweep's evidence. Neither is covered by the per-file guards that let
_resolved_directory tolerate a symlinked root.
"""
payload = b'{"type":"result"}\n'
outside = tmp_path / "outside"
(outside / "transcripts").mkdir(parents=True)
(outside / "transcripts" / "session-1.jsonl").write_bytes(payload)
row = _row(transcript_artifacts=[_artifact(payload=payload)])
linked_source = tmp_path / "linked-source"
linked_source.mkdir()
(linked_source / "transcripts").symlink_to(outside / "transcripts", target_is_directory=True)
dest = tmp_path / "fresh"
dest.mkdir()
with pytest.raises(SandboxError, match="transcript source must be a real directory"):
materialize_reused_row(row, source_dir=linked_source, dest_dir=dest)
source = tmp_path / "prior"
(source / "transcripts").mkdir(parents=True)
(source / "transcripts" / "session-1.jsonl").write_bytes(payload)
linked_dest = tmp_path / "linked-dest"
linked_dest.mkdir()
(linked_dest / "transcripts").symlink_to(outside / "transcripts", target_is_directory=True)
with pytest.raises(SandboxError, match="transcript destination must be a real directory"):
materialize_reused_row(row, source_dir=source, dest_dir=linked_dest)
@requires_openat
@pytest.mark.skipif(os.name == "nt", reason="symlink creation may require elevated Windows privileges")
def test_a_renamed_transcripts_directory_cannot_redirect_a_copy(tmp_path: Path) -> None:
"""The directory is pinned, not re-walked from its name.
An lstat that passed and a pathname used afterwards are two different
directories the moment a concurrent writer renames the first one away. This
performs exactly that substitution — rename, then leave a symlink in its
place — while the descriptor is held, which is what makes the race testable
without timing.
"""
payload = b'{"type":"result"}\n'
results = tmp_path / "results"
transcripts = results / "transcripts"
transcripts.mkdir(parents=True)
(transcripts / "session-1.jsonl").write_bytes(payload)
outside = tmp_path / "outside"
outside.mkdir()
with comparator_reuse._open_real_directory(results, label="reuse source") as root_fd:
with comparator_reuse._open_real_directory(
"transcripts", dir_fd=root_fd, label="transcript source"
) as dir_fd:
transcripts.rename(results / "moved")
(results / "transcripts").symlink_to(outside, target_is_directory=True)
with comparator_reuse._open_regular(
"session-1.jsonl", dir_fd=dir_fd, label="transcript"
) as artifact_fd:
comparator_reuse._copy_owner_only(artifact_fd, "copy.jsonl", dir_fd=dir_fd)
assert (results / "moved" / "copy.jsonl").read_bytes() == payload
assert not (outside / "copy.jsonl").exists()
@requires_openat
def test_a_transcript_rewritten_mid_copy_is_refused_not_recorded(tmp_path: Path, monkeypatch) -> None:
"""The digest has to describe the bytes that were written.
A held descriptor stops the pathname being substituted; it does not stop the
inode being rewritten, and the prior sweep's directory is one this sweep
treats as concurrently writable. Hashing the source and then reading it
again to copy let the row keep the expected digest while the destination
held different bytes.
"""
payload = b'{"type":"result"}\n'
source = tmp_path / "prior"
(source / "transcripts").mkdir(parents=True)
transcript = source / "transcripts" / "session-1.jsonl"
transcript.write_bytes(payload)
dest = tmp_path / "fresh"
dest.mkdir()
row = _row(transcript_artifacts=[_artifact(payload=payload)])
# Rewrite the inode in the window the copy reads through — same length, so
# only the digest can tell, which is the point.
real_read = comparator_reuse.os.read
rewritten = {"done": False}
def rewrite_then_read(fd: int, size: int) -> bytes:
if not rewritten["done"]:
rewritten["done"] = True
with open(transcript, "r+b") as handle:
handle.write(b'{"type":"TAMPER"}')
return real_read(fd, size)
monkeypatch.setattr(comparator_reuse.os, "read", rewrite_then_read)
with pytest.raises(SandboxError, match="drifted"):
materialize_reused_row(row, source_dir=source, dest_dir=dest)
monkeypatch.undo()
# And nothing unvouched-for is left behind for the proposer to read.
assert not (dest / "transcripts" / "session-1.jsonl").exists()
@requires_openat
def test_materialize_rejects_same_directory_and_missing_transcript(tmp_path: Path) -> None:
source = tmp_path / "prior"
source.mkdir()
row = _row()
with pytest.raises(SandboxError, match="same results directory"):
materialize_reused_row(row, source_dir=source, dest_dir=source)
dest = tmp_path / "fresh"
dest.mkdir()
with pytest.raises(SandboxError, match="missing"):
materialize_reused_row(row, source_dir=source, dest_dir=dest)
@requires_openat
def test_a_reused_row_ages_from_its_first_measurement_not_the_copy():
"""Reuse chains must not refresh the clock.
materialize_reused_row restamps recorded_at with the copy time, so aging
against that field let a row be copied forward every generation and outlive
max_age forever. The original measurement time is the one that counts.
"""
original = (datetime.now(UTC) - timedelta(days=91)).isoformat()
chained = _row(recorded_at=datetime.now(UTC).isoformat(), reused_from_recorded_at=original)
assert row_is_reusable_comparator(chained, _expected()) is False
# The same row inside the window is still reusable.
fresh = _row(
recorded_at=datetime.now(UTC).isoformat(),
reused_from_recorded_at=(datetime.now(UTC) - timedelta(days=1)).isoformat(),
)
assert row_is_reusable_comparator(fresh, _expected()) is True
def test_a_future_dated_row_is_corrupt_not_fresh():
ahead = (datetime.now(UTC) + timedelta(days=2)).isoformat()
assert row_is_reusable_comparator(_row(recorded_at=ahead), _expected()) is False
def test_a_changed_sandbox_dependency_is_not_the_same_baseline():
"""The environment is part of the measurement.
This branch itself changes `sandbox_dependencies` in the review corpus, so a
prior row measured against the old set is a measurement of a different
machine. Reusing it would compare a fresh candidate to a baseline built
somewhere else and hand the promotion gate a false comparison.
"""
assert row_is_reusable_comparator(
_row(sandbox_dependency_manifest_digest=_digest("other-deps")), _expected()
) is False
assert row_is_reusable_comparator(
_row(task_asset_manifest_digest=_digest("other-assets")), _expected()
) is False
# A row that predates the field is not evidence of agreement either.
assert row_is_reusable_comparator(_row(sandbox_dependency_manifest_digest=None), _expected()) is False
@pytest.mark.skipif(os.name == "nt", reason="symlink creation may require elevated Windows privileges")
def test_reuse_directories_allow_a_symlinked_parent_but_not_a_symlinked_leaf(tmp_path: Path):
"""Pins a deliberate difference from the sandbox's mount-root check.
proposer_sandbox refuses every symlink hop because a hop changes what an
untrusted session is handed. A reuse directory is data, and every file
inside it is validated on its own, so a symlinked parent is allowed -
rejecting it would break a symlinked artifacts directory or macOS's /var
for no gain. The leaf itself must still be a real directory.
"""
real = tmp_path / "real"
real.mkdir()
(real / "inner").mkdir()
linked_parent = tmp_path / "linked"
linked_parent.symlink_to(real, target_is_directory=True)
# Reached through a symlinked parent: allowed, and resolved to the real path.
# The identity returned alongside it is what pins the root against a swap
# between the check and the open; the symlink policy itself is unchanged.
resolved, identity = comparator_reuse._resolved_directory(linked_parent / "inner", label="probe")
assert resolved == (real / "inner").resolve()
inner_stat = (real / "inner").stat()
assert identity == (inner_stat.st_dev, inner_stat.st_ino)
# The leaf itself being a symlink is still refused.
with pytest.raises(SandboxError, match="must be a real directory"):
comparator_reuse._resolved_directory(linked_parent, label="probe")
def test_a_review_row_without_its_artifact_is_not_reusable() -> None:
"""A score is a claim about evidence, not the evidence itself.
materialize_reused_row copies the review artifact only when the row names
one, so accepting a row without it would carry a scored review forward with
nothing for a proposer to read.
"""
row = _row()
assert row_is_reusable_comparator(row, _expected()) is True
without = {**row, "review_artifact": ""}
assert row_is_reusable_comparator(without, _expected()) is False
missing = {k: v for k, v in row.items() if k != "review_artifact"}
assert row_is_reusable_comparator(missing, _expected()) is False
@requires_openat
def test_a_reuse_root_replaced_after_the_check_is_refused(tmp_path: Path, monkeypatch) -> None:
"""Check and use must name the same directory, not the same string.
_resolved_directory lstats a name and the open re-walks that same name, so
a prior sweep that swaps its results root in between is opened somewhere
else. The leaf-symlink rule does not cover it - a replacement that is
itself a real directory passes every check the policy makes - and the
failure is silent, folding another directory's rows into this sweep's
comparator baseline.
"""
original = tmp_path / "results"
original.mkdir()
resolved, stale_identity = comparator_reuse._resolved_directory(original, label="probe")
# Replaced by a different REAL directory: the name still resolves and still
# passes the symlink policy, but it is not the inode that was checked.
original.rename(tmp_path / "moved")
original.mkdir()
assert comparator_reuse._resolved_directory(original, label="probe")[1] != stale_identity
monkeypatch.setattr(
comparator_reuse, "_resolved_directory", lambda *_a, **_k: (resolved, stale_identity)
)
with pytest.raises(SandboxError, match="replaced between the check and the open"):
with comparator_reuse._open_pinned_root(original, label="probe"):
pass
@requires_openat
def test_a_stable_reuse_root_opens_normally(tmp_path: Path) -> None:
"""The guard rejects nothing that holds still - a directory matches itself."""
root = tmp_path / "results"
root.mkdir()
with comparator_reuse._open_pinned_root(root, label="probe") as fd:
assert os.fstat(fd).st_ino == root.stat().st_ino

1030
eval/tests/test_evolve.py
View file

File diff suppressed because it is too large Load diff

2
eval/tests/test_mcp_bridge.py
View file

@ -1,7 +1,7 @@
"""Tests for MCPBridge._find_gitnexus_command() and subprocess spawn."""
import subprocess
import unittest
from unittest.mock import MagicMock, call, patch
from unittest.mock import MagicMock, patch
class TestFindGitnexusCommand(unittest.TestCase):

122
eval/tests/test_measure_evolution_cost.py Normal file
View file

@ -0,0 +1,122 @@
"""Cost model for the evolution wall clock: measured cells, real schedules."""
from __future__ import annotations
import pytest
from workflow_bench.measure_evolution_cost import (
CANDIDATE_ARM,
SHA_OVERHEAD_SECONDS,
DURATIONS_BY_ARM,
PROPOSER_SECONDS,
REVIEW_ARMS,
expected_task_seconds,
fed_makespan,
fed_pool_enabled,
generation_seconds,
graph_pipeline_enabled,
paid_arms,
task_cells,
wave_makespan,
)
def test_every_arm_has_its_own_unsorted_sample():
assert set(DURATIONS_BY_ARM) == set(REVIEW_ARMS)
for arm, sample in DURATIONS_BY_ARM.items():
assert len(sample) >= 10, arm
# Sorting would hand each task a uniform block and hide the variance
# the whole model exists to price.
assert list(sample) != sorted(sample), arm
assert PROPOSER_SECONDS > 0
assert SHA_OVERHEAD_SECONDS > 0
def test_weekly_reuse_pays_the_candidate_arm_only():
assert paid_arms(weekly=True, reuse_enabled=True) == (CANDIDATE_ARM,)
assert paid_arms(weekly=False, reuse_enabled=True) == REVIEW_ARMS
assert paid_arms(weekly=True, reuse_enabled=False) == REVIEW_ARMS
def test_cells_are_submitted_run_major_arm_minor():
# runner.py: [(run_idx, arm) for run_idx in range(runs) for arm in arms].
# At workers=3 that puts one cell of each arm in every wave.
cells = task_cells(2, REVIEW_ARMS, 0)
assert len(cells) == 6
expected = [DURATIONS_BY_ARM[arm][run] for run in range(2) for arm in REVIEW_ARMS]
assert cells == expected
def test_overhead_is_charged_per_sha_and_outside_the_pool():
# Two properties at once: the residual sits outside the schedule, where more
# workers cannot dissolve it, and it scales with SHAs rather than cells.
assert task_cells(1, (CANDIDATE_ARM,), 0) == [DURATIONS_BY_ARM[CANDIDATE_ARM][0]]
wide = generation_seconds(
task_count=1, runs=3, arms=REVIEW_ARMS, workers=9, fed_pool=True, unique_shas=5
)
assert wide >= PROPOSER_SECONDS + 5 * SHA_OVERHEAD_SECONDS
def test_sweep_overhead_does_not_shrink_with_the_arm_count():
"""The bias that made weekly look cheaper than it is.
A seeded weekly generation pays one arm instead of three but builds exactly
the same graphs. Charging the residual per cell billed it a third of a cost
the real sweep still pays; per SHA, the two attribute the same setup.
"""
kwargs = dict(task_count=6, runs=3, workers=3, fed_pool=False, unique_shas=5)
weekly = generation_seconds(arms=(CANDIDATE_ARM,), **kwargs)
cold = generation_seconds(arms=REVIEW_ARMS, **kwargs)
weekly_sessions = 6 * expected_task_seconds(3, (CANDIDATE_ARM,), 3, fed_pool=False)
cold_sessions = 6 * expected_task_seconds(3, REVIEW_ARMS, 3, fed_pool=False)
# Whatever each wall is, the non-session part is identical.
assert round(weekly - weekly_sessions) == round(cold - cold_sessions)
# Cycling wraps, so a task can ask for more runs than the sample holds.
long_sample = task_cells(len(DURATIONS_BY_ARM[CANDIDATE_ARM]) + 2, (CANDIDATE_ARM,), 0)
assert len(long_sample) == len(DURATIONS_BY_ARM[CANDIDATE_ARM]) + 2
def test_a_wave_costs_its_slowest_cell_and_a_fed_pool_does_not():
slow = [10.0, 1.0, 1.0, 10.0, 1.0, 1.0]
assert wave_makespan(slow, 3) == 20.0
# Fed: one worker takes the first 10; the second 10 lands on a worker that
# has already cleared a 1, and the remaining 1s fill the third.
assert fed_makespan(slow, 3) == 11.0
assert fed_makespan(slow, 1) == wave_makespan(slow, 1) == 24.0
def test_expected_task_seconds_is_alignment_averaged_and_deterministic():
waved = expected_task_seconds(3, REVIEW_ARMS, 3, fed_pool=False)
assert waved == expected_task_seconds(3, REVIEW_ARMS, 3, fed_pool=False)
assert expected_task_seconds(0, REVIEW_ARMS, 3, fed_pool=False) == 0.0
assert expected_task_seconds(3, (), 3, fed_pool=False) == 0.0
# The barrier can only cost time, never save it.
assert waved >= expected_task_seconds(3, REVIEW_ARMS, 3, fed_pool=True)
def test_a_generation_pays_one_proposer_session_on_top_of_its_tasks():
one = generation_seconds(
task_count=1, runs=3, arms=REVIEW_ARMS, workers=3, fed_pool=False, unique_shas=1
)
two = generation_seconds(
task_count=2, runs=3, arms=REVIEW_ARMS, workers=3, fed_pool=False, unique_shas=1
)
# Each extra task adds exactly one task's makespan. The proposer and the
# per-SHA sweep overhead are both paid once, not per task.
assert two - one == pytest.approx(
one - PROPOSER_SECONDS - SHA_OVERHEAD_SECONDS, abs=2.0
)
def test_feature_flags_read_the_runner_not_the_wish():
assert graph_pipeline_enabled("def _run_sweep(): pass") == 0
assert graph_pipeline_enabled("graph_prefetch = GraphPrefetch(...)") == 1
assert fed_pool_enabled("def _run_wave(): pass") == 0
assert fed_pool_enabled("def _run_fed_pool(): pass") == 1
@pytest.mark.parametrize("workers", [1, 3, 8])
def test_more_workers_never_lengthen_a_task(workers):
serial = expected_task_seconds(3, REVIEW_ARMS, 1, fed_pool=True)
assert expected_task_seconds(3, REVIEW_ARMS, workers, fed_pool=True) <= serial

280
eval/tests/test_mock_provider.py Normal file
View file

@ -0,0 +1,280 @@
"""The mock has to be right about the wire, or every test built on it lies."""
from __future__ import annotations
import json
import os
import shutil
import subprocess
import urllib.request
from pathlib import Path
import pytest
from workflow_bench.mock_provider import MockProvider, Reply
from workflow_bench.provider_usage import (
ANTHROPIC,
LITELLM_NORMALIZED,
OPENAI_RESPONSES,
normalize_usage,
)
def _post(url: str, payload: dict) -> tuple[int, bytes]:
request = urllib.request.Request(
url, data=json.dumps(payload).encode(), headers={"Content-Type": "application/json"}
)
with urllib.request.urlopen(request, timeout=10) as response:
return response.status, response.read()
def test_anthropic_messages_returns_a_usable_message() -> None:
with MockProvider([Reply(text="reviewed")]) as provider:
_status, raw = _post(provider.base_url + "/v1/messages", {"model": "m", "messages": []})
body = json.loads(raw)
assert body["role"] == "assistant"
assert body["content"][0]["text"] == "reviewed"
assert body["stop_reason"] == "end_turn"
def test_a_scripted_tool_call_is_carried_as_a_tool_use_block() -> None:
"""Tool blocks are how a mocked run produces real artifacts.
The CLI executes what it is asked to run, so a Write block makes it write
that file for real inside the sandbox - which is how an artifact-producing
cell can be exercised with no model involved.
"""
write = {"name": "Write", "input": {"file_path": "/review-output/review-output.json", "content": "{}"}}
with MockProvider([Reply(text="writing", tools=[write])]) as provider:
_status, raw = _post(provider.base_url + "/v1/messages", {"model": "m", "messages": []})
body = json.loads(raw)
block = body["content"][1]
assert block["type"] == "tool_use" and block["name"] == "Write"
assert block["input"]["file_path"] == "/review-output/review-output.json"
assert body["stop_reason"] == "tool_use", "a turn ending in a tool call must say so"
def test_streaming_emits_the_event_sequence_a_consumer_expects() -> None:
with MockProvider([Reply(text="hi")]) as provider:
request = urllib.request.Request(
provider.base_url + "/v1/messages",
data=json.dumps({"model": "m", "messages": [], "stream": True}).encode(),
headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(request, timeout=10) as response:
assert response.headers["Content-Type"] == "text/event-stream"
body = response.read().decode()
events = [line[len("event: ") :] for line in body.splitlines() if line.startswith("event: ")]
assert events[0] == "message_start"
assert events[-1] == "message_stop"
assert "content_block_delta" in events
# message_delta carries the final usage, which is where output tokens land.
assert events[-2] == "message_delta"
def test_each_protocol_reports_usage_in_its_own_arithmetic() -> None:
"""The whole point: the two providers count the same numbers differently.
Anthropic's cache fields ADD to input_tokens; OpenAI's are SUBSETS of it.
Scripting one Reply and serving it both ways is what makes that asymmetry
testable without a paid request.
"""
reply = Reply(input_tokens=2_000, output_tokens=300, cache_read_input_tokens=7_000, cache_creation_input_tokens=1_000)
with MockProvider([reply, reply]) as provider:
_s, anthropic_raw = _post(provider.base_url + "/v1/messages", {"model": "m", "messages": []})
_s, openai_raw = _post(provider.base_url + "/v1/responses", {"model": "m", "input": []})
anthropic = normalize_usage(ANTHROPIC, json.loads(anthropic_raw)["usage"])
openai = normalize_usage(OPENAI_RESPONSES, json.loads(openai_raw)["usage"])
assert anthropic.total_input_tokens == 10_000
assert openai.total_input_tokens == 10_000, "same billed work, stated as the whole"
assert anthropic.ordinary_input_tokens == 2_000
assert openai.ordinary_input_tokens == 2_000, "recovered by subtraction, not addition"
assert openai.cache_read_input_tokens == 7_000
def test_a_scripted_failure_is_returned_as_one() -> None:
"""Billed failures are part of what the accounting must survive."""
with MockProvider([Reply(status_code=529, error_body={"error": {"type": "overloaded_error"}})]) as provider:
try:
_post(provider.base_url + "/v1/messages", {"model": "m", "messages": []})
raise AssertionError("the scripted failure was not returned")
except urllib.error.HTTPError as exc:
assert exc.code == 529
def test_requests_are_recorded_for_assertions() -> None:
with MockProvider() as provider:
_post(provider.base_url + "/v1/messages", {"model": "claude-sonnet-4-5", "messages": [{"role": "user"}]})
assert len(provider.requests) == 1
assert provider.requests[0].body["model"] == "claude-sonnet-4-5"
assert provider.requests[0].path.endswith("/v1/messages")
def test_an_unscripted_turn_gets_the_default_rather_than_stalling() -> None:
"""A real run makes more calls than a test wants to enumerate."""
with MockProvider([Reply(text="first")], default=Reply(text="fallback")) as provider:
_s, one = _post(provider.base_url + "/v1/messages", {"model": "m", "messages": []})
_s, two = _post(provider.base_url + "/v1/messages", {"model": "m", "messages": []})
assert json.loads(one)["content"][0]["text"] == "first"
assert json.loads(two)["content"][0]["text"] == "fallback"
def test_a_request_through_the_real_gateway_records_native_usage(tmp_path, monkeypatch) -> None:
"""The whole stack minus the model: proxy, translation, callback, log.
This is the path that shipped three separate defects invisible to unit
tests - the usage variable never reaching the proxy subprocess, the
callback failing to import when loaded by path, and failures never
recorded. All three live between the gateway and the provider, which is
exactly the span this exercises.
"""
import yaml
from workflow_bench import model_gateway
from workflow_bench.model_gateway import OpenAIGateway
from workflow_bench.provider_usage import USAGE_LOG_ENV_VAR
if shutil.which("litellm") is None:
import pytest
pytest.skip("litellm console script absent; the proxy cannot start here")
usage_log = tmp_path / "provider_usage.jsonl"
monkeypatch.setenv(USAGE_LOG_ENV_VAR, str(usage_log))
reply = Reply(input_tokens=2_000, output_tokens=300, cache_read_input_tokens=7_000, cache_creation_input_tokens=1_000)
with MockProvider(default=reply) as provider:
original = model_gateway.write_openai_litellm_config
def config(path, names):
original(path, names)
document = yaml.safe_load(path.read_text())
for entry in document["model_list"]:
entry["litellm_params"]["api_base"] = f"{provider.base_url}/v1"
path.write_text(yaml.safe_dump(document))
return path
monkeypatch.setattr(model_gateway, "write_openai_litellm_config", config)
with OpenAIGateway(
openai_api_key="mock-key", model_names=["gpt-4.1"], work_dir=tmp_path / "gw", ready_timeout_s=60
) as gateway:
request = urllib.request.Request(
gateway.base_url + "/v1/messages",
data=json.dumps({"model": "gpt-4.1", "max_tokens": 32, "messages": [{"role": "user", "content": "ping"}]}).encode(),
headers={"Content-Type": "application/json", "x-api-key": gateway.auth_token, "anthropic-version": "2023-06-01"},
)
with urllib.request.urlopen(request, timeout=60):
pass
assert usage_log.exists(), "the callback never wrote - the env did not reach the proxy"
events = [json.loads(line) for line in usage_log.read_text().splitlines()]
assert events, "the proxy started but recorded nothing"
event = events[-1]
native = event["native_usage"]
# LiteLLM hands a callback its OWN normalised object, not the upstream body:
# an OpenAI Responses reply arrives as prompt_tokens / prompt_tokens_details.
# Asserting the wire shape here is what proved the shipped adapter read keys
# that are never present.
assert native["prompt_tokens_details"]["cached_tokens"] == 7_000
assert native["prompt_tokens_details"]["cache_write_tokens"] == 1_000
assert event["provider"] == LITELLM_NORMALIZED
assert event["call_type"] == "anthropic_messages", "the observed call type, not a Responses one"
usage = normalize_usage(event["provider"], native)
assert usage.total_input_tokens == 10_000
assert usage.cache_read_input_tokens == 7_000
assert usage.cache_write_input_tokens == 1_000
assert usage.ordinary_input_tokens == 2_000
assert usage.complete, "a run that cannot interpret its own usage measured nothing"
def test_probe_what_identity_the_real_cli_actually_sends(tmp_path: Path) -> None:
"""An experiment, not an assertion: which fields could correlate a request to a cell?
Per-cell usage attribution is unbuilt because one proxy serves the whole
sweep, so anything read from the proxy environment is identical for every
request. Attribution needs something that travels WITH the request, and
what the Claude Code CLI actually sends is not documented anywhere I can
check - guessing it is how the last three accounting bugs happened.
So this drives the REAL pinned CLI against the mock and prints the
identity-bearing fields that arrive. It asserts only that a request was
made; the value is the recorded evidence, which the job log preserves.
"""
claude = os.environ.get("CLAUDE_CANARY_BIN")
if not claude or not Path(claude).exists():
pytest.skip("no pinned Claude CLI here; the containment job supplies CLAUDE_CANARY_BIN")
with MockProvider(default=Reply(text="ok")) as provider:
subprocess.run(
[claude, "-p", "--input-format", "text", "--output-format", "stream-json", "--verbose"],
input=b"say ok",
capture_output=True,
timeout=120,
env={
**os.environ,
"ANTHROPIC_BASE_URL": provider.base_url,
"ANTHROPIC_API_KEY": "offline-probe",
"HOME": str(tmp_path),
},
)
assert provider.requests, "the real CLI never reached the mock provider"
request = provider.requests[0]
interesting = {
"header:" + name: value
for name, value in request.headers.items()
if any(k in name.lower() for k in ("session", "user", "trace", "request-id", "conversation", "metadata"))
}
interesting.update(
{f"body:{key}": request.body[key] for key in ("metadata", "user", "session_id") if key in request.body}
)
print("\nIDENTITY FIELDS THE REAL CLI SENDS:")
print(" body keys:", sorted(request.body))
print(" candidate correlators:", interesting or "NONE — per-cell attribution needs another mechanism")
def test_scripted_tools_survive_the_responses_protocol_too() -> None:
"""The gateway uses Responses BECAUSE it carries tool use.
Emitting only output_text there meant a scripted Write or Skill crossed the
gateway with the tool dropped, so a mock claiming to serve both protocols
was wrong about the one the gateway actually runs.
"""
write = {"name": "Write", "input": {"file_path": "/review-output/review-output.json", "content": "{}"}}
with MockProvider([Reply(text="writing", tools=[write])]) as provider:
_status, raw = _post(provider.base_url + "/v1/responses", {"model": "m", "input": []})
output = json.loads(raw)["output"]
calls = [item for item in output if item["type"] == "function_call"]
assert len(calls) == 1, "the scripted tool must cross the Responses path"
assert calls[0]["name"] == "Write"
assert json.loads(calls[0]["arguments"])["file_path"] == "/review-output/review-output.json"
def test_an_omitted_cache_field_stays_omitted_on_the_responses_wire_too() -> None:
"""Absence must survive both protocols, not just the Anthropic one.
`_int_or_none` reads an absent detail key as unknown and a present 0 as a
measured zero, so serializing 0 for a scripted None would claim a
measurement the reply never made.
"""
with MockProvider([Reply(input_tokens=2_000, cache_read_input_tokens=None)]) as provider:
_status, raw = _post(provider.base_url + "/v1/responses", {"model": "m", "input": []})
details = json.loads(raw)["usage"]["input_tokens_details"]
assert "cached_tokens" not in details, "an omitted field must not serialize as a measured zero"
assert details["cache_write_tokens"] == 0, "a scripted 0 is still a real measurement"

435
eval/tests/test_model_gateway.py Normal file
View file

@ -0,0 +1,435 @@
"""Credential routing for the OpenAI loopback gateway."""
from __future__ import annotations
import subprocess
import os
import json
import signal
import socket
import time
import sys
import threading
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from unittest import mock
import pytest
import yaml
from workflow_bench.model_gateway import (
DEFAULT_GATEWAY_READY_TIMEOUT_S,
GATEWAY_READY_TIMEOUT_ENV,
GATEWAY_REQUEST_TIMEOUT_S,
OpenAIGateway,
gateway_ready_timeout_s,
anthropic_api_key_from_environ,
claude_gateway_model_env,
is_openai_model,
litellm_proxy_argv,
openai_backend_model,
openai_litellm_config,
resolve_model_access,
write_openai_litellm_config,
)
def test_supervisor_reports_proxy_failure_without_aborting_on_its_stdin_reader():
supervisor = Path(__file__).resolve().parents[1] / "workflow_bench" / "gateway_supervisor.py"
process = subprocess.Popen(
[sys.executable, str(supervisor), sys.executable, "-c", "raise RuntimeError('proxy failed')"],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
try:
# Keep the owner pipe open: the proxy exits independently of its owner.
process.wait(timeout=10)
assert process.returncode == 1
assert b"proxy failed" in process.stderr.read()
finally:
process.stdin.close()
if process.poll() is None:
process.kill()
process.wait(timeout=5)
def test_locked_litellm_translates_messages_to_offline_responses(monkeypatch, tmp_path):
from workflow_bench import model_gateway
observed = []
class Upstream(BaseHTTPRequestHandler):
def log_message(self, *args):
pass
def do_POST(self):
body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
observed.append((self.path, self.headers.get("Authorization"), body))
response = {
"id": "resp_offline",
"object": "response",
"created_at": int(time.time()),
"status": "completed",
"model": "gpt-4.1",
"error": None,
"output": [
{
"id": "msg_offline",
"type": "message",
"role": "assistant",
"status": "completed",
"content": [{"type": "output_text", "text": "offline pong", "annotations": []}],
}
],
"usage": {
"input_tokens": 1,
"output_tokens": 2,
"total_tokens": 3,
"input_tokens_details": {"cached_tokens": 0},
"output_tokens_details": {"reasoning_tokens": 0},
},
}
payload = json.dumps(response).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(payload)))
self.end_headers()
self.wfile.write(payload)
upstream = ThreadingHTTPServer(("127.0.0.1", 0), Upstream)
worker = threading.Thread(target=upstream.serve_forever, daemon=True)
worker.start()
original = model_gateway.write_openai_litellm_config
def config(path, names):
original(path, names)
document = yaml.safe_load(path.read_text())
for entry in document["model_list"]:
entry["litellm_params"]["api_base"] = f"http://127.0.0.1:{upstream.server_port}/v1"
path.write_text(yaml.safe_dump(document))
return path
monkeypatch.setattr(model_gateway, "write_openai_litellm_config", config)
try:
with OpenAIGateway(
openai_api_key="offline-upstream-secret",
model_names=["gpt-4.1"],
work_dir=tmp_path / "gateway",
ready_timeout_s=60,
) as gateway:
port = gateway.port
request = urllib.request.Request(
gateway.base_url + "/v1/messages",
data=json.dumps(
{
"model": "gpt-4.1",
"max_tokens": 32,
"messages": [{"role": "user", "content": "ping"}],
}
).encode(),
headers={
"Content-Type": "application/json",
"x-api-key": gateway.auth_token,
"anthropic-version": "2023-06-01",
},
)
with urllib.request.urlopen(request, timeout=30) as response:
translated = json.load(response)
assert "offline pong" in json.dumps(translated)
assert len(observed) == 1 and observed[0][0] == "/v1/responses"
assert observed[0][1] == "Bearer offline-upstream-secret"
assert gateway.auth_token != "offline-upstream-secret"
with socket.socket() as client:
assert client.connect_ex(("127.0.0.1", port)) != 0
gateway.close() # ownership close is idempotent
finally:
upstream.shutdown()
upstream.server_close()
worker.join(timeout=5)
@pytest.mark.parametrize("termination", ["terminate", "kill"])
@pytest.mark.parametrize("phase", ["ready", "startup"])
def test_gateway_lifetime_ends_with_its_parent(tmp_path, termination, phase):
ready = tmp_path / "ready.json"
proxy_pid = tmp_path / "proxy-pid"
proxy = tmp_path / "proxy.py"
proxy.write_text(f"""import os,sys
from pathlib import Path
from http.server import BaseHTTPRequestHandler, HTTPServer
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response({200 if phase == "ready" else 503}); self.end_headers()
Path(sys.argv[2]).write_text(str(os.getpid()))
HTTPServer(('127.0.0.1', int(sys.argv[1])), Handler).serve_forever()
""")
parent_code = f"""
import json,os,sys,time,subprocess
from pathlib import Path
from workflow_bench import model_gateway
model_gateway.litellm_proxy_argv=lambda **kwargs: [sys.executable, {str(proxy)!r}, str(kwargs['port']), {str(proxy_pid)!r}]
gateway=model_gateway.OpenAIGateway(openai_api_key='offline-secret', model_names=['gpt-4.1'], work_dir=Path({str(tmp_path / "gateway")!r}), ready_timeout_s=10)
if {phase == "startup"!r}:
Path({str(ready)!r}).write_text(json.dumps({{'port':gateway.port}}))
gateway.__enter__()
if gateway._process.stdin is not None:
assert not os.get_inheritable(gateway._process.stdin.fileno())
Path({str(ready)!r}).write_text(json.dumps({{'port':gateway.port}}))
time.sleep(60)
"""
parent = subprocess.Popen(
[sys.executable, "-c", parent_code],
cwd=Path(__file__).resolve().parents[1],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
)
try:
deadline = time.monotonic() + 12
while (not ready.exists() or not proxy_pid.exists()) and parent.poll() is None and time.monotonic() < deadline:
time.sleep(0.02)
if not ready.exists():
_, stderr = parent.communicate(timeout=1)
pytest.fail(stderr)
port = json.loads(ready.read_text())["port"]
getattr(parent, termination)()
parent.wait(timeout=5)
deadline = time.monotonic() + 15
while time.monotonic() < deadline:
with socket.socket() as client:
if client.connect_ex(("127.0.0.1", port)) != 0:
break
time.sleep(0.02)
else:
pytest.fail("gateway port survived abrupt parent death")
finally:
if parent.poll() is None:
parent.kill()
parent.wait(timeout=5)
if proxy_pid.exists():
try:
if os.name == "nt":
os.kill(int(proxy_pid.read_text()), signal.SIGTERM)
else:
os.killpg(int(proxy_pid.read_text()), signal.SIGKILL)
except OSError:
# Successful ownership cleanup has already removed this process.
pass
@pytest.mark.parametrize(
("model", "expected"),
[
("gpt-4.1", True),
("gpt-4o-mini", True),
("openai/gpt-4.1", True),
("o3", True),
("o4-mini", True),
("claude-sonnet-5", False),
("free-coder", False),
("pinned-model", False),
],
)
def test_is_openai_model(model: str, expected: bool) -> None:
assert is_openai_model(model) is expected
def test_openai_litellm_config_routes_each_id_to_openai_and_env_key(tmp_path: Path) -> None:
config = openai_litellm_config(["gpt-4.1", "openai/gpt-4.1-mini", "gpt-4.1"])
assert [row["model_name"] for row in config["model_list"]] == ["gpt-4.1", "openai/gpt-4.1-mini"]
assert config["model_list"][0]["litellm_params"]["model"] == "openai/gpt-4.1"
assert config["model_list"][1]["litellm_params"]["model"] == "openai/gpt-4.1-mini"
assert all(row["litellm_params"]["api_key"] == "os.environ/OPENAI_API_KEY" for row in config["model_list"])
assert all(row["model_info"] == {"mode": "responses"} for row in config["model_list"])
assert all(row["litellm_params"]["timeout"] == GATEWAY_REQUEST_TIMEOUT_S for row in config["model_list"])
assert config["litellm_settings"]["request_timeout"] == GATEWAY_REQUEST_TIMEOUT_S
path = write_openai_litellm_config(tmp_path / "litellm.yaml", ["gpt-4.1"])
assert yaml.safe_load(path.read_text())["model_list"][0]["model_name"] == "gpt-4.1"
if os.name != "nt":
# Windows chmod exposes a read-only flag, not POSIX access bits.
assert path.stat().st_mode & 0o777 == 0o600
def test_resolve_model_access_starts_proxy_only_for_openai_ids() -> None:
openai = resolve_model_access(
auth_token=None,
openai_api_key="sk-openai",
base_url=None,
models=["gpt-4.1", "gpt-4.1"],
)
assert openai.start_proxy is True
assert openai.openai_api_key == "sk-openai"
anthropic = resolve_model_access(
auth_token="sk-ant",
openai_api_key="sk-openai",
base_url=None,
models=["claude-sonnet-5"],
)
assert anthropic.start_proxy is False
existing = resolve_model_access(
auth_token="proxy-master",
openai_api_key=None,
base_url="http://127.0.0.1:4000",
models=["free-coder"],
)
assert existing.start_proxy is False
def test_resolve_model_access_rejects_openai_ids_without_a_key_and_mixed_providers() -> None:
with pytest.raises(ValueError, match="GITNEXUS_BENCH_OPENAI_API_KEY"):
resolve_model_access(
auth_token="sk-ant",
openai_api_key=None,
base_url=None,
models=["gpt-4.1"],
)
with pytest.raises(ValueError, match="mix"):
resolve_model_access(
auth_token=None,
openai_api_key="sk-openai",
base_url=None,
models=["gpt-4.1", "claude-sonnet-5"],
)
with pytest.raises(ValueError, match="--base-url"):
resolve_model_access(
auth_token=None,
openai_api_key=None,
base_url="http://127.0.0.1:4000",
models=["free-coder"],
)
def test_claude_gateway_aliases_pin_every_internal_tier_to_the_session_model() -> None:
env = claude_gateway_model_env("gpt-4.1")
assert env["ANTHROPIC_MODEL"] == "gpt-4.1"
assert env["ANTHROPIC_DEFAULT_HAIKU_MODEL"] == "gpt-4.1"
assert env["CLAUDE_CODE_SUBAGENT_MODEL"] == "gpt-4.1"
# High-effort reasoning outlives Claude Code's default client timeout.
assert env["API_TIMEOUT_MS"] == str(GATEWAY_REQUEST_TIMEOUT_S * 1000)
def test_openai_gateway_never_leaves_proxy_output_on_an_undrained_pipe(tmp_path: Path) -> None:
# Nothing reads the proxy's output after startup, so a pipe would block the
# proxy once its request logs filled the buffer and hang every session.
gateway = OpenAIGateway(
openai_api_key="sk-openai-secret",
model_names=["gpt-4.1"],
work_dir=tmp_path / "gw",
ready_timeout_s=0.1,
)
captured: dict[str, object] = {}
def fake_popen(argv, **kwargs):
captured.update(kwargs)
raise OSError("no proxy in this test")
with mock.patch.object(subprocess, "Popen", fake_popen):
with pytest.raises(RuntimeError, match="failed to start the OpenAI LiteLLM gateway"):
gateway.__enter__()
assert captured["stderr"] is subprocess.STDOUT
assert captured["stdout"] is not subprocess.PIPE
assert getattr(captured["stdout"], "name", "") == str(gateway.log_path)
if os.name != "nt":
assert gateway.log_path.stat().st_mode & 0o777 == 0o600
def test_gateway_startup_budget_outlives_a_cold_litellm_import(monkeypatch, tmp_path: Path) -> None:
# Importing LiteLLM takes ~17s on a cold container filesystem and the proxy
# binds its port only afterwards, so a sub-20s budget fails as "connection
# refused" on a proxy that was merely still starting.
monkeypatch.delenv(GATEWAY_READY_TIMEOUT_ENV, raising=False)
assert DEFAULT_GATEWAY_READY_TIMEOUT_S >= 60
assert gateway_ready_timeout_s() == DEFAULT_GATEWAY_READY_TIMEOUT_S
assert (
OpenAIGateway(
openai_api_key="sk-openai-secret",
model_names=["gpt-4.1"],
work_dir=tmp_path / "gw",
).ready_timeout_s
== DEFAULT_GATEWAY_READY_TIMEOUT_S
)
monkeypatch.setenv(GATEWAY_READY_TIMEOUT_ENV, "42.5")
assert gateway_ready_timeout_s() == 42.5
for bad in ("0", "-1", "soon", "nan", "inf", "-inf", "1e999"):
monkeypatch.setenv(GATEWAY_READY_TIMEOUT_ENV, bad)
with pytest.raises(ValueError, match=GATEWAY_READY_TIMEOUT_ENV):
gateway_ready_timeout_s()
@pytest.mark.parametrize("timeout", [0.0, -1.0, float("nan"), float("inf"), float("-inf")])
def test_gateway_rejects_invalid_explicit_readiness_budgets(tmp_path: Path, timeout: float) -> None:
with pytest.raises(ValueError, match="finite and positive"):
OpenAIGateway(
openai_api_key="sk-offline-test",
model_names=["gpt-4.1"],
work_dir=tmp_path / "gw",
ready_timeout_s=timeout,
)
def test_gateway_readiness_timeout_reports_the_proxy_log_and_the_override(tmp_path: Path) -> None:
gateway = OpenAIGateway(
openai_api_key="sk-openai-secret",
model_names=["gpt-4.1"],
work_dir=tmp_path / "gw",
ready_timeout_s=0.1,
)
gateway.work_dir.mkdir(parents=True)
gateway.log_path.write_text("ImportError: litellm proxy extras missing")
with pytest.raises(RuntimeError) as excinfo:
gateway._wait_until_ready()
message = str(excinfo.value)
assert "ImportError: litellm proxy extras missing" in message
assert GATEWAY_READY_TIMEOUT_ENV in message
def test_openai_backend_model_preserves_openai_prefix() -> None:
assert openai_backend_model("gpt-4.1") == "openai/gpt-4.1"
assert openai_backend_model("openai/gpt-4.1") == "openai/gpt-4.1"
def test_litellm_proxy_argv_uses_console_script_not_python_module(tmp_path: Path, monkeypatch) -> None:
# litellm 1.87 ships a console script and no litellm.__main__, so
# `python -m litellm` dies before the health check. Pin the supported argv.
# Under `uv run`, sys.executable is the base CPython — the script lives in
# VIRTUAL_ENV/bin instead.
python = tmp_path / "base" / "python"
venv_bin = tmp_path / "venv" / "bin"
python.parent.mkdir(parents=True)
venv_bin.mkdir(parents=True)
litellm = venv_bin / "litellm"
python.write_text("#!/bin/sh\n")
litellm.write_text("#!/bin/sh\n")
python.chmod(0o755)
litellm.chmod(0o755)
monkeypatch.setenv("VIRTUAL_ENV", str(tmp_path / "venv"))
monkeypatch.delenv("PATH", raising=False)
config = tmp_path / "litellm.yaml"
config.write_text("model_list: []\n")
argv = litellm_proxy_argv(
config=config,
host="127.0.0.1",
port=4010,
python_executable=str(python),
)
assert argv[0] == str(litellm.resolve())
assert "-m" not in argv
assert argv[1:] == ["--config", str(config), "--host", "127.0.0.1", "--port", "4010"]
def test_anthropic_api_key_prefers_the_named_env_and_keeps_the_legacy_alias(monkeypatch) -> None:
monkeypatch.delenv("GITNEXUS_BENCH_ANTHROPIC_API_KEY", raising=False)
monkeypatch.setenv("GITNEXUS_BENCH_AUTH_TOKEN", "legacy-secret")
assert anthropic_api_key_from_environ() == "legacy-secret"
monkeypatch.setenv("GITNEXUS_BENCH_ANTHROPIC_API_KEY", "named-secret")
assert anthropic_api_key_from_environ() == "named-secret"

195
eval/tests/test_offline_session_integration.py Normal file
View file

@ -0,0 +1,195 @@
"""A session end to end with only the model faked.
The layers between the CLI and the row are where this harness has actually
shipped bugs - the artifact that could not be written, the usage that was never
recorded, the evidence that was scored from the wrong directory. Every one of
them sat below the level its tests exercised. These run the real session path
against a scripted provider, so the only thing not real is what the model says.
"""
from __future__ import annotations
import sys
from pathlib import Path
import pytest
from workflow_bench.mock_provider import MockProvider, Reply
from workflow_bench.proposer_sandbox import (
host_workspace_write_boundary,
prepare_review_workspace,
prepare_sandbox,
)
from workflow_bench.review_scoring import REVIEW_OUTPUT, parse_review_output
from workflow_bench.runner_sessions import run_claude
FAKE_CLI = Path(__file__).parent / "fixtures" / "fake_claude.py"
REVIEW_JSON = '{"schema_version": 1, "verdict": "approve", "findings": []}'
def _session(clone: Path, provider: MockProvider, **overrides):
return run_claude(
"review the change",
clone,
claude_bin=str(FAKE_CLI),
timeout=60,
env={
"ANTHROPIC_BASE_URL": provider.base_url,
"ANTHROPIC_API_KEY": "offline",
"PATH": "/usr/bin:/bin",
},
**overrides,
)
@pytest.fixture
def clone(tmp_path: Path) -> Path:
workspace = tmp_path / "clone"
workspace.mkdir()
(workspace / "source.ts").write_text("export const answer = 42;\n")
return workspace
def test_a_session_records_the_usage_the_provider_reported(clone: Path) -> None:
"""Token counts must survive the CLI boundary, not be invented after it."""
reply = Reply(input_tokens=2_000, output_tokens=300, cache_read_input_tokens=7_000, cache_creation_input_tokens=1_000)
with MockProvider(default=reply) as provider:
record = _session(clone, provider)
assert record["ok"] is True, record.get("error_detail")
assert record["input_tokens"] == 2_000
assert record["cache_read_input_tokens"] == 7_000
assert record["cache_creation_input_tokens"] == 1_000
assert record["output_tokens"] == 300
# A measured zero would be indistinguishable from an unmeasured one.
assert record["cost_usd"] == 0.42
assert record["num_turns"] == 1
def test_a_scripted_write_produces_a_review_artifact_the_scorer_accepts(clone: Path) -> None:
"""The full artifact path: model asks, CLI writes atomically, scorer reads.
This is the operation that shipped empty for a whole run. Nothing here
fakes the write, the directory, or the parse - only the decision to write.
"""
with prepare_sandbox(
clone=clone, claude_bin=Path(sys.executable), backend="host-unsafe", preflight=False
) as sandbox:
artifact = prepare_review_workspace(sandbox, REVIEW_OUTPUT)
write = {"name": "Write", "input": {"file_path": str(artifact), "content": REVIEW_JSON}}
with MockProvider(default=Reply(text="reviewing", tools=[write])) as provider:
# Take the command configuration from the sandbox the way run_arm
# does, rather than calling run_claude bare. On host-unsafe the
# prefix is [] by construction, so this pins the WIRING, not the
# isolation - a bwrap run would carry a real prefix through here.
record = _session(
clone,
provider,
command_prefix=sandbox.command_prefix_for(),
require_pid_namespace=sandbox.require_pid_namespace,
)
assert record["ok"] is True, record.get("error_detail")
# Read inside the scope: prepare_sandbox removes the private root on exit.
verdict, findings = parse_review_output(artifact)
assert verdict == "approve"
assert findings == ()
def test_the_provider_saw_the_prompt_the_harness_meant_to_send(clone: Path) -> None:
"""A run that measures the wrong prompt measures nothing."""
with MockProvider() as provider:
_session(clone, provider)
assert provider.requests, "the session never reached the provider"
sent = provider.requests[0].body["messages"][0]["content"]
assert "review the change" in sent
def test_a_provider_failure_surfaces_as_a_failed_session_not_a_silent_pass(clone: Path) -> None:
"""An upstream 529 must not be recorded as a usable measurement."""
failing = Reply(status_code=529, error_body={"error": {"type": "overloaded_error"}})
with MockProvider(default=failing) as provider:
record = _session(clone, provider)
assert record["ok"] is False
assert record["error_kind"] is not None
def test_the_write_boundary_refuses_the_workspace_and_permits_the_artifact(clone: Path, tmp_path: Path) -> None:
"""The contract the empty-artifact run violated, on the backend available here.
A review must not change the workspace, and must still be able to write its
artifact ATOMICALLY - temp file beside the target, then rename - which is
what needs a writable parent DIRECTORY rather than a writable file. Both
halves are asserted through the real session, with the real boundary
applied, and the model scripted to attempt each one.
Scope: this is the host-unsafe boundary, which its own docstring calls
best-effort because a session that can chmod can undo it. The kernel-enforced
version is bubblewrap's --ro-bind, which needs namespaces this machine cannot
create; that half stays with the real-sandbox canary in CI.
"""
artifacts = tmp_path / "artifacts"
artifacts.mkdir()
target = artifacts / REVIEW_OUTPUT
protected = clone / "source.ts"
before = protected.read_text()
write_artifact = {"name": "Write", "input": {"file_path": str(target), "content": REVIEW_JSON}}
tamper = {"name": "Write", "input": {"file_path": str(protected), "content": "tampered"}}
# No writable= entry: the boundary only governs paths INSIDE the workspace
# (it refuses one that escapes), and the artifact directory deliberately
# lives outside it - that relocation is the fix for the empty-artifact run.
with host_workspace_write_boundary(clone):
with MockProvider(default=Reply(text="writing", tools=[write_artifact, tamper])) as provider:
record = _session(clone, provider)
assert record["ok"] is True, record.get("error_detail")
# The artifact landed, written the way the agent's Write tool does it.
verdict, _findings = parse_review_output(target)
assert verdict == "approve"
assert not list(artifacts.glob("*.tmp.*")), "the rename landed rather than a copy"
# The workspace did not move.
assert protected.read_text() == before, "the read-only workspace was modified"
def test_a_reply_missing_cache_usage_is_refused_not_zero_filled(clone: Path) -> None:
"""An omitted cache field must not arrive as a measured zero.
The parent already demands all four USAGE_FIELDS before it calls a session
measured (runner_sessions.well_formed). The stand-in used to default the
absent ones to 0, which both fabricated a complete measurement AND made
that parent guard unfirable from any offline test - it was always
satisfied. Scripting the absence is what proves the guard still fires.
"""
partial = Reply(input_tokens=2_000, output_tokens=300, cache_read_input_tokens=None)
with MockProvider(default=partial) as provider:
record = _session(clone, provider)
assert record["ok"] is False, "an incomplete usage report is not a usable measurement"
assert record["error_kind"] == "session-error"
@pytest.mark.parametrize("bad", [-5, True, "1200"], ids=["negative", "boolean", "string"])
def test_a_nonsense_cache_value_is_refused_rather_than_forwarded(clone: Path, bad: object) -> None:
"""A field good enough to report is good enough to validate.
The parent's well_formed check tests only that the four keys are PRESENT,
so an unvalidated cache value would ride into a success result and be
recorded as a real measurement.
"""
reply = Reply(input_tokens=2_000, output_tokens=300)
object.__setattr__(reply, "cache_read_input_tokens", bad)
with MockProvider(default=reply) as provider:
record = _session(clone, provider)
assert record["ok"] is False, f"{bad!r} must not be recorded as a measured cache value"

348
eval/tests/test_offline_sweep_integration.py Normal file
View file

@ -0,0 +1,348 @@
"""A whole sweep, offline: real runner, real sessions, scripted model.
The layers between a model turn and a promotion decision had never been
exercised together. Unit tests covered each in isolation and the paid runs that
would have covered the composition kept dying, so the contracts BETWEEN them
went unverified - and that is where this harness has repeatedly shipped bugs.
This drives runner.main() the way the workflow does. Everything is real: task
selection, hidden-oracle capture, the sandbox, the CLI subprocess, artifact
capture, review scoring against the oracle, aggregation, the health guard, and
the promotion gate. Only the model is scripted, through MockProvider.
Two provisioning steps are stubbed because this environment cannot supply them,
and neither is harness logic: the pinned gitnexus runtime mounts (no
node_modules in a worktree) and the sanitized graph build (needs the gitnexus
CLI at a mounted path). Containment is host-unsafe here; bubblewrap stays with
the real-sandbox canary in the containment job.
"""
from __future__ import annotations
import json
import re
import subprocess
import sys
from pathlib import Path
from types import SimpleNamespace
import os
import shutil
import pytest
from workflow_bench import oracle_assets, runner
from workflow_bench.mock_provider import MockProvider, Reply
FAKE_CLI = Path(__file__).parent / "fixtures" / "fake_claude.py"
ARMS = ("ce_review", "review", "candidate_review")
# When set, the sweep runs with NOTHING provisioning-stubbed: real bubblewrap
# containment, the real pinned runtime mounts, and the real sanitized graph
# build. The named CI job installs all three, so a missing one there is a
# regression rather than an unsupported machine - it FAILS instead of quietly
# degrading to the stubbed path, which is the whole point of the gate.
FULL_SWEEP_ENV = "GITNEXUS_REQUIRE_FULL_SWEEP"
FULL_SWEEP = os.environ.get(FULL_SWEEP_ENV) == "1"
# The runner refuses --unsafe-no-bwrap whenever CI is set, because that mode runs
# sessions with bypassPermissions behind a boundary its own docstring calls "not
# a security boundary". Deleting CI to get past that refusal would run an
# uncontained agent sweep on the runner holding the checkout and credentials, so
# the stubbed path is skipped under CI instead. The containment job sets
# GITNEXUS_REQUIRE_FULL_SWEEP=1 and takes the real bubblewrap path, so CI keeps
# its coverage; only the uncontained convenience run is given up.
pytestmark = pytest.mark.skipif(
not FULL_SWEEP and bool(os.environ.get("CI")),
reason="an uncontained sweep must not run in CI; the containment job runs it with GITNEXUS_REQUIRE_FULL_SWEEP=1",
)
# The review output and the hidden labels are DELIBERATELY different shapes -
# the labels carry line_start/line_end and no recommendation. Only a real run
# surfaces that; it is why these are written out rather than shared.
FINDING = {
"id": "f1", "severity": "high", "category": "correctness", "path": "src/sum.js",
"line": 1, "end_line": 1, "blocking": True, "scenario": "review-defect",
"evidence": "export const total = (a, b) => a - b;", "recommendation": "use a + b",
}
LABEL = {"id": "f1", "severity": "high", "category": "correctness",
"path": "src/sum.js", "line_start": 1, "line_end": 1}
SECOND_LABEL = {"id": "f2", "severity": "high", "category": "correctness",
"path": "src/scale.js", "line_start": 1, "line_end": 1}
SECOND_FINDING = {
"id": "f2", "severity": "high", "category": "correctness", "path": "src/scale.js",
"line": 1, "end_line": 1, "blocking": True, "scenario": "review-defect",
"evidence": "export const twice = (n) => n + 2;", "recommendation": "use n * 2",
}
def _git(repo: Path, *args: str) -> str:
return subprocess.run(["git", "-C", str(repo), *args], check=True,
capture_output=True, text=True).stdout.strip()
@pytest.fixture
def bench(tmp_path: Path):
"""A self-contained corpus: one repo, one task, one hidden label."""
repo = tmp_path / "repo"
(repo / "src").mkdir(parents=True)
(repo / "src" / "sum.js").write_text("export const total = (a, b) => a - b;\n")
(repo / "src" / "scale.js").write_text("export const twice = (n) => n + 2;\n")
_git(repo, "init", "-q", ".")
_git(repo, "config", "user.email", "t@t")
_git(repo, "config", "user.name", "t")
_git(repo, "add", "-A")
_git(repo, "commit", "-q", "-m", "fixture")
sha = _git(repo, "rev-parse", "HEAD")
oracles = tmp_path / "oracles"
oracles.mkdir()
(oracles / "review-fixture-defect.labels.json").write_text(
json.dumps({"schema_version": 1, "findings": [LABEL]})
)
(oracles / "review-fixture-second.labels.json").write_text(
json.dumps({"schema_version": 1, "findings": [SECOND_LABEL]})
)
tasks = tmp_path / "tasks.yaml"
tasks.write_text(
"tasks:\n"
" - id: review-fixture-defect\n"
" class: review-defect\n"
f" repo: {repo}\n"
f" ref: {sha}\n"
" prompt: Review this change and report actionable defects.\n"
' verify: test -s "$GITNEXUS_BENCH_REVIEW_OUTPUT"\n'
" oracle:\n"
' command: test -s "$GITNEXUS_BENCH_REVIEW_OUTPUT"\n'
" files: [{ source: review-fixture-defect.labels.json, target: review-labels.json }]\n"
# A SECOND task, because the thing a cross-task scheduler changes is
# invisible with one: waves are per-task, so a single task cannot show
# ordering, packing, or a breaker that spans a task boundary.
" - id: review-fixture-second\n"
" class: review-defect\n"
f" repo: {repo}\n"
f" ref: {sha}\n"
" prompt: Review the scaling helper and report actionable defects.\n"
' verify: test -s "$GITNEXUS_BENCH_REVIEW_OUTPUT"\n'
" oracle:\n"
' command: test -s "$GITNEXUS_BENCH_REVIEW_OUTPUT"\n'
" files: [{ source: review-fixture-second.labels.json, target: review-labels.json }]\n"
)
plugin = tmp_path / "ce-plugin"
(plugin / ".claude-plugin").mkdir(parents=True)
(plugin / ".claude-plugin" / "plugin.json").write_text(
json.dumps({"name": "compound-engineering", "version": "0.0.0-fixture"})
)
for skill in ("ce-plan", "ce-work", "ce-code-review"):
directory = plugin / "skills" / skill
directory.mkdir(parents=True)
(directory / "SKILL.md").write_text(f"---\nname: {skill}\ndescription: fixture\n---\nFixture.\n")
overlay = tmp_path / "overlay" / ".claude" / "skills" / "gitnexus-review"
overlay.mkdir(parents=True)
(overlay / "SKILL.md").write_text("---\nname: gitnexus-review\ndescription: fixture\n---\nCandidate.\n")
return SimpleNamespace(tasks=tasks, oracles=oracles, plugin=plugin,
overlay=tmp_path / "overlay", out=tmp_path / "out")
def _stub_provisioning(monkeypatch: pytest.MonkeyPatch) -> None:
"""Replace what this machine cannot supply - and nothing else.
Under FULL_SWEEP nothing is replaced: the runtime mounts and the graph are
built for real, so the sweep exercises containment and provisioning too.
"""
if FULL_SWEEP:
if shutil.which("bwrap") is None:
pytest.fail(f"{FULL_SWEEP_ENV}=1 but bubblewrap is absent")
return
monkeypatch.setattr(runner, "trusted_gitnexus_runtime_mounts", lambda: ())
def materialize(worktree, *, sanitized_head=None, **_kwargs):
# The one-clone registry guard reads this before any session runs.
meta = Path(worktree) / ".gitnexus"
meta.mkdir(parents=True, exist_ok=True)
(meta / "meta.json").write_text(
json.dumps({"indexedAt": "2026-09-08T00:00:00Z", "lastCommit": sanitized_head or "0" * 40})
)
def fake_graph(**kwargs):
kwargs["env"].graph_snapshots[kwargs["graph_key"]] = SimpleNamespace(
digest="fixture-graph", manifest_digest="fixture-graph-manifest",
dependency_content_digest=None, dependency_manifest_digest=None,
materialize=materialize,
)
monkeypatch.setattr(runner, "ensure_task_graph", fake_graph)
def _sweep(bench, monkeypatch: pytest.MonkeyPatch, findings: list[dict], verdict: str, *, invoke_skill: bool = True):
"""Run the real CLI against a model scripted to return `findings`."""
_stub_provisioning(monkeypatch)
monkeypatch.setattr(
oracle_assets, "ORACLE_ROOT", bench.oracles, raising=False
)
monkeypatch.setattr(
runner, "capture_task_oracles",
lambda tasks, root=bench.oracles: oracle_assets.capture_task_oracles(tasks, root=root),
)
def review_for(body: str) -> str:
# Per task: the second task's defect is in another file, so replying
# with the first task's finding would score it wrong. A cross-task
# scheduler makes which task a request belongs to load-bearing.
chosen = findings
if findings and "scaling helper" in body:
chosen = [SECOND_FINDING if f is FINDING else f for f in findings]
return json.dumps({"schema_version": 1, "verdict": verdict, "findings": chosen})
class Scripted(MockProvider):
def next_reply(self) -> Reply:
body = json.dumps(self.requests[-1].body if self.requests else {})
target = re.search(r"(/[^\s\"']*review-output\.json)", body)
skill = re.search(r"\b(gitnexus-review|ce-code-review)\b", body)
return Reply(
text="reviewing",
tools=[
# The evidence gate needs a Skill request with a non-error
# result: a review that never invoked its skill measured the
# model, not the skill.
*([{"name": "Skill", "input": {"skill": skill.group(1) if skill else "gitnexus-review"}}]
if invoke_skill else []),
{"name": "Write", "input": {
"file_path": target.group(1) if target else str(bench.out / "unmatched-review-output.json"),
"content": review_for(body)}},
],
input_tokens=2_000, output_tokens=300,
cache_read_input_tokens=7_000, cache_creation_input_tokens=1_000,
)
with Scripted() as provider:
monkeypatch.setattr(sys, "argv", [
"runner", "--tasks", str(bench.tasks), "--arms", *ARMS,
"--runs", "1", "--workers", "1", "--out", str(bench.out),
"--base-url", provider.base_url, "--anthropic-api-key", "offline",
"--claude-bin", str(FAKE_CLI),
*([] if FULL_SWEEP else ["--unsafe-no-bwrap"]),
"--model", "mock-model",
"--ce-plugin-dir", str(bench.plugin), "--ce-plugin-version", "0.0.0-fixture",
"--candidate-overlay", str(bench.overlay),
])
try:
code = runner.main()
except SystemExit as exc:
code = exc.code
rows = [json.loads(line) for line in (bench.out / "results.jsonl").read_text().splitlines()]
return code, rows, provider
def _row(rows: list[dict], arm: str, task: str = "review-fixture-defect") -> dict:
return next(r for r in rows if r["arm"] == arm and r["task"] == task)
def test_a_correct_review_scores_and_the_sweep_exits_clean(bench, monkeypatch) -> None:
"""The whole path, green: every arm measured, scored, and accounted for."""
code, rows, provider = _sweep(bench, monkeypatch, [FINDING], "request_changes")
assert code in (None, 0), f"sweep did not succeed: {code}"
tasks = {"review-fixture-defect", "review-fixture-second"}
assert len(rows) == len(ARMS) * len(tasks)
assert len(provider.requests) == len(ARMS) * len(tasks), "each cell must reach the provider once"
assert {r["task"] for r in rows} == tasks, "both tasks must have run"
row = _row(rows, "review")
assert row["ok"] is True and row["resolved"] is True
assert row["skill_invoked"] is True
assert (row["review_true_positives"], row["review_false_positives"], row["review_false_negatives"]) == (1, 0, 0)
assert row["review_f1"] == 1.0
# The provider's own numbers survived the CLI, the parser and the row.
assert row["cache_read_input_tokens"] == 7_000
assert row["input_tokens"] == 2_000
# Each task scored against ITS OWN oracle. This is what a cross-task
# scheduler puts at risk: interleaving cells from different tasks means a
# mis-routed context or artifact scores one task against another's labels,
# and both would still look "green" per row.
second = _row(rows, "review", task="review-fixture-second")
assert second["resolved"] is True and second["review_f1"] == 1.0
assert second["review_artifact"] == "review-fixture-second-review-run0.review.json"
for name in ("results.jsonl", "report.md", "promotion.json"):
assert (bench.out / name).is_file(), f"{name} was not written"
assert (bench.out / "review-fixture-defect-review-run0.review.json").is_file()
def test_one_run_cannot_promote_a_candidate(bench, monkeypatch) -> None:
"""The gate refuses on insufficient paired runs, and says so."""
_sweep(bench, monkeypatch, [FINDING], "request_changes")
promotion = json.loads((bench.out / "promotion.json").read_text())
assert promotion["run_status"] == "complete"
decision = next(d for d in promotion["decisions"] if d["candidate_arm"] == "candidate_review")
assert decision["decision"] == "insufficient_evidence"
assert any("valid paired runs" in reason for reason in decision["reasons"])
def test_a_finding_in_the_wrong_place_scores_zero_but_stays_valid_evidence(bench, monkeypatch) -> None:
"""Being wrong is a quality result, not a broken measurement.
The negative control that makes the passing case mean something: same
harness, same well-formed artifact, only the answer changed.
"""
wrong = {**FINDING, "path": "src/WRONG.js", "line": 99, "end_line": 99}
_code, rows, _provider = _sweep(bench, monkeypatch, [wrong], "request_changes")
row = _row(rows, "review")
assert (row["review_true_positives"], row["review_false_positives"], row["review_false_negatives"]) == (0, 1, 1)
assert row["review_f1"] == 0.0
assert row["resolved"] is False
assert row["error_kind"] == "oracle-failed", "a wrong answer is not a session or evidence failure"
assert row["review_evidence_valid"] is True, "the artifact was well formed; only the answer was wrong"
def test_approving_defective_code_is_a_miss_with_no_false_positive(bench, monkeypatch) -> None:
"""The other half of the control: silence scores differently from a wrong guess."""
_code, rows, _provider = _sweep(bench, monkeypatch, [], "approve")
row = _row(rows, "review")
assert (row["review_true_positives"], row["review_false_positives"], row["review_false_negatives"]) == (0, 0, 1)
assert row["review_precision"] is None, "precision is undefined with no predictions, not zero"
assert row["review_verdict_correct"] is False, "approving defective code is the wrong verdict"
assert row["review_evidence_valid"] is True
def test_a_review_that_never_invoked_its_skill_is_not_a_measurement(bench, monkeypatch) -> None:
"""The gate that separates measuring a SKILL from measuring a model.
Added because a mutation exposed it: forcing skill_was_invoked_events to
return True left every other test here passing, so nothing pinned the gate.
The artifact is written and correct in this run - only the skill request is
missing - so a pass would mean the arm scored a review it never performed.
"""
code, rows, _provider = _sweep(bench, monkeypatch, [FINDING], "request_changes", invoke_skill=False)
row = _row(rows, "review")
assert row["skill_invoked"] is False
assert row["error_kind"] == "skill-not-invoked"
assert code not in (None, 0), "the sweep must not report success on unusable evidence"
# The row still carries its own score - the artifact was well formed - and
# aggregate() DOES count it in the arm's quality median (the KNOWN GAP noted
# above aggregate(); test_workflow_bench pins the resulting 0.5). Filtering
# it out of the median alone inverted a promotion, because valid_runs and
# excluded_runs kept counting it. It counts for cost either way: the session
# ran and was billed.
assert row["review_weighted_f1"] == 1.0
assert row["review_evidence_valid"] is True

98
eval/tests/test_oracle_assets.py
View file

@ -5,7 +5,9 @@ from __future__ import annotations
import argparse
import hashlib
import os
import shutil
import subprocess
import sys
from pathlib import Path
from types import SimpleNamespace
@ -13,7 +15,13 @@ import pytest
from workflow_bench import oracle_assets, runner
from workflow_bench.evolution import evaluate_candidate
from workflow_bench.oracle_assets import capture_task_oracle, staged_task_oracle
from workflow_bench.oracle_assets import (
capture_task_oracle,
require_hidden_harness_absent,
review_case_setup_command,
staged_task_oracle,
with_hidden_harness_apply_exclude,
)
def oracle_task(*, command: str = "true", source: str = "oracle.test.ts") -> dict[str, object]:
@ -52,6 +60,7 @@ def bench_args() -> argparse.Namespace:
claude_bin="claude",
timeout=5,
model="pinned-model",
effort="xhigh",
base_url=None,
auth_token=None,
)
@ -148,6 +157,73 @@ def test_clone_sanitization_prunes_harness_checkout_and_recoverable_history(tmp_
assert git(clone, "status", "--porcelain=v1", "--untracked-files=all").stdout == ""
def test_require_hidden_harness_absent_fails_closed_on_leftover_tree(tmp_path: Path) -> None:
clone = tmp_path / "clone"
hidden = clone / "eval" / "workflow_bench"
hidden.mkdir(parents=True)
(hidden / "review_cases").mkdir()
with pytest.raises(ValueError, match="hidden harness visible"):
require_hidden_harness_absent(clone)
shutil.rmtree(hidden)
require_hidden_harness_absent(clone)
def test_hidden_harness_apply_exclude_is_idempotent() -> None:
raw = "git apply eval/workflow_bench/review_cases/pr.patch && rm -rf eval/workflow_bench"
once = with_hidden_harness_apply_exclude(raw)
assert once == review_case_setup_command("pr.patch")
assert with_hidden_harness_apply_exclude(once) == once
assert with_hidden_harness_apply_exclude("true") == "true"
def test_review_setup_skips_sanitized_harness_hunks(tmp_path: Path) -> None:
repo = tmp_path / "repo"
repo.mkdir()
def git(*args: str, check: bool = True) -> subprocess.CompletedProcess[str]:
result = subprocess.run(
["git", "-C", str(repo), *args],
check=False,
capture_output=True,
text=True,
)
if check and result.returncode != 0:
pytest.fail(f"git {' '.join(args)} failed: {result.stderr}")
return result
git("init", "--quiet", "--initial-branch=main")
git("config", "user.name", "Review Setup")
git("config", "user.email", "review-setup.invalid")
(repo / "visible.py").write_text("old\n")
hidden = repo / "eval" / "workflow_bench"
hidden.mkdir(parents=True)
(hidden / "learnings.jsonl").write_text("{}\n")
git("add", "--all")
git("commit", "--quiet", "-m", "base with harness file")
(repo / "visible.py").write_text("new\n")
(hidden / "learnings.jsonl").write_text("{}\nextra\n")
patch = git("diff").stdout
git("checkout", "--", ".")
shutil.rmtree(hidden)
patch_path = hidden / "review_cases" / "case.patch"
patch_path.parent.mkdir(parents=True)
patch_path.write_text(patch)
rejected = git("apply", "--check", str(patch_path.relative_to(repo)), check=False)
assert rejected.returncode != 0
assert "learnings.jsonl" in rejected.stderr
setup = with_hidden_harness_apply_exclude(
"git apply eval/workflow_bench/review_cases/case.patch && rm -rf eval/workflow_bench"
)
applied = subprocess.run(["/bin/sh", "-lc", setup], cwd=repo, check=False, capture_output=True, text=True)
assert applied.returncode == 0, applied.stderr
assert (repo / "visible.py").read_text() == "new\n"
assert not hidden.exists()
def test_clone_sanitization_prunes_remote_history_when_head_never_had_harness(tmp_path: Path) -> None:
source = tmp_path / "source"
source.mkdir()
@ -299,6 +375,26 @@ def test_vacuous_authored_test_cannot_self_certify_resolution(
assert record["error_kind"] == "oracle-failed"
def test_host_unsafe_oracle_executes_beside_candidate_and_is_removed(tmp_path: Path) -> None:
from workflow_bench.proposer_sandbox import prepare_sandbox
source = tmp_path / "oracles"
write_oracle(
source,
b"from pathlib import Path\nassert (Path(__file__).resolve().parents[2] / 'candidate.txt').read_text() == 'candidate'\n",
)
snapshot = capture_task_oracle(
oracle_task(command='python3 "$GITNEXUS_BENCH_ORACLE_ROOT/nested/oracle.test.ts"'), root=source
)
clone = tmp_path / "clone"
clone.mkdir()
(clone / "candidate.txt").write_text("candidate")
with prepare_sandbox(clone=clone, claude_bin=sys.executable, backend="host-unsafe") as session:
passed, detail = runner._run_hidden_oracle(snapshot, clone, bench_args(), session)
assert passed, detail
assert not list(clone.glob(".wfbench-oracle-*"))
def test_oracle_path_and_bytes_appear_only_after_the_model_session(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,

212
eval/tests/test_process_control.py
View file

@ -7,7 +7,12 @@ import os
import signal
import sys
import time
import threading
import subprocess
import json
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from types import SimpleNamespace
import pytest
@ -23,6 +28,101 @@ from workflow_bench.process_control import (
PYTHON = sys.executable
def test_cancel_before_spawn_never_starts_the_child(tmp_path):
event = threading.Event()
event.set()
sentinel = tmp_path / "spawned"
result = run_managed([PYTHON, "-c", f"open({str(sentinel)!r}, 'w').close()"], timeout=60, cancel_event=event)
assert result.state == "cancelled"
assert not sentinel.exists()
@pytest.mark.skipif(os.name == "nt", reason="POSIX signal escalation")
def test_cancel_after_spawn_kills_a_term_ignoring_child():
event = threading.Event()
started = time.monotonic()
result = run_managed(
[
PYTHON,
"-c",
"import signal,time; signal.signal(signal.SIGTERM, signal.SIG_IGN); print('ready',flush=True); time.sleep(60)",
],
timeout=60,
terminate_grace=0.1,
cancel_event=event,
stdout_observer=lambda chunk: event.set() if b"ready" in chunk else None,
)
assert result.state == "cancelled" and result.forced_kill
assert not result.timed_out
assert time.monotonic() - started < 5
@pytest.mark.skipif(os.name == "nt", reason="POSIX parent signals")
@pytest.mark.parametrize("signum", [signal.SIGINT, signal.SIGTERM])
def test_signal_cancels_active_wave_before_assets_are_released(tmp_path, signum):
ready = tmp_path / "child-pid"
assets = tmp_path / "assets"
assets.write_text("shared")
command = (
f"import os,time; from pathlib import Path; Path({str(ready)!r}).write_text(str(os.getpid())); time.sleep(60)"
)
script = f"""
import json,sys
from pathlib import Path
from workflow_bench.process_control import cancellation_scope, run_managed
from workflow_bench.runner import sweep_task_cells
rows=[]
def run(index, arm):
if index == 0:
return {{'resolved': True, 'error_kind': None}}
result=run_managed([sys.executable, '-c', {command!r}], timeout=60)
assert Path({str(assets)!r}).exists(), 'assets removed while a worker was active'
return {{'resolved': False, 'error_kind': result.state}}
with cancellation_scope(handle_signals=True) as event:
streak, tripped=sweep_task_cells([(i, 'review') for i in range(10)], workers=2, run=run,
on_start=lambda *args: None, on_record=lambda i,a,r: rows.append([i,r]),
outage_streak=0, outage_limit=5, cancel_event=event)
Path({str(assets)!r}).unlink()
print(json.dumps({{'rows': rows, 'stopped': event.is_set(), 'tripped': tripped}}))
"""
process = subprocess.Popen(
[PYTHON, "-c", script],
cwd=Path(__file__).resolve().parents[1],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
)
try:
deadline = time.monotonic() + 10
while not ready.exists() and process.poll() is None and time.monotonic() < deadline:
time.sleep(0.01)
assert ready.exists()
child_pid = int(ready.read_text())
started = time.monotonic()
process.send_signal(signum)
stdout, stderr = process.communicate(timeout=15)
assert process.returncode == 0, stderr
assert time.monotonic() - started < 15
report = json.loads(stdout)
assert report["stopped"] and not report["tripped"], "cancelled, not an outage"
assert [row[0] for row in report["rows"]] == [0, 1]
assert report["rows"][0][1]["resolved"] is True
assert report["rows"][1][1]["error_kind"] == "cancelled"
with pytest.raises(ProcessLookupError):
os.kill(child_pid, 0)
assert not assets.exists()
finally:
if process.poll() is None:
process.kill()
process.wait(timeout=5)
if ready.exists():
try:
os.killpg(int(ready.read_text()), signal.SIGKILL)
except ProcessLookupError:
# Successful cancellation has already reaped this process group.
pass
def test_managed_process_captures_normal_exit() -> None:
result = run_managed(
[PYTHON, "-c", "import sys; print('out'); print('err', file=sys.stderr)"],
@ -124,6 +224,64 @@ def test_parent_stdout_capture_reports_overflow_without_stopping_drain() -> None
assert result.stdout_tail.endswith("END")
def test_echo_stdout_streams_child_progress_and_stays_off_by_default(capfd) -> None:
command = [PYTHON, "-c", "import os; os.write(1, b'[task][arm][run 0] starting\\n')"]
quiet = run_managed(command, timeout=5)
assert quiet.ok
assert "starting" not in capfd.readouterr().err
echoed = run_managed(command, timeout=5, echo_stdout=True)
assert echoed.ok
captured = capfd.readouterr()
assert "[task][arm][run 0] starting" in captured.err
# Echoing is a passthrough, not a redirect: the tail stays intact for the
# caller that reports it after the process ends.
assert "starting" in echoed.stdout_tail
def test_echo_reaches_the_log_while_the_child_is_still_running(tmp_path: Path, monkeypatch) -> None:
"""Streaming has to be prompt, not merely eventual.
A sweep emits one line every ~45 minutes. Draining with `read(8192)` still
delivers every byte, so the tail and the capture look correct — but nothing
surfaces until the pipe closes, which turns a 15-hour job into a silent one
and is the whole reason this passthrough exists.
The child here refuses to exit until the echoed line has been observed, so
an implementation that only flushes at EOF deadlocks and fails on the
timeout rather than passing on a technicality.
"""
released = tmp_path / "echo-observed"
class Sink:
def write(self, data: bytes) -> int:
if b"first-line" in data:
released.write_text("go")
return len(data)
def flush(self) -> None:
pass
monkeypatch.setattr(process_control.sys, "stderr", SimpleNamespace(buffer=Sink()))
script = """
import pathlib, sys, time
sys.stdout.write('first-line\\n')
sys.stdout.flush()
target = pathlib.Path(%r)
for _ in range(400):
if target.exists():
break
time.sleep(0.05)
""" % str(released)
result = run_managed([PYTHON, "-c", script], timeout=15, echo_stdout=True)
assert released.exists(), "the line never reached the echo sink while the child ran"
assert result.ok
assert "first-line" in result.stdout_tail
def test_incomplete_stdin_delivery_cannot_report_success() -> None:
result = run_managed(
[PYTHON, "-c", "import os,time; os.close(0); time.sleep(0.05)"],
@ -453,3 +611,57 @@ def test_windows_normal_parent_with_grandchild_is_not_successful_evidence(tmp_pa
assert result.forced_kill
assert not result.ok
assert not sentinel.exists()
@pytest.mark.skipif(os.name == "nt", reason="POSIX process-group ownership canary")
def test_concurrent_cells_reap_only_their_own_process_tree(tmp_path: Path) -> None:
"""One cell timing out must not touch a sibling cell running beside it.
`run_managed` reaps by process group. Cells only ever ran one at a time
before, so nothing exercised what happens when a `killpg` fires while other
owned trees are alive — a leaked or shared pgid would take the siblings
down with it, and the sweep would read that as two more excluded runs.
"""
survivor_sentinel = tmp_path / "survivor-finished"
victim_sentinel = tmp_path / "victim-escaped"
# Each cell spawns a descendant, like a sandboxed session does.
survivor = """
import pathlib, subprocess, sys, time
child = subprocess.Popen([sys.executable, '-c', "import time; time.sleep(2)"])
time.sleep(1.0)
pathlib.Path(%r).write_text('finished')
child.wait()
print('survivor-done', flush=True)
""" % str(survivor_sentinel)
victim = """
import pathlib, signal, subprocess, sys, time
signal.signal(signal.SIGTERM, signal.SIG_IGN)
subprocess.Popen([
sys.executable, '-c',
"import signal,time,pathlib; signal.signal(signal.SIGTERM, signal.SIG_IGN); time.sleep(1.5); pathlib.Path(%r).write_text('escaped')"
])
while True:
time.sleep(0.01)
""" % str(victim_sentinel)
def cell(source: str, timeout: float):
return run_managed([PYTHON, "-c", source], timeout=timeout, terminate_grace=0.1)
with ThreadPoolExecutor(max_workers=3) as pool:
futures = [
pool.submit(cell, survivor, 10.0),
pool.submit(cell, victim, 0.2),
pool.submit(cell, survivor, 10.0),
]
first, doomed, second = (future.result() for future in futures)
time.sleep(1.8)
assert doomed.state == "forced-kill"
assert not victim_sentinel.exists(), "the timed-out cell leaked a descendant"
# The siblings were mid-flight when the killpg fired.
assert first.ok and second.ok
assert "survivor-done" in first.stdout_tail
assert "survivor-done" in second.stdout_tail
assert survivor_sentinel.exists()

172
eval/tests/test_promotion_apply.py
View file

@ -3,11 +3,15 @@
import json
import os
import stat
import shlex
import subprocess
from pathlib import Path, PurePosixPath
import pytest
import yaml
from workflow_bench import evolve, promotion_apply
from workflow_bench import evolution, runner
from workflow_bench.evolution import (
CANDIDATE_SKILLS,
MAX_CANDIDATE_OVERLAY_BYTES,
@ -36,6 +40,120 @@ def _git(repo: Path, *arguments: str) -> str:
)
@pytest.mark.parametrize(
"arms", [["candidate_review"], ["candidate_workflow"], ["candidate_review", "candidate_workflow"]]
)
@pytest.mark.parametrize(
"tamper",
[
None,
"aborted",
"metric",
"verdict",
"nan",
"huge-int",
"runs",
"task",
"duplicate-task",
"exclusions",
"old-schema",
],
)
def test_produced_promotion_round_trips_to_transactional_apply(tmp_path, arms, tamper):
from tests.test_evolve import bound_task_fixture, promotion_fixture
overlay = tmp_path / "overlay"
repo = tmp_path / "repo"
for arm in arms:
skill = "gitnexus-review" if arm == "candidate_review" else "gitnexus-plan"
relative = PurePosixPath(f".claude/skills/{skill}/SKILL.md")
(overlay / relative).parent.mkdir(parents=True, exist_ok=True)
(overlay / relative).write_text("candidate")
for target in mirror_targets(relative):
(repo / target).parent.mkdir(parents=True, exist_ok=True)
(repo / target).write_text("incumbent")
frozen = tmp_path / "frozen"
digest = freeze_overlay(overlay, frozen)
bases = destination_base_digests(frozen, repo_root=repo)
paired = {}
for arm in arms:
for name, candidate in ((evolution.CANDIDATE_ARMS[arm], False), (arm, True)):
paired[name] = runner.aggregate(
[
{
"resolved": True,
"cost_usd": 0.8 if candidate else 1.0,
"review_weighted_f1": 1.0 if candidate else 0.5,
"review_blocker_recall": 1.0,
"review_false_positives": 0,
"review_verdict_correct": True,
"review_clean_control": False,
"review_clean_pass": False,
}
for _ in range(3)
]
)
policy = evolution.promotion_policy(arms)
promotion = {
**promotion_fixture(),
**evolution.promotion_evidence(
{"task-a": paired},
policy=policy,
model="bench-model",
complete=True,
),
"required_candidate_arms": arms,
"candidate_overlay_digest": digest,
"target_base_digests": bases,
"selected_tasks": [bound_task_fixture()],
}
promotion = json.loads(json.dumps(promotion))
decision = promotion["decisions"][0]
row = decision["tasks"][0]
if tamper == "aborted":
promotion["run_status"] = "aborted"
elif tamper == "metric":
decision["metric"] = "fabricated"
elif tamper == "verdict":
decision["decision"] = "keep_incumbent"
elif tamper == "nan":
row["candidate"][policy[arms[0]]["metric"]] = float("nan")
elif tamper == "huge-int":
row["candidate"][policy[arms[0]]["metric"]] = 10**1000
elif tamper == "runs":
row["candidate"]["valid_runs"] = 1
elif tamper == "task":
row["task"] = "unselected"
elif tamper == "duplicate-task":
decision["tasks"].append(dict(row))
elif tamper == "exclusions":
row["candidate"]["excluded_runs"] = 1
elif tamper == "old-schema":
promotion["schema_version"] = 5
def validate():
return evolve.validate_promotion_for_apply(
promotion,
overlay_digest=digest,
benchmark_model="bench-model",
proposer_model="proposer-model",
effort="xhigh",
selected_tasks=[bound_task_fixture()],
target_base_digests=bases,
required_candidate_arms=arms,
policy=policy,
)
if tamper:
with pytest.raises(ValueError):
validate()
assert all((repo / path).read_text() == "incumbent" for path in bases)
else:
assert all(decision["decision"] == "promote" for decision in validate())
written = apply_promoted_overlay(frozen, repo_root=repo, expected_target_bases=bases)
assert all((repo / path).read_text() == "candidate" for path in written)
def test_evolve_reexports_public_promotion_helpers():
assert evolve.mirror_targets is mirror_targets
assert evolve.freeze_overlay is freeze_overlay
@ -52,6 +170,44 @@ def test_mirror_targets_cover_canonical_and_shipped_copies():
]
def test_review_mirror_targets_include_cursor_distribution():
targets = mirror_targets(PurePosixPath(".claude/skills/gitnexus-review/SKILL.md"))
assert targets == [
PurePosixPath(".claude/skills/gitnexus-review/SKILL.md"),
PurePosixPath("gitnexus/skills/gitnexus-review/SKILL.md"),
PurePosixPath("gitnexus-claude-plugin/skills/gitnexus-review/SKILL.md"),
PurePosixPath("gitnexus-cursor-integration/skills/gitnexus-review/SKILL.md"),
]
def test_workflow_stages_every_review_mirror_and_rejects_unrelated_changes(tmp_path):
overlay = tmp_path / "overlay"
relative = PurePosixPath(".claude/skills/gitnexus-review/SKILL.md")
(overlay / relative).parent.mkdir(parents=True)
(overlay / relative).write_text("candidate")
repo = tmp_path / "repo"
targets = mirror_targets(relative)
for path in targets:
(repo / path).parent.mkdir(parents=True, exist_ok=True)
(repo / path).write_text("incumbent")
(repo / "unrelated.txt").write_text("before")
_git(repo, "init", "-q")
_git(repo, "add", ".")
_git(repo, "-c", "user.name=Fixture", "-c", "user.email=fixture@example.test", "commit", "-qm", "base")
apply_promoted_overlay(overlay, repo_root=repo)
workflow_path = Path(__file__).resolve().parents[2] / ".github/workflows/gitnexus-skill-evolution.yml"
steps = yaml.safe_load(workflow_path.read_text())["jobs"]["evolve"]["steps"]
publish = next(step["run"] for step in steps if step.get("name") == "Open the promotion PR")
staging = next(line for line in publish.splitlines() if line.startswith("git add "))
_git(repo, *shlex.split(staging)[1:])
assert _git(repo, "diff", "--cached", "--name-only").splitlines() == sorted(map(str, targets))
assert {(repo / path).read_text() for path in targets} == {"candidate"}
(repo / "unrelated.txt").write_text("after")
guard = next(step["run"] for step in steps if step.get("name") == "Detect and bound the applied promotion")
guarded = subprocess.run(["bash", "-c", guard], cwd=repo, capture_output=True, text=True)
assert guarded.returncode == 1 and "outside the skill trees: unrelated.txt" in guarded.stdout
def test_apply_promoted_overlay_writes_all_mirrors(tmp_path):
overlay = tmp_path / "overlay"
skill_md = overlay / ".claude" / "skills" / "gitnexus-plan" / "SKILL.md"
@ -492,14 +648,7 @@ def test_committed_destination_bases_ignore_and_reject_live_target_edits(tmp_pat
assert dirty.read_text() == "user edit"
def test_mirror_roots_cover_every_candidate_skill_and_omit_none_that_ships_to_cursor():
# promotion_apply.mirror_targets writes canonical + MIRROR_SKILL_ROOTS, which
# today omits the Cursor tree. That is only safe because no candidate skill is
# cursor-shipped. If a future edit adds a cursor-shipped skill (e.g.
# gitnexus-review) to CANDIDATE_SKILLS, apply_promoted_overlay would rewrite
# the other trees and silently skip Cursor — the PR #2488 asymmetric-sync bug
# class. Pin the invariant to the filesystem, the source of truth the TS drift
# guard already enforces.
def test_mirror_roots_cover_every_candidate_skill_including_cursor_review():
repo_root = Path(__file__).resolve().parents[2]
cursor_root = repo_root / "gitnexus-cursor-integration" / "skills"
for skill in sorted(CANDIDATE_SKILLS):
@ -507,10 +656,9 @@ def test_mirror_roots_cover_every_candidate_skill_and_omit_none_that_ships_to_cu
assert canonical.is_dir(), f"candidate skill {skill} has no canonical .claude/skills dir"
for target in mirror_targets(PurePosixPath(".claude", "skills", skill, "SKILL.md")):
assert (repo_root / target).is_file(), f"candidate skill mirror missing on disk: {target}"
assert not (cursor_root / skill).exists(), (
f"candidate skill {skill} ships to Cursor, but MIRROR_SKILL_ROOTS does not cover "
"gitnexus-cursor-integration/skills — promotion would sync it asymmetrically"
)
if (cursor_root / skill).exists():
expected = PurePosixPath("gitnexus-cursor-integration/skills", skill, "SKILL.md")
assert expected in mirror_targets(PurePosixPath(".claude", "skills", skill, "SKILL.md"))
def test_committed_destination_bases_reject_overlay_adding_uncommitted_target(tmp_path):

525
eval/tests/test_proposer_sandbox.py
View file

@ -9,37 +9,130 @@ import stat
import subprocess
import sys
import threading
from dataclasses import replace
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from types import SimpleNamespace
import pytest
from workflow_bench import runner
from workflow_bench import runner, runner_artifacts
from workflow_bench import proposer_sandbox
from workflow_bench.process_control import ManagedProcessResult, run_managed
from workflow_bench.proposer_sandbox import (
MAX_BUNDLE_BYTES,
MAX_EVIDENCE_FILE_BYTES,
SANDBOX_CLAUDE,
SANDBOX_NODE,
SANDBOX_NODE_PREFIX,
SANDBOX_EVIDENCE,
SANDBOX_GITNEXUS_CLI,
SANDBOX_GIT_EXCLUDES,
VITE_TEMP_DIR,
SANDBOX_PATH,
SANDBOX_REVIEW_OUTPUT,
SANDBOX_PYTHON3,
SANDBOX_SHELL_PREFIX,
SANDBOX_USER_SKILLS,
SANDBOX_WORKSPACE,
ReadOnlyMount,
SandboxError,
_runtime_mount_args,
build_claude_settings,
build_sandbox_environment,
_force_rmtree,
host_workspace_write_boundary,
prepare_review_workspace,
prepare_sandbox,
preflight_bubblewrap,
sandbox_workspace_write_boundary,
stage_evidence_bundle,
stage_task_assets,
)
from workflow_bench.review_scoring import REVIEW_OUTPUT, parse_review_output
from workflow_bench.task_assets import TaskAssetCache, stage_task_assets as stage_immutable_task_assets
@pytest.mark.parametrize("entry", ["directory", "relative-link", "absolute-link"])
def test_review_preparation_rejects_a_reused_artifact_directory(tmp_path, entry):
clone = tmp_path / "clone"
clone.mkdir()
sentinel = tmp_path / "sentinel"
sentinel.write_text("must survive")
with prepare_sandbox(clone=clone, claude_bin=sys.executable, backend="host-unsafe") as sandbox:
stale = proposer_sandbox.review_output_path(sandbox, "review-output.json").parent
if entry == "directory":
stale.mkdir()
(stale / "review-output.json").write_text("a previous cell's verdict")
else:
# relpath, not a hand-written "../sentinel": stale is
# <private_root>/review-output, which is nowhere near tmp_path, so the
# literal produced a dangling link and the assertion below proved nothing.
stale.symlink_to(
sentinel if entry == "absolute-link" else Path(os.path.relpath(sentinel, stale.parent))
)
with pytest.raises(SandboxError, match="already exists"):
proposer_sandbox.prepare_review_workspace(sandbox, "review-output.json")
assert sentinel.read_text() == "must survive"
def test_review_preparation_leaves_a_clone_entry_of_the_same_name_alone(tmp_path):
# The artifact no longer lives in the workspace, so a file that happens to
# share its name is just one of the repository's own files.
clone = tmp_path / "clone"
clone.mkdir()
(clone / "review-output.json").write_text("repository content")
with prepare_sandbox(clone=clone, claude_bin=sys.executable, backend="host-unsafe") as sandbox:
output = proposer_sandbox.prepare_review_workspace(sandbox, "review-output.json")
assert (clone / "review-output.json").read_text() == "repository content"
assert clone not in output.parents
def test_review_preparation_creates_a_private_directory_and_not_the_file(tmp_path):
clone = tmp_path / "clone"
clone.mkdir()
with prepare_sandbox(clone=clone, claude_bin=sys.executable, backend="host-unsafe") as sandbox:
output = proposer_sandbox.prepare_review_workspace(sandbox, "review-output.json")
assert output == proposer_sandbox.review_output_path(sandbox, "review-output.json")
# The DIRECTORY is what has to exist and be writable: the agent writes
# a temp file beside the target and renames it.
assert output.parent.is_dir()
assert stat.S_IMODE(output.parent.stat().st_mode) == 0o700
# The file is deliberately absent — absence is how "never written" is
# told apart from "written badly".
assert not output.exists()
def test_review_preparation_preserves_existing_runtime_files_and_tracks_only_created_paths(tmp_path):
clone = tmp_path / "clone"
clone.mkdir()
(clone / "bunfig.toml").write_text("existing configuration\n")
with prepare_sandbox(clone=clone, claude_bin=sys.executable, backend="host-unsafe") as sandbox:
proposer_sandbox.prepare_review_workspace(replace(sandbox, backend="bwrap"), "review-output.json")
created = json.loads((sandbox.private_root / "review-created-paths.json").read_text())
assert "bunfig.toml" not in created
assert ".npmrc" in created
assert ".mcp.json" in created
assert json.loads((clone / ".mcp.json").read_text()) == {}
assert (clone / "bunfig.toml").read_text() == "existing configuration\n"
assert (clone / ".git/commondir").read_text() == ".\n"
def test_review_preparation_rejects_a_runtime_symlink_parent(tmp_path):
clone = tmp_path / "clone"
clone.mkdir()
outside = tmp_path / "outside"
outside.mkdir()
(clone / ".claude").symlink_to(outside, target_is_directory=True)
with prepare_sandbox(clone=clone, claude_bin=sys.executable, backend="host-unsafe") as sandbox:
with pytest.raises(SandboxError):
proposer_sandbox.prepare_review_workspace(replace(sandbox, backend="bwrap"), "review-output.json")
assert list(outside.iterdir()) == []
def test_environment_is_allowlisted_and_shell_children_are_credential_free(monkeypatch) -> None:
monkeypatch.setenv("AWS_SECRET_ACCESS_KEY", "cloud-secret")
monkeypatch.setenv("GITHUB_TOKEN", "github-secret")
@ -65,6 +158,11 @@ def test_environment_is_allowlisted_and_shell_children_are_credential_free(monke
assert settings["sandbox"]["failIfUnavailable"] is True
assert settings["sandbox"]["allowUnsandboxedCommands"] is False
assert settings["sandbox"]["network"]["deniedDomains"] == ["*"]
assert SANDBOX_EVIDENCE in settings["sandbox"]["filesystem"]["allowRead"]
# Headless `claude -p` (2.1.247) never dispatches PreToolUse from any
# settings source, so a hook here would be confinement theater: it would
# read as a control in review while enforcing nothing at runtime.
assert "hooks" not in settings
# ENV_SCRUB forces "default" mode; the proposer's tools (Bash writes the
# overlay) run headless only because they are explicitly pre-approved.
# Requesting a non-default defaultMode would merely warn, so it must be gone.
@ -72,6 +170,169 @@ def test_environment_is_allowlisted_and_shell_children_are_credential_free(monke
assert "defaultMode" not in settings["permissions"]
def test_unsafe_host_session_translates_virtual_paths_and_disables_containment(tmp_path) -> None:
clone = tmp_path / "clone"
evidence = tmp_path / "evidence"
for directory in (clone, evidence):
directory.mkdir()
with prepare_sandbox(
clone=clone,
backend="host-unsafe",
claude_bin=sys.executable,
read_only_mounts=(ReadOnlyMount(evidence, "/evidence"),),
) as sandbox:
assert sandbox.command_prefix == []
assert sandbox.require_pid_namespace is False
assert sandbox.host_path("/workspace/review-output.json") == str(clone / "review-output.json")
assert sandbox.host_path("/evidence/selected-rows.json") == str(evidence / "selected-rows.json")
# The review artifact left the workspace, so the host-unsafe backend has
# to translate its new home too. Untranslated, the review prompt names a
# path that exists on neither backend and the cell writes nothing.
assert sandbox.host_path(
f"{proposer_sandbox.SANDBOX_REVIEW_OUTPUT}/review-output.json"
) == str(proposer_sandbox.review_output_path(sandbox, "review-output.json"))
assert sandbox.host_text(
f"write {proposer_sandbox.SANDBOX_REVIEW_OUTPUT}/review-output.json"
) == f"write {proposer_sandbox.review_output_path(sandbox, 'review-output.json')}"
assert sandbox.host_text("read /evidence and write /workspace/out") == (
f"read {evidence} and write {clone}/out"
)
# Sessions spawn the binary directly, so it must be the host executable
# rather than the sandbox-only mount target.
assert sandbox.claude_bin != SANDBOX_CLAUDE
assert Path(sandbox.claude_bin).exists()
assert sandbox.environment()["HOME"] == str(sandbox.home)
assert "CLAUDE_CODE_SUBPROCESS_ENV_SCRUB" not in sandbox.environment()
assert all(
Path(entry).is_dir() for entry in sandbox.environment()["PATH"].split(":")
)
unsafe_settings = json.loads(sandbox.settings_json)
assert unsafe_settings["sandbox"]["enabled"] is False
assert unsafe_settings["sandbox"]["failIfUnavailable"] is False
assert "disableBypassPermissionsMode" not in unsafe_settings["permissions"]
def test_host_workspace_write_boundary_keeps_only_the_review_artifact_writable(tmp_path) -> None:
clone = tmp_path / "clone"
nested = clone / "src"
nested.mkdir(parents=True)
source = nested / "source.ts"
source.write_text("trusted\n")
output = clone / "review-output.json"
output.write_text("")
original_source_mode = stat.S_IMODE(source.stat().st_mode)
original_output_mode = stat.S_IMODE(output.stat().st_mode)
with host_workspace_write_boundary(clone, writable=(output,)):
with pytest.raises(OSError):
source.write_text("tampered\n")
with pytest.raises(OSError):
(clone / "extra.py").write_text("nope\n")
output.write_text('{"schema_version":1}\n')
assert source.read_text() == "trusted\n"
assert output.read_text() == '{"schema_version":1}\n'
assert not (clone / "extra.py").exists()
assert stat.S_IMODE(source.stat().st_mode) == original_source_mode
assert stat.S_IMODE(output.stat().st_mode) == original_output_mode
def test_host_workspace_write_boundary_keeps_files_under_an_allowed_directory(tmp_path) -> None:
clone = tmp_path / "clone"
artifacts = clone / "artifacts"
artifacts.mkdir(parents=True)
existing = artifacts / "review-output.json"
existing.write_text("{}\n")
(clone / "src").mkdir()
locked = clone / "src" / "source.ts"
locked.write_text("trusted\n")
with host_workspace_write_boundary(clone, writable=(artifacts,)):
existing.write_text('{"schema_version":1}\n')
with pytest.raises(OSError):
locked.write_text("tampered\n")
assert existing.read_text() == '{"schema_version":1}\n'
assert locked.read_text() == "trusted\n"
def test_host_workspace_write_boundary_rejects_a_symlinked_writable_artifact(tmp_path) -> None:
clone = tmp_path / "clone"
clone.mkdir()
target = tmp_path / "outside.json"
target.write_text("{}\n")
output = clone / "review-output.json"
output.symlink_to(target)
with pytest.raises(SandboxError, match="non-symlink"):
with host_workspace_write_boundary(clone, writable=(output,)):
pass
def test_sandbox_workspace_write_boundary_is_noop_unless_host_unsafe(tmp_path) -> None:
clone = tmp_path / "clone"
clone.mkdir()
source = clone / "source.ts"
source.write_text("trusted\n")
bwrap_sandbox = SimpleNamespace(backend="bwrap", clone=clone)
with sandbox_workspace_write_boundary(
bwrap_sandbox,
read_only_workspace=True,
writable=(),
):
source.write_text("still writable under bwrap no-op\n")
assert source.read_text() == "still writable under bwrap no-op\n"
output = clone / "review-output.json"
output.write_text("")
source.write_text("trusted\n")
unsafe = SimpleNamespace(backend="host-unsafe", clone=clone)
with sandbox_workspace_write_boundary(
unsafe,
read_only_workspace=True,
writable=(output,),
):
with pytest.raises(OSError):
source.write_text("tampered\n")
output.write_text("ok\n")
assert source.read_text() == "trusted\n"
assert output.read_text() == "ok\n"
def test_force_rmtree_deletes_nonempty_directories_copied_from_a_locked_workspace(tmp_path) -> None:
locked = tmp_path / "locked"
nested = locked / "gitnexus-shared" / "src"
nested.mkdir(parents=True)
(nested / "index.ts").write_text("export {}\n")
os.chmod(nested, 0o500)
os.chmod(locked / "gitnexus-shared", 0o500)
os.chmod(locked, 0o500)
copied = tmp_path / "sandbox-tmp" / "tmp.XXXX" / "gitnexus-shared"
copied.parent.mkdir(parents=True)
shutil.copytree(locked / "gitnexus-shared", copied)
assert stat.S_IMODE(copied.stat().st_mode) & 0o222 == 0
_force_rmtree(copied.parent)
assert not copied.parent.exists()
def test_host_unsafe_sandbox_cleanup_survives_readonly_tmpdir_copies(tmp_path) -> None:
clone = tmp_path / "clone"
clone.mkdir()
leftover = None
with prepare_sandbox(clone=clone, backend="host-unsafe", claude_bin=sys.executable) as sandbox:
leftover = sandbox.private_root
copied = sandbox.temp / "tmp.XXXX" / "gitnexus-shared" / "src"
copied.mkdir(parents=True)
(copied / "index.ts").write_text("export {}\n")
os.chmod(copied, 0o500)
os.chmod(copied.parent, 0o500)
os.chmod(copied.parent.parent, 0o500)
assert leftover is not None
assert not leftover.exists()
@pytest.mark.parametrize(
"bad_url",
["https://user:secret@example.test", "https://example.test/path?token=x", "file:///tmp/model"],
@ -170,6 +431,16 @@ def test_sandbox_command_has_minimal_mounts_and_no_host_root_bind(tmp_path: Path
assert probe.returncode == 0, probe.stderr
assert probe.stdout == f"/home/agent|{SANDBOX_PATH}"
gitnexus_index = argv.index(SANDBOX_GITNEXUS_CLI)
gitnexus_wrapper = Path(argv[gitnexus_index - 1])
assert stat.S_IMODE(gitnexus_wrapper.stat().st_mode) == 0o500
assert "/opt/gitnexus/dist/cli/index.js" in gitnexus_wrapper.read_text()
excludes_index = argv.index(SANDBOX_GIT_EXCLUDES)
excludes = Path(argv[excludes_index - 1])
assert stat.S_IMODE(excludes.stat().st_mode) == 0o400
assert "/.bash_profile" in excludes.read_text().splitlines()
# The evidence-provenance.mjs plan-writer's PATH-scan trusts a Python 3
# candidate only if it (and its directory) is owned by root or by the
# current process — real /usr/bin/python3 is root-owned on the host,
@ -617,6 +888,63 @@ finally:
assert not (clone / "oracle-leak.txt").exists()
@pytest.mark.skipif(
os.environ.get("GITNEXUS_REQUIRE_BWRAP_CANARY") != "1",
reason="real Bubblewrap canary is mandatory in the named Ubuntu CI job",
)
def test_read_only_review_workspace_exposes_only_one_writable_artifact(tmp_path: Path) -> None:
clone = tmp_path / "clone"
clone.mkdir()
source = clone / "source.ts"
source.write_text("trusted\n")
script = """
import os
from pathlib import Path
try:
Path('/workspace/source.ts').write_text('tampered')
except OSError:
pass
else:
raise SystemExit('review source remained writable')
# Write the way the agent's Write tool does: a temp file beside the target,
# then rename. Writing in place would pass against the mount shape that
# shipped every artifact empty, which is the regression this canary exists for.
target = Path('/review-output/review-output.json')
staging = target.with_name(target.name + '.tmp.1.abc')
staging.write_text('{"schema_version":1}')
os.replace(staging, target)
"""
with prepare_sandbox(clone=clone, claude_bin=Path(sys.executable), preflight=True) as sandbox:
output = proposer_sandbox.prepare_review_workspace(sandbox, "review-output.json")
result = run_managed(
[
*sandbox.command_prefix_for(
read_only_workspace=True,
extra_writable_mounts=(
ReadOnlyMount(
source=output.parent, target=proposer_sandbox.SANDBOX_REVIEW_OUTPUT
),
),
),
"/usr/bin/python3",
"-c",
script,
],
timeout=10,
env=sandbox.environment(),
require_pid_namespace=True,
)
# Inside the sandbox scope: the artifact now lives under the session's
# private root, which prepare_sandbox removes on exit. run_arm reads it
# here too, while the session is still alive.
assert result.ok, result.stderr_tail
assert source.read_text() == "trusted\n"
assert output.read_text() == '{"schema_version":1}'
# The staging file is gone: the rename landed rather than a copy.
assert list(output.parent.iterdir()) == [output]
@pytest.mark.skipif(os.name == "nt", reason="symlink creation may require elevated Windows privileges")
@pytest.mark.parametrize("operation", ["stage", "sandbox"])
def test_clone_root_symlink_is_rejected_before_host_access(tmp_path: Path, operation: str) -> None:
@ -839,13 +1167,15 @@ def test_clone_controlled_mcp_replacement_is_never_executed_or_credentialed(tmp_
os.environ.get("GITNEXUS_REQUIRE_CLAUDE_CANARY") != "1",
reason="real Claude/Bash/MCP canary is mandatory in the named Ubuntu CI job",
)
def test_real_claude_bare_auth_inner_sandbox_and_mcp_permissions(tmp_path: Path) -> None:
@pytest.mark.parametrize("review_layout", [False, True])
def test_real_claude_auth_inner_sandbox_and_mcp_permissions(tmp_path: Path, review_layout: bool) -> None:
"""Exercise the exact CLI boundary without contacting a paid model."""
claude = Path(os.environ["CLAUDE_CANARY_BIN"]).resolve()
assert claude.is_file()
clone = tmp_path / "clone"
clone.mkdir()
(clone / "canary.txt").write_text("hook-readable")
fake_mcp = clone / "fake_mcp.py"
fake_mcp.write_text(
"""import json
@ -872,7 +1202,7 @@ for line in sys.stdin:
}]
}
elif method == "tools/call":
Path("/workspace/mcp-called").write_text("ok")
Path("/tmp/mcp-called").write_text("ok")
result = {"content": [{"type": "text", "text": "repository list ready"}]}
else:
result = {}
@ -881,6 +1211,35 @@ for line in sys.stdin:
)
fake_mcp.chmod(0o500)
review_command = """test -z "${ANTHROPIC_API_KEY:-}" && python3 - <<'PY'
import json
import os
import subprocess
from pathlib import Path
source = Path('/workspace/canary.txt')
assert 'hook-readable' in source.read_text()
assert 'changed for review' in subprocess.check_output(['git', 'diff', '--', 'canary.txt'], text=True)
for operation in (lambda: source.write_text('forbidden'), lambda: source.rename(source.with_name('renamed')), source.unlink):
try:
operation()
except OSError:
pass
else:
raise AssertionError('source mutation was allowed')
target = Path('/review-output/review-output.json')
staging = target.with_name(target.name + '.tmp.1.abc')
staging.write_text(json.dumps({'schema_version': 1, 'verdict': 'approve', 'findings': []}))
os.replace(staging, target)
PY"""
if review_layout:
for command in (
["git", "init", "-q"],
["git", "add", "canary.txt", "fake_mcp.py"],
["git", "-c", "user.name=Canary", "-c", "user.email=canary@example.test", "commit", "-qm", "fixture"],
):
subprocess.run(command, cwd=clone, check=True, capture_output=True)
(clone / "canary.txt").write_text("hook-readable\nchanged for review\n")
observed_tool_results: dict[str, dict] = {}
class ModelHandler(BaseHTTPRequestHandler):
@ -910,7 +1269,19 @@ for line in sys.stdin:
and isinstance(block.get("tool_use_id"), str)
}
)
if "toolu_mcp_canary" not in tool_result_ids:
if "toolu_read_canary" not in tool_result_ids:
blocks = [
{
"type": "tool_use",
"id": "toolu_read_canary",
"name": "Read",
"input": {
"file_path": "/workspace/canary.txt",
},
}
]
stop_reason = "tool_use"
elif "toolu_mcp_canary" not in tool_result_ids:
blocks = [
{
"type": "tool_use",
@ -927,7 +1298,9 @@ for line in sys.stdin:
"id": "toolu_bash_canary",
"name": "Bash",
"input": {
"command": ('test -z "${ANTHROPIC_API_KEY:-}" && printf canary > /workspace/bash-called')
"command": review_command
if review_layout
else ('test -z "${ANTHROPIC_API_KEY:-}" && printf canary > /workspace/bash-called')
},
}
]
@ -1024,6 +1397,22 @@ for line in sys.stdin:
}
)
with prepare_sandbox(clone=clone, claude_bin=claude, preflight=True) as sandbox:
output = None
before = {}
if review_layout:
output = proposer_sandbox.prepare_review_workspace(sandbox, "review-output.json")
before = runner_artifacts.workspace_snapshot(clone)
sandbox = replace(
sandbox,
command_prefix=sandbox.command_prefix_for(
read_only_workspace=True,
extra_writable_mounts=(
ReadOnlyMount(
output.parent, proposer_sandbox.SANDBOX_REVIEW_OUTPUT
),
),
),
)
result = sandbox.run(
[
sandbox.claude_bin,
@ -1032,7 +1421,6 @@ for line in sys.stdin:
"text",
"--output-format",
"json",
"--bare",
"--settings",
sandbox.settings_json,
"--strict-mcp-config",
@ -1044,7 +1432,12 @@ for line in sys.stdin:
# authoritative empirical gate for that behavior.
"--model",
"claude-canary-20260718",
"--tools",
"Read",
"Bash",
"mcp__gitnexus__list_repos",
"--allowedTools",
"Read",
"Bash",
"mcp__gitnexus__list_repos",
],
@ -1053,17 +1446,121 @@ for line in sys.stdin:
auth_token="offline-canary-key",
base_url=f"http://127.0.0.1:{server.server_port}",
),
stdin_data=b"Use both available tools, then finish.",
stdin_data=b"Use all three available tools, then finish.",
)
assert result.ok, result.stderr_tail + result.stdout_tail
report = json.loads(result.stdout_tail)
assert report["subtype"] == "success" and report["is_error"] is False, report
read_result = observed_tool_results["toolu_read_canary"]
assert read_result.get("is_error") is not True, read_result
assert "hook-readable" in json.dumps(read_result)
bash_result = observed_tool_results["toolu_bash_canary"]
assert bash_result.get("is_error") is not True, bash_result
assert (sandbox.temp / "mcp-called").read_text() == "ok"
if review_layout:
assert output is not None
runner_artifacts.enforce_phase_workspace(clone, before, allowed_artifact=None)
assert json.loads(output.read_text())["verdict"] == "approve"
assert (clone / "canary.txt").read_text() == "hook-readable\nchanged for review\n"
finally:
server.shutdown()
server.server_close()
thread.join(timeout=5)
assert result.ok, result.stderr_tail + result.stdout_tail
report = json.loads(result.stdout_tail)
assert report["subtype"] == "success" and report["is_error"] is False, report
bash_result = observed_tool_results["toolu_bash_canary"]
assert bash_result.get("is_error") is not True, bash_result
assert (clone / "bash-called").read_text() == "canary"
assert (clone / "mcp-called").read_text() == "ok"
if not review_layout:
assert (clone / "bash-called").read_text() == "canary"
def test_review_artifact_binds_a_writable_directory_outside_the_workspace(tmp_path):
"""The bwrap argv, since the mount shape is the whole bug.
bwrap cannot create a mount point inside an already-read-only bind, so a
writable path has to live outside /workspace — and it has to be the
directory, or the agent has nowhere to put the temp file it renames into
place.
"""
clone = tmp_path / "clone"
clone.mkdir()
with prepare_sandbox(clone=clone, claude_bin=sys.executable, backend="host-unsafe") as session:
sandbox = replace(session, backend="bwrap")
output = proposer_sandbox.review_output_path(sandbox, "review-output.json")
output.parent.mkdir(mode=0o700)
argv = sandbox.command_prefix_for(
read_only_workspace=True,
extra_writable_mounts=(
proposer_sandbox.ReadOnlyMount(
source=output.parent,
target=proposer_sandbox.SANDBOX_REVIEW_OUTPUT,
),
),
)
target = proposer_sandbox.SANDBOX_REVIEW_OUTPUT
assert not target.startswith(proposer_sandbox.SANDBOX_WORKSPACE + "/")
# The workspace itself is bound read-only...
workspace_at = argv.index(proposer_sandbox.SANDBOX_WORKSPACE)
assert argv[workspace_at - 2] == "--ro-bind"
# ...and the artifact directory is bound writable, as a directory.
artifact_at = argv.index(target)
assert argv[artifact_at - 2] == "--bind"
assert Path(argv[artifact_at - 1]) == output.parent
assert Path(argv[artifact_at - 1]).is_dir()
assert f"{proposer_sandbox.SANDBOX_WORKSPACE}/review-output.json" not in argv
@pytest.mark.skipif(
os.environ.get("GITNEXUS_REQUIRE_BWRAP_CANARY") != "1",
reason="real Bubblewrap canary is mandatory in the named Ubuntu CI job",
)
def test_real_bubblewrap_lets_a_review_artifact_be_written_atomically(tmp_path: Path) -> None:
"""The filesystem contract the EROFS defect broke, under a real sandbox.
Argv assertions cannot establish this. The artifact came back empty because
an atomic write - temp file beside the target, then rename - needs a
WRITABLE PARENT DIRECTORY, and only a real bwrap invocation shows whether
the mount grants one. A deterministic writer stands in for the agent: no
model session, no credentials.
Scope: this proves the filesystem and process contract of the production
mount configuration. It does not establish that a particular agent CLI's
own file-access policy permits the same operation - that is a second,
independent gate.
"""
clone = tmp_path / "clone"
clone.mkdir()
(clone / "tracked.txt").write_text("original\n")
with prepare_sandbox(clone=clone, claude_bin=Path(sys.executable), preflight=True) as sandbox:
review_output = prepare_review_workspace(sandbox, REVIEW_OUTPUT)
# The production configuration, not a hand-built mount tuple: the same
# command_prefix_for call run_arm makes for a review cell.
prefix = sandbox.command_prefix_for(
read_only_workspace=True,
extra_writable_mounts=(
ReadOnlyMount(source=review_output.parent, target=SANDBOX_REVIEW_OUTPUT),
),
)
target = f"{SANDBOX_REVIEW_OUTPUT}/{REVIEW_OUTPUT}"
script = (
# 1. temp file beside the destination, then atomic rename over it.
f'printf %s \'{{"schema_version": 1, "verdict": "approve", "findings": []}}\' > {target}.tmp && '
f"mv {target}.tmp {target} && "
# 2. the workspace must refuse the write that the mount forbids.
f"(printf x >> {SANDBOX_WORKSPACE}/tracked.txt 2>/dev/null && echo WORKSPACE-WRITABLE || echo workspace-readonly)"
)
result = subprocess.run(
[*prefix, "/bin/sh", "-c", script],
capture_output=True, text=True, timeout=60, check=False,
)
assert result.returncode == 0, f"atomic write failed inside the sandbox: {result.stderr[-400:]}"
assert "workspace-readonly" in result.stdout, "the workspace must stay read-only"
assert (clone / "tracked.txt").read_text() == "original\n", "the clone was modified"
# Read while the session is alive: the artifact lives under the private
# root that prepare_sandbox removes on exit, which is also why run_arm
# consumes it before leaving the scope.
_verdict, findings = parse_review_output(review_output)
assert findings == ()

132
eval/tests/test_provider_usage.py Normal file
View file

@ -0,0 +1,132 @@
"""The two providers' accounting equations, encoded literally.
Adding OpenAI's cache fields to its input_tokens double-counts, because they are
subsets of it. Subtracting Anthropic's under-counts, because they are additional
categories. A single generic struct cannot be right for both, so these tests
pin each equation rather than the field names.
"""
from __future__ import annotations
import pytest
from workflow_bench.provider_usage import (
ANTHROPIC,
OPENAI_RESPONSES,
UsageSemanticsError,
normalize_usage,
)
def _openai(input_tokens: int, cached: int | None = None, cache_write: int | None = None) -> dict:
details: dict[str, int] = {}
if cached is not None:
details["cached_tokens"] = cached
if cache_write is not None:
details["cache_write_tokens"] = cache_write
return {
"input_tokens": input_tokens,
"input_tokens_details": details,
"output_tokens": 300,
"output_tokens_details": {"reasoning_tokens": 250},
}
def test_openai_uncached_request_is_all_ordinary_input() -> None:
usage = normalize_usage(OPENAI_RESPONSES, _openai(1000, cached=0, cache_write=0))
assert usage.ordinary_input_tokens == 1000
assert usage.total_input_tokens == 1000
assert (usage.cache_read_input_tokens, usage.cache_write_input_tokens) == (0, 0)
def test_openai_cache_creation_keeps_the_parts_summing_to_input_tokens() -> None:
"""The subsets must reconstruct the whole, never exceed it."""
usage = normalize_usage(OPENAI_RESPONSES, _openai(1000, cached=0, cache_write=400))
assert usage.ordinary_input_tokens == 600
assert (
usage.ordinary_input_tokens
+ usage.cache_read_input_tokens
+ usage.cache_write_input_tokens
== usage.total_input_tokens
)
def test_openai_cache_hit_plus_new_write_uses_the_documented_subtraction() -> None:
usage = normalize_usage(OPENAI_RESPONSES, _openai(10_000, cached=7_000, cache_write=1_000))
assert usage.ordinary_input_tokens == 2_000
assert usage.total_input_tokens == 10_000, "input_tokens is the whole, not a component"
def test_openai_reasoning_tokens_decompose_output_rather_than_adding_to_it() -> None:
usage = normalize_usage(OPENAI_RESPONSES, _openai(100, cached=0, cache_write=0))
assert usage.output_tokens == 300
assert usage.reasoning_output_tokens == 250
assert usage.reasoning_output_tokens <= usage.output_tokens
def test_anthropic_uncached_total_is_just_input_tokens() -> None:
usage = normalize_usage(
ANTHROPIC,
{"input_tokens": 1000, "cache_creation_input_tokens": 0,
"cache_read_input_tokens": 0, "output_tokens": 200},
)
assert usage.total_input_tokens == 1000
assert usage.ordinary_input_tokens == 1000
def test_anthropic_cached_total_adds_the_cache_categories() -> None:
"""The opposite equation to OpenAI's, on deliberately identical numbers."""
usage = normalize_usage(
ANTHROPIC,
{"input_tokens": 2_000, "cache_creation_input_tokens": 1_000,
"cache_read_input_tokens": 7_000, "output_tokens": 200},
)
assert usage.total_input_tokens == 10_000
assert usage.ordinary_input_tokens == 2_000
def test_the_same_numbers_mean_different_totals_on_the_two_providers() -> None:
"""The whole reason a shared struct is unsafe, in one assertion."""
openai = normalize_usage(OPENAI_RESPONSES, _openai(10_000, cached=7_000, cache_write=1_000))
anthropic = normalize_usage(
ANTHROPIC,
{"input_tokens": 10_000, "cache_creation_input_tokens": 1_000,
"cache_read_input_tokens": 7_000, "output_tokens": 300},
)
assert openai.total_input_tokens == 10_000
assert anthropic.total_input_tokens == 18_000
assert openai.ordinary_input_tokens == 2_000
assert anthropic.ordinary_input_tokens == 10_000
def test_missing_native_cache_fields_are_unknown_and_never_zero() -> None:
"""A zero we invented is indistinguishable from a zero the provider reported."""
usage = normalize_usage(OPENAI_RESPONSES, {"input_tokens": 1000, "output_tokens": 10})
assert usage.cache_read_input_tokens is None
assert usage.cache_write_input_tokens is None
assert usage.ordinary_input_tokens is None, "cannot subtract what was never reported"
assert usage.total_input_tokens == 1000
assert not usage.complete
assert "cache_read_input_tokens" in usage.unknown_fields
def test_an_absent_usage_object_is_entirely_unknown() -> None:
usage = normalize_usage(ANTHROPIC, None)
assert not usage.complete
assert usage.total_input_tokens is None
def test_an_unknown_provider_is_refused_rather_than_guessed() -> None:
with pytest.raises(UsageSemanticsError, match="refusing to guess"):
normalize_usage("some-new-provider", {"input_tokens": 1})
def test_cache_subsets_larger_than_the_whole_are_rejected() -> None:
"""Nonsense arithmetic must surface, not silently produce a negative."""
with pytest.raises(UsageSemanticsError, match="exceed input_tokens"):
normalize_usage(OPENAI_RESPONSES, _openai(100, cached=90, cache_write=50))

327
eval/tests/test_provider_usage_capture.py Normal file
View file

@ -0,0 +1,327 @@
"""What the proxy writes must outlive the translation that follows it.
Claude Code receives an Anthropic-shaped response, which has nowhere to put
OpenAI's cached_tokens, cache_write_tokens or reasoning_tokens. If those are not
captured before the translation, the only remaining record of them is a bill.
"""
from __future__ import annotations
import contextlib
import json
from pathlib import Path
from types import SimpleNamespace
import pytest
from workflow_bench import litellm_usage_callback, provider_usage
from workflow_bench.litellm_usage_callback import USAGE_LOG_ENV_VAR, ProviderUsageLogger
from workflow_bench.model_gateway import (
OpenAIGateway,
USAGE_CALLBACK_MODULE,
openai_litellm_config,
write_openai_litellm_config,
)
from workflow_bench.provider_usage import (
ANTHROPIC,
LITELLM_NORMALIZED,
USAGE_ENV_VARS,
normalize_usage,
)
class _Usage:
"""Stands in for the provider usage model LiteLLM hands the callback."""
def __init__(self, payload: dict) -> None:
self._payload = payload
def model_dump(self) -> dict:
return self._payload
def _openai_response(usage: dict) -> SimpleNamespace:
return SimpleNamespace(
id="resp_68f2c1",
# The model that actually answered, which is not the role the caller asked for.
model="gpt-5.6-sol-2026-08-01",
usage=_Usage(usage),
)
# The shape a callback actually receives: LiteLLM normalises usage into its own
# Chat-Completions-style object before any logger sees it, so an OpenAI reply
# arrives as prompt_tokens / prompt_tokens_details. Confirmed against a real
# proxy in tests/test_mock_provider.py; a fixture in the wire shape would test
# an object this code path never gets.
NATIVE = {
"prompt_tokens": 48_000,
"prompt_tokens_details": {"cached_tokens": 44_000, "cache_write_tokens": 1_000},
"completion_tokens": 900,
"completion_tokens_details": {"reasoning_tokens": 640},
}
@pytest.fixture
def logged(tmp_path: Path, monkeypatch: pytest.MonkeyPatch):
log = tmp_path / "provider_usage.jsonl"
monkeypatch.setenv(USAGE_LOG_ENV_VAR, str(log))
def emit(usage: dict) -> dict:
ProviderUsageLogger()._append(
"success",
{"model": "claude-sonnet-4-5", "custom_llm_provider": "openai", "call_type": "responses"},
_openai_response(usage),
0.0,
1.0,
)
return json.loads(log.read_text().splitlines()[-1])
return emit
def test_native_openai_usage_survives_the_anthropic_translation(logged) -> None:
event = logged(NATIVE)
native = event["native_usage"]
# Verbatim: the fields an Anthropic-shaped response cannot carry.
assert native["prompt_tokens_details"]["cached_tokens"] == 44_000
assert native["prompt_tokens_details"]["cache_write_tokens"] == 1_000
assert native["completion_tokens_details"]["reasoning_tokens"] == 640
assert event["response_id"] == "resp_68f2c1"
def test_the_actual_model_is_recorded_separately_from_the_requested_role(logged) -> None:
"""Pricing must follow what answered, not what the caller named."""
event = logged(NATIVE)
assert event["requested_model"] == "claude-sonnet-4-5"
assert event["actual_model"] == "gpt-5.6-sol-2026-08-01"
assert "cell_id" not in event, "a proxy-wide variable cannot identify a cell"
def test_the_captured_event_normalizes_with_openai_arithmetic(logged) -> None:
"""Capture and normalization must agree end to end, not just in isolation."""
event = logged(NATIVE)
# The provider the LOG recorded, not one the test supplies - passing
# OPENAI_RESPONSES by hand here is what hid the adapter-key mismatch.
# LITELLM_NORMALIZED, not OPENAI_RESPONSES: a proxy callback never sees the
# upstream body. Measured against a real gateway - the Responses adapter
# found none of its keys there and reported every field unknown.
assert event["provider"] == LITELLM_NORMALIZED
assert event["provider_label"] == "openai"
usage = normalize_usage(event["provider"], event["native_usage"])
assert usage.total_input_tokens == 48_000
assert usage.ordinary_input_tokens == 3_000
assert usage.cache_read_input_tokens == 44_000
assert usage.complete
def test_usage_without_details_normalizes_to_unknown_rather_than_zero(logged) -> None:
"""The mutation the accounting must not survive: dropped details, silent zeros."""
stripped = {k: v for k, v in NATIVE.items() if k != "prompt_tokens_details"}
event = logged(stripped)
usage = normalize_usage(event["provider"], event["native_usage"])
assert usage.cache_read_input_tokens is None
assert usage.ordinary_input_tokens is None
assert not usage.complete
def test_a_failed_request_is_still_accounted_for(logged, tmp_path: Path) -> None:
"""The money was spent whether or not the cell produced an artifact."""
import asyncio
logger = ProviderUsageLogger()
args = ({"model": "claude-sonnet-4-5"}, _openai_response(NATIVE), 0.0, 1.0)
# Every hook LiteLLM can call, not the private helper underneath them: the
# sync failure hook was missing entirely and _append could never show that.
logger.log_failure_event(*args)
asyncio.run(logger.async_log_failure_event(*args))
events = [json.loads(line) for line in (tmp_path / "provider_usage.jsonl").read_text().splitlines()]
assert len(events) == 2, "both failure hooks must record"
assert all(e["status"] == "failure" for e in events)
def test_every_public_outcome_hook_records(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Overriding a subset silently drops whichever path LiteLLM actually uses."""
import asyncio
monkeypatch.setenv(USAGE_LOG_ENV_VAR, str(tmp_path / "usage.jsonl"))
logger = ProviderUsageLogger()
args = ({"model": "m"}, _openai_response(NATIVE), 0.0, 1.0)
logger.log_success_event(*args)
logger.log_failure_event(*args)
asyncio.run(logger.async_log_success_event(*args))
asyncio.run(logger.async_log_failure_event(*args))
events = [json.loads(line) for line in (tmp_path / "usage.jsonl").read_text().splitlines()]
assert [e["status"] for e in events] == ["success", "failure", "success", "failure"]
def test_the_logger_never_raises_into_the_proxy(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Accounting is evidence, not control flow."""
monkeypatch.setenv(USAGE_LOG_ENV_VAR, str(tmp_path / "missing-dir" / "usage.jsonl"))
ProviderUsageLogger()._append("success", {}, object(), 0.0, 1.0)
def test_no_log_is_written_when_the_destination_is_unset(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv(USAGE_LOG_ENV_VAR, raising=False)
ProviderUsageLogger()._append("success", {}, _openai_response(NATIVE), 0.0, 1.0)
assert not list(tmp_path.iterdir())
def test_the_generated_config_loads_the_callback_from_beside_itself(tmp_path: Path) -> None:
"""LiteLLM resolves the dotted path relative to the config directory."""
config = write_openai_litellm_config(tmp_path / "litellm.yaml", ["gpt-5.6-sol"])
assert openai_litellm_config(["gpt-5.6-sol"])["litellm_settings"]["callbacks"] == [
f"{USAGE_CALLBACK_MODULE}.handler"
]
installed = config.parent / f"{USAGE_CALLBACK_MODULE}.py"
assert installed.is_file(), "the proxy cannot import a callback that was never placed"
# Importing it, not grepping it: a text search passes even when the module
# cannot load, which is exactly how a package-relative import survived
# review here. This is the deployment configuration, so load it the way the
# proxy does - by path, as a top-level module.
import importlib.util
spec = importlib.util.spec_from_file_location(USAGE_CALLBACK_MODULE, installed)
assert spec is not None and spec.loader is not None
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
assert isinstance(module.handler, module.ProviderUsageLogger)
def test_the_gateway_forwards_the_usage_environment_into_the_proxy(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The proxy is a separate process with a constructed environment.
Popen(env=...) replaces the parent environment rather than extending it, so
a variable the callback reads is simply absent unless the gateway forwards
it by name. Without this the accounting looks configured and silently
records nothing on every request - the in-process tests above cannot see
that, because they never cross the subprocess boundary.
"""
for name in USAGE_ENV_VARS:
monkeypatch.setenv(name, f"value-for-{name}")
captured: dict[str, dict[str, str]] = {}
class _Popen:
def __init__(self, *_a, **kwargs):
captured["env"] = kwargs["env"]
raise RuntimeError("stop before launching a real proxy")
# The console-script resolver runs before Popen and is absent in this
# environment (the same reason two gateway tests fail here); the argv it
# builds is not what this test is about.
monkeypatch.setattr(
"workflow_bench.model_gateway.litellm_proxy_argv",
lambda **_k: ["/bin/true"],
)
monkeypatch.setattr("workflow_bench.model_gateway.subprocess.Popen", _Popen)
gateway = OpenAIGateway(
openai_api_key="sk-test",
model_names=["gpt-5.6-sol"],
work_dir=tmp_path,
)
with contextlib.suppress(Exception):
gateway.__enter__()
env = captured.get("env")
assert env is not None, "the proxy was never constructed"
for name in USAGE_ENV_VARS:
assert env.get(name) == f"value-for-{name}", f"{name} never reached the proxy"
# The credential allowlist is still an allowlist, not the parent environment.
assert "PATH" in env and len(env) < 40
def test_an_unresolvable_provider_is_refused_rather_than_guessed() -> None:
"""LiteLLM says "openai" for Chat Completions too, and it counts differently."""
from workflow_bench.provider_usage import canonical_provider
# Every openai call reaching this callback has already been normalised by
# LiteLLM, whatever endpoint it used - the observed call_type for a Claude
# Code request through the gateway is "anthropic_messages". The adapter has
# to match the object in hand, not the protocol on the wire.
assert canonical_provider("openai", "responses") == LITELLM_NORMALIZED
assert canonical_provider("openai", "anthropic_messages") == LITELLM_NORMALIZED
assert canonical_provider("anthropic", "completion") == ANTHROPIC
# An unrecognised provider is still refused rather than guessed.
assert canonical_provider("some-new-provider", "responses") is None
assert canonical_provider(None, None) is None
def test_request_identity_cannot_come_from_the_proxy_environment() -> None:
"""One proxy serves the whole sweep, so its environment identifies the sweep.
attach_openai_gateway wraps all of _run_sweep, and cells run concurrently
under --workers, interleaving requests through that single process. Any
variable forwarded at launch is therefore constant for every event it ever
records. Pinned so a future change does not reintroduce a per-cell
environment variable that would silently stamp one value on every request.
"""
assert USAGE_ENV_VARS == (
"GITNEXUS_BENCH_PROVIDER_USAGE",
"GITNEXUS_BENCH_SWEEP_ID",
), "a per-cell variable here would be constant across concurrent cells"
def test_a_request_records_its_session_so_attribution_stays_possible(logged) -> None:
"""The per-request half of identity, recorded even when the provider omits it."""
event = logged(NATIVE)
assert "session_id" in event, "absent attribution is still a fact about the run"
def test_the_callback_imports_the_way_litellm_actually_loads_it(tmp_path: Path) -> None:
"""By path, as a top-level module, with no parent package and no sys.path entry.
LiteLLM resolves a dotted callback through spec_from_file_location against
the config directory, so the copied file is not part of workflow_bench when
it runs. A relative or sibling import therefore raises ImportError and the
proxy exits before becoming ready - which the in-package tests cannot see,
because they import it as workflow_bench.litellm_usage_callback.
"""
import importlib.util
import shutil
source = Path(litellm_usage_callback.__file__)
installed = tmp_path / f"{USAGE_CALLBACK_MODULE}.py"
shutil.copy(source, installed)
spec = importlib.util.spec_from_file_location(USAGE_CALLBACK_MODULE, installed)
assert spec is not None and spec.loader is not None
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module) # ImportError here is the proxy refusing to start
assert hasattr(module, "handler")
def test_the_callbacks_copied_constants_match_the_canonical_ones() -> None:
"""The copies are deliberate; drifting apart silently is not.
The callback cannot import from the package (see the test above), so it
carries its own literals. These assertions are what keep the duplication
honest.
"""
assert litellm_usage_callback.USAGE_LOG_ENV_VAR == provider_usage.USAGE_LOG_ENV_VAR
assert litellm_usage_callback.SWEEP_ID_ENV_VAR == provider_usage.SWEEP_ID_ENV_VAR
for label, call_type in (
("openai", "responses"),
("openai", "completion"),
("openai", None),
("anthropic", "completion"),
("mystery", "responses"),
):
assert litellm_usage_callback.canonical_provider(label, call_type) == provider_usage.canonical_provider(
label, call_type
), f"resolver drifted for {label!r}/{call_type!r}"

268
eval/tests/test_reuse_round_trip.py Normal file
View file

@ -0,0 +1,268 @@
"""A row the runner actually emits must satisfy the reuse reader.
Every existing comparator-reuse test builds its rows by hand. That proves the
predicate's logic and nothing about the producer: a fixture can satisfy
eligibility while a real emitted row never does, and the audit that counts key
names cannot tell the difference. These tests carry one record through the
production path instead:
real run_cell -> production JSONL writer -> load_result_rows
-> row_is_reusable_comparator
Only the expensive dependencies are replaced - the model session, sandbox
launch, repository acquisition, graph preparation. The digest fields the reuse
binding compares are assembled by run_cell itself from its TaskCellContext, so
they stay real: they are the subject of the test, not scaffolding around it.
"""
from __future__ import annotations
import json
from datetime import UTC, datetime, timedelta
from pathlib import Path
from types import SimpleNamespace
from typing import Any
import pytest
from workflow_bench import runner
from workflow_bench.proposer_sandbox import redact_text
from workflow_bench.model_gateway import credential_secrets
from workflow_bench.runner_sessions import PARENT_EVENT_STREAM_SOURCE
from workflow_bench.comparator_reuse import (
ComparatorReuseExpectation,
TaskReuseBinding,
load_result_rows,
row_is_reusable_comparator,
)
TASK_ID = "review-pr-2718-defect"
SHA = "a" * 40
def _snapshot(prefix: str) -> SimpleNamespace:
return SimpleNamespace(
digest=f"{prefix}-content",
manifest_digest=f"{prefix}-manifest",
dependency_content_digest=f"{prefix}-dep-content",
dependency_manifest_digest=f"{prefix}-dep-manifest",
command_digest=f"{prefix}-command",
materialize=lambda *a, **k: None,
)
def _write_like_the_sweep(tmp_path: Path, row: dict[str, Any]) -> Path:
"""Serialize exactly as ``keep`` does in _run_sweep, redaction included.
json.dumps + write_text would skip the redaction the real writer applies,
so a change there could break reusable rows without failing this test - and
redaction is not cosmetic here, since it rewrites the row's own bytes.
"""
results = tmp_path / "results.jsonl"
secrets = credential_secrets(
SimpleNamespace(auth_token="sk-ant-should-never-appear", base_url=None)
)
with results.open("a") as handle:
handle.write(redact_text(json.dumps(row), secrets) + "\n")
return results
@pytest.fixture
def emitted_row(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> dict[str, Any]:
"""One record from the real run_cell, with only expensive work replaced."""
worktree = tmp_path / "clone"
worktree.mkdir()
# The session is what costs money; everything it returns is scripted. The
# record's binding fields are NOT set here - run_cell derives them.
def fake_run_arm(*_a: Any, **_k: Any) -> dict[str, Any]:
return {
"ok": True,
"error_kind": None,
"error_detail": None,
"resolved": True,
"review_evidence_valid": True,
"review_score": {"weighted_f1": 0.5},
"review_weighted_f1": 0.5,
"skill_invoked": True,
"skill_digest": "skill-digest",
"transcript_missing": False,
"transcript_artifacts": [
{
"path": "transcripts/session-1.jsonl",
"sha256": __import__("hashlib").sha256(b'{"type":"ok"}\n').hexdigest(),
"bytes": 14,
"source": PARENT_EVENT_STREAM_SOURCE,
}
],
"session_ids": ["s1"],
"num_turns": 3,
"duration_s": 1.0,
"cost_usd": 0.5,
"input_tokens": 1,
"output_tokens": 1,
"cache_creation_input_tokens": 0,
"cache_read_input_tokens": 0,
}
for name, value in {
"run_arm": fake_run_arm,
"copy_isolated_tree": lambda *a, **k: worktree,
"make_worktree": lambda *a, **k: worktree,
"sanitize_clone_for_hidden_oracles": lambda *a, **k: SHA,
"stage_task_assets": lambda *a, **k: (),
"isolated_gitnexus_registry_mount": lambda *a, **k: None,
"seed_evaluated_skills": lambda *a, **k: None,
"apply_candidate_overlay": lambda *a, **k: None,
"require_hidden_harness_absent": lambda *a, **k: None,
"require_skill_fingerprint": lambda *a, **k: None,
"enforce_work_evidence": lambda *a, **k: None,
"skill_fingerprint": lambda *a, **k: "skill-digest",
"capture_patch": lambda *a, **k: b"",
"implementation_diff_digest": lambda *a, **k: "",
"diff_churn": lambda *a, **k: {},
"_prepare_untracked_for_diff": lambda *a, **k: None,
"remove_clone": lambda *a, **k: None,
"ce_plugin_dir_for_arm": lambda *a, **k: None,
"ce_plugin_mounts_for_arm": lambda *a, **k: (),
"current_runtime_digest": lambda: "runtime-digest",
"build_sandbox_environment": lambda *a, **k: {},
"credential_secrets": lambda *a, **k: (),
# run_cell requires an immutable base commit before it will record a
# cell; the git plumbing is expensive setup, the SHA it returns is not
# part of the reuse binding under test.
"_sandbox_git": lambda *a, **k: SHA,
# The artifact copy is real; only the read of the agent-written file is
# replaced, since no agent ran to write one.
"_bounded_regular_bytes": lambda *a, **k: b'{"schema_version":1}',
}.items():
monkeypatch.setattr(runner, name, value)
class _Sandbox:
clone = worktree
private_root = tmp_path / "private"
backend = "test-double"
settings_json = "{}"
require_pid_namespace = False
def __enter__(self) -> _Sandbox:
return self
def __exit__(self, *_exc: Any) -> bool:
return False
def command_prefix_for(self, **_k: Any) -> list[str]:
return []
def run(self, *_a: Any, **_k: Any) -> SimpleNamespace:
return SimpleNamespace(ok=True, returncode=0, stdout_tail="", stderr_tail="")
def environment(self, **_k: Any) -> dict[str, str]:
return {}
def host_text(self, value: str) -> str:
return value
monkeypatch.setattr(runner, "prepare_sandbox", lambda **_k: _Sandbox())
ctx = runner.TaskCellContext(
task={"id": TASK_ID, "prompt": "review it", "verify": "true"},
oracle_snapshot=_snapshot("oracle"),
repo=tmp_path / "repo",
task_sha=SHA,
graph_snapshot=_snapshot("graph"),
graph_snapshot_error=None,
asset_snapshot=_snapshot("asset"),
asset_snapshot_error=None,
args=SimpleNamespace(
model="gpt-5.6-sol", effort="xhigh", timeout=60, claude_bin="claude",
base_url=None, auth_token=None, permission_mode=None, arms=["review"],
proposer_model=None, outage_streak=5, runs=1, workers=1,
),
out_dir=tmp_path / "out",
ce_plugin_snapshot=None,
trees_dir=tmp_path / "trees",
bwrap_bin=Path("/bin/true"),
runtime_mounts=(),
candidate_overlay=None,
overlay_digest=None,
sandbox_backend="test-double",
clone_template=None,
sanitized_head=SHA,
)
(tmp_path / "out").mkdir(exist_ok=True)
(tmp_path / "trees").mkdir(exist_ok=True)
(tmp_path / "private").mkdir(exist_ok=True)
# run_cell records review_artifact only when the review source exists, and
# reuse now requires it - a scored review with no artifact is a claim about
# evidence rather than the evidence. Production writes this file; the
# fixture has to as well, or the emitted row is one production never emits.
review_dir = tmp_path / "private" / "review-output"
review_dir.mkdir(exist_ok=True)
(review_dir / "review-output.json").write_text('{"schema_version": 1, "verdict": "approve", "findings": []}')
return runner.run_cell(ctx, 0, "review")
def _expectation(**overrides: Any) -> ComparatorReuseExpectation:
"""Bindings from the sweep's own configuration, not copied out of the row.
Copying the emitted values back in would make producer and consumer agree
because the test arranged it, which is the blind spot being closed.
"""
binding = TaskReuseBinding(
task_base_sha=SHA,
task_prompt_digest=runner.hashlib.sha256(b"review it").hexdigest(),
oracle_digest="oracle-content",
oracle_command_digest="oracle-command",
oracle_manifest_digest="oracle-manifest",
task_asset_manifest_digest="asset-manifest",
sandbox_dependency_manifest_digest="asset-dep-manifest",
)
values: dict[str, Any] = dict(
model="gpt-5.6-sol",
effort="xhigh",
sandbox_backend="test-double",
runtime_digest="runtime-digest",
now=datetime.now(UTC),
max_age=timedelta(days=90),
tasks={TASK_ID: binding},
skill_digests={"review": "skill-digest"},
ce_plugin_version=None,
ce_plugin_manifest_digest=None,
)
values.update(overrides)
return ComparatorReuseExpectation(**values)
def test_a_row_the_runner_emitted_survives_serialization_and_qualifies(
emitted_row: dict[str, Any], tmp_path: Path
) -> None:
"""The producer/consumer contract, end to end through the real writer."""
results = _write_like_the_sweep(tmp_path, emitted_row)
rows = load_result_rows(results)
assert len(rows) == 1, "the production row must survive the reader"
assert row_is_reusable_comparator(rows[0], _expectation()) is True
def test_a_changed_binding_rejects_the_same_emitted_row(
emitted_row: dict[str, Any], tmp_path: Path
) -> None:
"""Fails closed on drift, so the positive case is not vacuous."""
row = load_result_rows(_write_like_the_sweep(tmp_path, emitted_row))[0]
binding = TaskReuseBinding(
task_base_sha=SHA,
task_prompt_digest=runner.hashlib.sha256(b"review it").hexdigest(),
oracle_digest="oracle-content",
oracle_command_digest="oracle-command",
oracle_manifest_digest="oracle-manifest",
task_asset_manifest_digest="asset-manifest",
sandbox_dependency_manifest_digest="DIFFERENT-dependencies",
)
assert row_is_reusable_comparator(row, _expectation(tasks={TASK_ID: binding})) is False

59
eval/tests/test_review_corpus.py Normal file
View file

@ -0,0 +1,59 @@
import hashlib
import json
from pathlib import Path
import yaml
from workflow_bench.oracle_assets import review_case_setup_command
BENCH_ROOT = Path(__file__).parents[1] / "workflow_bench"
def test_review_corpus_is_immutable_and_task_bound():
manifest = json.loads((BENCH_ROOT / "review_cases" / "manifest.json").read_text())
tasks = yaml.safe_load((BENCH_ROOT / "tasks.review.scenarios.yaml").read_text())["tasks"]
by_id = {task["id"]: task for task in tasks}
assert len(manifest["cases"]) >= 6
assert sum(case["id"].endswith("-defect") for case in manifest["cases"]) >= 4
assert sum(case["id"].endswith("-clean") for case in manifest["cases"]) >= 2
assert set(by_id) == {case["id"] for case in manifest["cases"]}
for case in manifest["cases"]:
assert len(case["base_sha"]) == len(case["head_sha"]) == 40
assert len(case["human_verification_commit"]) == 40
patch = BENCH_ROOT / "review_cases" / case["patch"]
assert case["patch"]
assert "defect" not in case["patch"]
assert "clean" not in case["patch"]
assert hashlib.sha256(patch.read_bytes()).hexdigest() == case["patch_sha256"]
task = by_id[case["id"]]
assert task["ref"] == case["base_sha"]
assert task["sandbox_copy"] == [f"eval/workflow_bench/review_cases/{patch.name}"]
assert task["setup"] == review_case_setup_command(patch.name)
assert any(
dep.get("source") == "gitnexus-shared/dist" and dep.get("target") == "gitnexus-shared/dist"
for dep in task["sandbox_dependencies"]
)
def test_hidden_labels_are_not_recoverable_from_visible_task_input():
tasks_path = BENCH_ROOT / "tasks.review.scenarios.yaml"
tasks = yaml.safe_load(tasks_path.read_text())["tasks"]
for task in tasks:
visible = json.dumps(
{
"prompt": task["prompt"],
"setup": task["setup"],
"sandbox_copy": task["sandbox_copy"],
},
sort_keys=True,
)
assert "review-labels.json" not in visible
assert "-defect" not in visible
assert "-clean" not in visible
for oracle_file in task["oracle"]["files"]:
assert oracle_file["source"] not in visible
assert oracle_file["target"] == "review-labels.json"

357
eval/tests/test_review_scoring.py Normal file
View file

@ -0,0 +1,357 @@
import json
from dataclasses import replace
from pathlib import Path
import pytest
from workflow_bench.oracle_assets import OracleFileSnapshot, TaskOracleSnapshot
from workflow_bench.review_scoring import (
ExpectedFinding,
ReviewFinding,
expected_findings,
parse_review_output,
score_review,
)
@pytest.mark.parametrize("noise", [False, True])
def test_complete_misses_are_measured_zero(noise):
actual = (ReviewFinding("noise", "low", "other.py", 1, 1, "style", "s", "e", "r", False),) if noise else ()
score = score_review("comment" if noise else "approve", actual, (expected(),))
assert score["f1"] == score["weighted_f1"] == 0
def test_downgraded_blocker_loses_weight_and_blocker_credit():
actual = ReviewFinding("a", "low", "src/api.ts", 20, 20, "correctness", "s", "e", "r", False)
score = score_review("comment", (actual,), (expected(),))
assert score["weighted_recall"] == 0.2
assert score["blocker_recall"] == 0
assert score["verdict_correct"] is False
@pytest.mark.parametrize("size", [2, 17, 100])
def test_maximum_matching_at_every_supported_size(size):
a = ReviewFinding("a", "high", "src/api.ts", 1, 1, "a", "s", "e", "r", True)
actual = [a, replace(a, finding_id="b", line=10, end_line=10, category="b")]
labels = [
expected(finding_id="broad", line_start=1, line_end=10, category="a"),
expected(finding_id="tight", line_start=1, line_end=1, category="b"),
]
for i in range(2, size):
actual.append(replace(a, finding_id=str(i), path=f"{i}.py"))
labels.append(expected(finding_id=str(i), path=f"{i}.py", line_start=1, line_end=1))
for findings in (actual, list(reversed(actual))):
for expected_labels in (labels, list(reversed(labels))):
assert score_review("request_changes", findings, expected_labels)["true_positives"] == size
def test_dense_matching_handles_the_full_finding_limit():
a = ReviewFinding("a", "high", "src/api.ts", 20, 20, "correctness", "s", "e", "r", True)
actual = [replace(a, finding_id=str(i)) for i in range(100)]
labels = [expected(finding_id=str(i)) for i in range(100)]
assert score_review("request_changes", actual, labels)["true_positives"] == 100
@pytest.mark.parametrize("large_side", ["actual", "expected"])
def test_maximum_matching_with_asymmetric_large_inputs(large_side):
a = ReviewFinding("a", "high", "src/api.ts", 1, 1, "a", "s", "e", "r", True)
actual = [a, replace(a, finding_id="b", line=10, end_line=10, category="b")]
labels = [
expected(finding_id="broad", line_start=1, line_end=10, category="a"),
expected(finding_id="tight", line_start=1, line_end=1, category="b"),
]
for i in range(15):
if large_side == "actual":
actual.append(replace(a, finding_id=str(i), path=f"extra-{i}.py"))
else:
labels.append(expected(finding_id=str(i), path=f"extra-{i}.py"))
assert score_review("request_changes", actual, labels)["true_positives"] == 2
def finding(**overrides):
values = {
"id": "actual-1",
"severity": "high",
"path": "src/api.ts",
"line": 20,
"end_line": 24,
"category": "correctness",
"scenario": "A missing guard lets an invalid request reach the sink.",
"evidence": "The changed call at line 20 bypasses validate().",
"recommendation": "Restore validation before the call.",
"blocking": True,
}
values.update(overrides)
return values
def expected(**overrides):
values = {
"finding_id": "expected-1",
"severity": "high",
"path": "src/api.ts",
"line_start": 18,
"line_end": 22,
"category": "correctness",
}
values.update(overrides)
return ExpectedFinding(**values)
def test_parse_review_output_requires_the_strict_schema(tmp_path: Path):
output = tmp_path / "review-output.json"
output.write_text(
json.dumps(
{
"schema_version": 1,
"verdict": "request_changes",
"findings": [finding()],
}
)
)
verdict, findings = parse_review_output(output)
assert verdict == "request_changes"
assert findings[0].path == "src/api.ts"
assert findings[0].blocking is True
@pytest.mark.parametrize(
"document, message",
[
({"schema_version": 1, "verdict": "approve", "findings": [finding()]}, "approve"),
(
{
"schema_version": 1,
"verdict": "request_changes",
"findings": [finding(blocking=False)],
},
"blocking",
),
(
{
"schema_version": 1,
"verdict": "comment",
"findings": [finding(path="../escape.ts")],
},
"repository-relative",
),
],
)
def test_parse_review_output_rejects_incoherent_or_unsafe_documents(tmp_path: Path, document, message):
output = tmp_path / "review-output.json"
output.write_text(json.dumps(document))
with pytest.raises(ValueError, match=message):
parse_review_output(output)
def test_expected_findings_are_loaded_from_hidden_snapshot_only():
payload = json.dumps(
{
"schema_version": 1,
"findings": [
{
"id": "hidden-1",
"severity": "critical",
"path": "src/auth.ts",
"line_start": 40,
"line_end": 44,
"category": "security",
}
],
}
).encode()
snapshot = TaskOracleSnapshot(
command="true",
command_digest="command",
manifest_digest="manifest",
digest="all",
files=(
OracleFileSnapshot(
target="review-labels.json",
payload=payload,
sha256="payload",
),
),
)
assert expected_findings(snapshot)[0].finding_id == "hidden-1"
def test_score_review_matches_by_path_and_overlapping_range():
actual = (
ReviewFinding(
finding_id="actual-1",
severity="high",
path="src/api.ts",
line=20,
end_line=24,
category="correctness",
scenario="scenario",
evidence="evidence",
recommendation="fix",
blocking=True,
),
ReviewFinding(
finding_id="noise",
severity="low",
path="src/other.ts",
line=1,
end_line=1,
category="style",
scenario="noise",
evidence="noise",
recommendation="noise",
blocking=False,
),
)
score = score_review("request_changes", actual, (expected(),))
assert score["true_positives"] == 1
assert score["false_positives"] == 1
assert score["false_negatives"] == 0
assert score["recall"] == 1
assert score["precision"] == 0.5
assert score["blocker_recall"] == 1
assert score["verdict_correct"] is True
def test_score_review_is_independent_of_finding_list_order():
expected_labels = (
expected(finding_id="broad", line_start=1, line_end=10, category="a"),
expected(finding_id="tight", line_start=5, line_end=5, category="b"),
)
first = ReviewFinding(
finding_id="a",
severity="high",
path="src/api.ts",
line=5,
end_line=5,
category="a",
scenario="s",
evidence="e",
recommendation="r",
blocking=True,
)
second = ReviewFinding(
finding_id="b",
severity="high",
path="src/api.ts",
line=1,
end_line=1,
category="b",
scenario="s",
evidence="e",
recommendation="r",
blocking=True,
)
forward = score_review("request_changes", (first, second), expected_labels)
reverse = score_review("request_changes", (second, first), expected_labels)
assert forward["true_positives"] == reverse["true_positives"]
assert forward["false_positives"] == reverse["false_positives"]
assert forward["false_negatives"] == reverse["false_negatives"]
assert forward["weighted_f1"] == reverse["weighted_f1"]
def test_score_review_prefers_maximum_cardinality_over_greedy_category_match():
expected_labels = (
expected(finding_id="broad", line_start=1, line_end=10, category="a"),
expected(finding_id="tight", line_start=1, line_end=1, category="b"),
)
actual = (
ReviewFinding(
finding_id="actual-1",
severity="high",
path="src/api.ts",
line=1,
end_line=1,
category="a",
scenario="s",
evidence="e",
recommendation="r",
blocking=True,
),
ReviewFinding(
finding_id="actual-2",
severity="high",
path="src/api.ts",
line=10,
end_line=10,
category="b",
scenario="s",
evidence="e",
recommendation="r",
blocking=True,
),
)
score = score_review("request_changes", actual, expected_labels)
assert score["true_positives"] == 2
assert score["false_positives"] == 0
assert score["false_negatives"] == 0
def test_clean_control_rewards_an_empty_approval_and_penalizes_noise():
clean = score_review("approve", (), ())
noisy = score_review(
"comment",
(
ReviewFinding(
finding_id="noise",
severity="medium",
path="src/ok.ts",
line=1,
end_line=1,
category="correctness",
scenario="noise",
evidence="noise",
recommendation="noise",
blocking=False,
),
),
(),
)
assert clean["weighted_f1"] is None
assert clean["precision"] is None
assert clean["recall"] is None
assert clean["clean_pass"] is True
assert clean["verdict_correct"] is True
assert noisy["false_positives"] == 1
assert noisy["weighted_precision"] == 0
assert noisy["recall"] is None
assert noisy["clean_pass"] is False
assert noisy["verdict_correct"] is False
def test_parse_review_output_names_the_actual_failure(tmp_path: Path):
"""One message per cause.
Folding empty, malformed and encoding failures together makes a sandbox that
left the artifact at 0 bytes indistinguishable from an encoding fault: every
such cell reports "not valid UTF-8 JSON". A file the agent never created
escaped that fold — lstat sat outside the try, so it raised
FileNotFoundError — but only as a bare OSError, naming no cause at all.
"""
missing = tmp_path / "never-written.json"
with pytest.raises(ValueError, match="was never written"):
parse_review_output(missing)
empty = tmp_path / "empty.json"
empty.touch()
with pytest.raises(ValueError, match="is empty"):
parse_review_output(empty)
not_utf8 = tmp_path / "latin1.json"
not_utf8.write_bytes(b'{"verdict": "\xff\xfe"}')
with pytest.raises(ValueError, match="not valid UTF-8"):
parse_review_output(not_utf8)
prose = tmp_path / "prose.json"
prose.write_text("Here is my review of the changes.", encoding="utf-8")
with pytest.raises(ValueError, match="not valid JSON"):
parse_review_output(prose)

722
eval/tests/test_runner_hardening.py
View file

@ -2,12 +2,19 @@
import hashlib
import json
import shutil
import subprocess
from contextlib import nullcontext
from pathlib import Path
from types import SimpleNamespace
import pytest
from workflow_bench import runner, runner_artifacts, runner_sessions
from workflow_bench import proposer_sandbox, runner, runner_artifacts, runner_sessions
from workflow_bench.evolution import skill_fingerprint
from workflow_bench.oracle_assets import review_case_setup_command
from workflow_bench.process_control import ManagedProcessError, ManagedProcessResult
from workflow_bench.proposer_sandbox import SandboxError
def _report(**overrides) -> str:
@ -275,6 +282,16 @@ def test_phase_workspace_ignores_claude_sandbox_bootstrap_noise(tmp_path):
(tmp_path / ".claude" / "commands").mkdir(parents=True)
(tmp_path / ".claude" / ".cc-writes").write_text("{}")
(tmp_path / ".env").write_text("")
(tmp_path / ".bash_profile").write_text("")
(tmp_path / ".bashrc").write_text("")
(tmp_path / ".gitconfig").write_text("")
(tmp_path / ".idea").mkdir()
(tmp_path / ".profile").write_text("")
(tmp_path / ".ripgreprc").write_text("")
(tmp_path / ".vscode").mkdir()
(tmp_path / ".zprofile").write_text("")
(tmp_path / ".zshrc").write_text("")
(tmp_path / "scripts").write_text("")
(tmp_path / ".env.development.local").write_text("")
(tmp_path / ".npmrc").write_text("")
(tmp_path / "package.json").write_text("{}")
@ -286,6 +303,30 @@ def test_phase_workspace_ignores_claude_sandbox_bootstrap_noise(tmp_path):
runner_artifacts.enforce_phase_workspace(tmp_path, before, allowed_artifact=artifact)
def test_phase_workspace_records_a_root_scripts_symlink(tmp_path):
target = tmp_path / "helper.py"
target.write_text("planted\n")
before = runner_artifacts.workspace_snapshot(tmp_path)
(tmp_path / "scripts").symlink_to(target)
artifact = tmp_path / "review-output.md"
artifact.write_text("new review")
with pytest.raises(ValueError, match="unauthorized workspace path"):
runner_artifacts.enforce_phase_workspace(tmp_path, before, allowed_artifact=artifact)
def test_phase_workspace_still_rejects_writes_under_scripts(tmp_path):
scripts = tmp_path / "scripts"
scripts.mkdir()
before = runner_artifacts.workspace_snapshot(tmp_path)
(scripts / "helper.py").write_text("planted\n")
artifact = tmp_path / "review-output.md"
artifact.write_text("new review")
with pytest.raises(ValueError, match="unauthorized workspace path"):
runner_artifacts.enforce_phase_workspace(tmp_path, before, allowed_artifact=artifact)
def test_phase_workspace_still_rejects_a_genuinely_unauthorized_change(tmp_path):
# The bootstrap-noise exclusion must stay narrow: an actual source-file
# edit outside the allowed artifact still has to be caught.
@ -381,3 +422,682 @@ def test_phase_workspace_still_sees_writes_under_a_pre_existing_nested_claude_di
with pytest.raises(ValueError, match="unauthorized workspace path"):
runner_artifacts.enforce_phase_workspace(tmp_path, before, allowed_artifact=artifact)
def _cell_context(tmp_path, **overrides):
"""A TaskCellContext whose per-task inputs are all present and valid."""
snapshot = SimpleNamespace(
digest="asset-digest",
manifest_digest="asset-manifest",
dependency_content_digest="dep-content",
dependency_manifest_digest="dep-manifest",
)
graph = SimpleNamespace(
digest="graph-digest",
manifest_digest="graph-manifest",
materialize=lambda *_a, **_k: None,
)
oracle = SimpleNamespace(
digest="oracle-digest",
command_digest="oracle-command",
manifest_digest="oracle-manifest",
)
fields = {
"task": {"id": "task-a", "class": "demo", "prompt": "do the thing"},
"oracle_snapshot": oracle,
"repo": tmp_path / "repo",
"task_sha": "a" * 40,
"graph_snapshot": graph,
"graph_snapshot_error": None,
"asset_snapshot": snapshot,
"asset_snapshot_error": None,
"args": SimpleNamespace(
claude_bin="claude",
model="pinned-model",
proposer_model=None,
effort="xhigh",
auth_token=None,
),
"out_dir": tmp_path / "out",
"ce_plugin_snapshot": None,
"trees_dir": tmp_path / "trees",
"bwrap_bin": tmp_path / "bwrap",
"runtime_mounts": (),
"candidate_overlay": None,
"overlay_digest": None,
}
fields.update(overrides)
fields["out_dir"].mkdir(parents=True, exist_ok=True)
return runner.TaskCellContext(**fields)
def _stub_cell_dependencies(monkeypatch, tmp_path):
"""Replace everything a cell shells out to, so only its own logic runs.
Returns the clone it will hand out and the list its teardown appends to.
"""
removed: list[Path] = []
worktree = tmp_path / "clone"
worktree.mkdir()
monkeypatch.setattr(runner, "make_worktree", lambda *_a, **_k: worktree)
monkeypatch.setattr(runner, "sanitize_clone_for_hidden_oracles", lambda *_a, **_k: "b" * 40)
monkeypatch.setattr(runner, "stage_task_assets", lambda *_a, **_k: [])
monkeypatch.setattr(runner, "isolated_gitnexus_registry_mount", lambda *_a, **_k: None)
monkeypatch.setattr(runner, "ce_plugin_mounts_for_arm", lambda *_a, **_k: [])
monkeypatch.setattr(runner, "ce_plugin_dir_for_arm", lambda *_a, **_k: None)
monkeypatch.setattr(runner, "prepare_sandbox", lambda **_k: nullcontext(SimpleNamespace(run=None)))
monkeypatch.setattr(runner, "skill_fingerprint", lambda *_a, **_k: "skill-digest")
monkeypatch.setattr(runner, "require_skill_fingerprint", lambda *_a, **_k: None)
monkeypatch.setattr(runner, "_sandbox_git", lambda *_a, **_k: "c" * 40)
monkeypatch.setattr(runner, "implementation_diff_digest", lambda *_a, **_k: "")
monkeypatch.setattr(runner, "_prepare_untracked_for_diff", lambda *_a, **_k: None)
monkeypatch.setattr(runner, "diff_churn", lambda *_a, **_k: {})
monkeypatch.setattr(runner, "enforce_work_evidence", lambda *_a, **_k: None)
monkeypatch.setattr(runner, "capture_patch", lambda *_a, **_k: b"diff")
monkeypatch.setattr(runner, "run_arm", lambda *_a, **_k: {"resolved": True, "ok": True, "error_kind": None})
monkeypatch.setattr(runner, "remove_clone", lambda path: removed.append(path))
return worktree, removed
def test_run_cell_returns_a_row_bound_to_its_task_and_snapshots(monkeypatch, tmp_path):
_, removed = _stub_cell_dependencies(monkeypatch, tmp_path)
record = runner.run_cell(_cell_context(tmp_path), 2, "workflow")
assert record["resolved"] is True
assert record["error_kind"] is None
# The row has to carry its own coordinates: once cells stop running in a
# predictable order, position in results.jsonl identifies nothing.
assert record["task"] == "task-a"
assert record["arm"] == "workflow"
assert record["run"] == 2
assert record["task_asset_snapshot_digest"] == "asset-digest"
assert record["sanitized_graph_snapshot_digest"] == "graph-digest"
assert record["oracle_digest"] == "oracle-digest"
assert removed == [tmp_path / "clone"]
@pytest.mark.parametrize(
"failure",
[
ManagedProcessError(
["setup"],
ManagedProcessResult(
state="timeout",
returncode=-15,
stdout_tail="",
stderr_tail="",
duration_s=1.0,
),
),
SandboxError("sandbox refused"),
OSError("disk went away"),
RuntimeError("overlay drifted"),
ValueError("bad binding"),
],
ids=["managed-process", "sandbox", "os", "runtime", "value"],
)
def test_run_cell_records_an_expected_failure_and_still_removes_its_clone(monkeypatch, tmp_path, failure):
_, removed = _stub_cell_dependencies(monkeypatch, tmp_path)
def explode(*_args, **_kwargs):
raise failure
monkeypatch.setattr(runner, "run_arm", explode)
record = runner.run_cell(_cell_context(tmp_path), 0, "workflow")
assert record["error_kind"] == "infra-error"
assert record["resolved"] is False
# A cell owns its clone for its whole lifetime; the sweep has no other
# chance to reclaim it, so the finally must survive every expected failure.
assert removed == [tmp_path / "clone"]
def test_run_cell_redacts_the_auth_token_from_the_failure_it_prints(monkeypatch, tmp_path, capsys):
_stub_cell_dependencies(monkeypatch, tmp_path)
secret = "sk-ant-not-a-real-key"
def explode(*_args, **_kwargs):
raise ManagedProcessError(
["claude"],
ManagedProcessResult(
state="exited",
returncode=1,
stdout_tail="",
stderr_tail=f"ANTHROPIC_API_KEY={secret}",
duration_s=1.0,
),
)
monkeypatch.setattr(runner, "run_arm", explode)
context = _cell_context(tmp_path)
context.args.auth_token = secret
# ManagedProcessError stringifies up to 1000 raw bytes of stderr_tail, and
# this line streams live into the CI log now that the sweep's stdout is
# echoed. results.jsonl already redacts the same field.
runner.run_cell(context, 0, "workflow")
assert secret not in capsys.readouterr().out
def test_run_cell_lets_an_unexpected_failure_escape_rather_than_scoring_it(monkeypatch, tmp_path):
_, removed = _stub_cell_dependencies(monkeypatch, tmp_path)
def explode(*_args, **_kwargs):
raise KeyError("harness bug")
monkeypatch.setattr(runner, "run_arm", explode)
# A harness bug recorded as an ordinary infra-error would be averaged into
# the evidence and counted toward the outage breaker. It must crash instead.
with pytest.raises(KeyError):
runner.run_cell(_cell_context(tmp_path), 0, "workflow")
assert removed == [tmp_path / "clone"]
def test_run_cell_reports_a_cleanup_failure_over_its_primary_outcome(monkeypatch, tmp_path):
_stub_cell_dependencies(monkeypatch, tmp_path)
def refuse(_path):
raise OSError("clone is busy")
monkeypatch.setattr(runner, "remove_clone", refuse)
record = runner.run_cell(_cell_context(tmp_path), 1, "workflow")
assert record["error_kind"] == "cleanup-failure"
assert record["resolved"] is False
assert "primary=None" in record["error_detail"]
assert "clone is busy" in record["error_detail"]
def _git(repo, *args):
return subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True, text=True)
def test_run_cell_runs_the_arm_against_a_copy_of_the_clone_template(monkeypatch, tmp_path):
"""run_cell must copy the template, never re-clone.
run_cell takes the clone-template branch on essentially every multi-cell
sweep: it copies a pre-sanitized template rather than paying `git clone
--no-local` plus repack/prune/fsck per cell. Asserting on a copy the test
makes itself proves nothing about that branch — the clone the arm receives
is what has to come from the template, carrying the template's sanitized
HEAD rather than a recomputed one.
"""
repo = tmp_path / "repo"
repo.mkdir()
_git(repo, "init", "--quiet")
_git(repo, "checkout", "--quiet", "-b", "main")
(repo / "from-template.txt").write_text("sanitized\n")
_git(repo, "add", "-A")
_git(repo, "-c", "user.name=test", "-c", "user.email=test@invalid", "commit", "--quiet", "-m", "base")
sha = _git(repo, "rev-parse", "HEAD").stdout.strip()
trees = tmp_path / "trees"
trees.mkdir()
template = runner.make_worktree(repo, sha, trees)
template_head = _git(template, "rev-parse", "HEAD").stdout.strip()
_stub_cell_dependencies(monkeypatch, tmp_path)
def fail_if_recloned(*_args, **_kwargs):
raise AssertionError("clone template present: run_cell must not re-clone")
monkeypatch.setattr(runner, "make_worktree", fail_if_recloned)
monkeypatch.setattr(runner, "sanitize_clone_for_hidden_oracles", fail_if_recloned)
seen: dict[str, object] = {}
def record_arm(_arm, _task, worktree, _args, **_kwargs):
seen["worktree"] = worktree
seen["head"] = _git(worktree, "rev-parse", "HEAD").stdout.strip()
seen["content"] = (worktree / "from-template.txt").read_text()
# The copy is a private checkout: what the cell writes must not reach
# the template the other cells of this task still copy from.
(worktree / "from-template.txt").write_text("cell-local\n")
return {"resolved": True, "ok": True, "error_kind": None}
monkeypatch.setattr(runner, "run_arm", record_arm)
runner.run_cell(
_cell_context(tmp_path, clone_template=template, sanitized_head=template_head),
0,
"workflow",
)
assert seen["content"] == "sanitized\n"
assert seen["head"] == template_head
assert seen["worktree"] != template
assert (template / "from-template.txt").read_text() == "sanitized\n"
def test_run_cell_does_not_mask_the_staged_review_patch_before_setup(monkeypatch, tmp_path):
"""Review setup applies a patch staged under eval/workflow_bench.
Overlaying the empty oracle mask on that path is the CI abort:
`git apply` dies with `can't open patch`. The staged copy must stay
visible to sandboxed setup, then be gone before the model starts.
"""
worktree, _ = _stub_cell_dependencies(monkeypatch, tmp_path)
patch = worktree / "eval" / "workflow_bench" / "review_cases" / "pr-2718.patch"
patch.parent.mkdir(parents=True)
patch.write_text("diff --git a/visible.py b/visible.py\n")
captured: dict[str, object] = {}
def fake_prepare(**kwargs):
captured["mounts"] = kwargs.get("read_only_mounts", [])
def run(_command, **_kwargs):
leftover = worktree / "eval" / "workflow_bench"
if leftover.exists():
shutil.rmtree(leftover)
return SimpleNamespace(ok=True)
return nullcontext(SimpleNamespace(run=run))
monkeypatch.setattr(runner, "prepare_sandbox", fake_prepare)
context = _cell_context(
tmp_path,
task={
"id": "review-pr-2718-defect",
"class": "review-defect",
"prompt": "review the local diff",
"setup": review_case_setup_command("pr-2718.patch"),
},
)
record = runner.run_cell(context, 0, "workflow")
assert record.get("error_kind") is None
targets = [getattr(mount, "target", None) for mount in captured["mounts"] if mount is not None]
assert not any(target and "eval/workflow_bench" in str(target) for target in targets)
assert not (worktree / "eval" / "workflow_bench").exists()
def test_run_cell_fails_closed_when_setup_leaves_the_hidden_harness(monkeypatch, tmp_path):
worktree, _ = _stub_cell_dependencies(monkeypatch, tmp_path)
leftover = worktree / "eval" / "workflow_bench" / "review_cases"
leftover.mkdir(parents=True)
(leftover / "pr-2718.patch").write_text("diff\n")
def fake_prepare(**kwargs):
return nullcontext(SimpleNamespace(run=lambda *_a, **_k: SimpleNamespace(ok=True)))
monkeypatch.setattr(runner, "prepare_sandbox", fake_prepare)
context = _cell_context(
tmp_path,
task={
"id": "review-pr-2718-defect",
"class": "review-defect",
"prompt": "review the local diff",
"setup": review_case_setup_command("pr-2718.patch"),
},
)
record = runner.run_cell(context, 0, "workflow")
assert record["error_kind"] == "infra-error"
assert "hidden harness visible" in str(record["error_detail"])
def test_run_cell_fails_closed_when_a_per_task_snapshot_never_materialized(tmp_path):
# The snapshots are prepared once per task, before any cell. If that failed,
# every cell of the task has to record it rather than run against nothing.
context = _cell_context(tmp_path, asset_snapshot=None, asset_snapshot_error=OSError("no assets"))
record = runner.run_cell(context, 0, "workflow")
assert record["error_kind"] == "infra-error"
assert "no assets" in str(record["error_detail"])
def _progress():
"""Collector for what a sweep started and kept, readable after it raises."""
return SimpleNamespace(started=[], kept=[], streak=0, tripped=False)
def _sweep(cells, *, workers, run, outage_limit=5, streak=0, into=None):
"""Drive sweep_task_cells, recording what it started and kept.
Pass ``into`` a ``_progress()`` when the sweep is expected to raise: the
collector survives the exception, the return value does not.
"""
result = _progress() if into is None else into
result.streak, result.tripped = runner.sweep_task_cells(
cells,
workers=workers,
run=run,
on_start=lambda run_idx, arm: result.started.append((run_idx, arm)),
on_record=lambda run_idx, arm, _record: result.kept.append((run_idx, arm)),
outage_streak=streak,
outage_limit=outage_limit,
)
return result
def _row(error_kind=None):
return {"resolved": error_kind is None, "error_kind": error_kind}
CELLS = [(run_idx, arm) for run_idx in range(3) for arm in ("workflow", "candidate_workflow")]
@pytest.mark.parametrize("workers", [1, 3, 8])
@pytest.mark.parametrize("primary", ["review-evidence-invalid", "skill-not-invoked", "session-error"])
def test_unusable_review_evidence_stops_a_54_cell_sweep(workers, primary):
cells = [(i, "review") for i in range(54)]
result = _sweep(
cells,
workers=workers,
run=lambda *_: {
"resolved": False,
"error_kind": primary,
"review_evidence_valid": False,
},
)
assert result.tripped
assert 5 <= len(result.started) <= 5 + workers - 1
assert result.kept == result.started
def test_measured_review_miss_resets_the_outage_streak():
result = _sweep(
CELLS,
workers=1,
streak=4,
run=lambda *_: {
"resolved": False,
"error_kind": "oracle-failed",
"review_evidence_valid": True,
"review_weighted_f1": 0.0,
},
)
assert result.streak == 0 and not result.tripped
def test_invalid_review_with_primary_skill_error_is_excluded_from_aggregation():
result = runner.aggregate(
[
{
"resolved": False,
"error_kind": "skill-not-invoked",
"review_evidence_valid": False,
}
]
)
assert result["excluded_runs"] == 1 and result["valid_runs"] == 0
def test_sweep_keeps_rows_in_submission_order_whatever_order_they_finish():
# Cells finish in whatever order the machine allows, but a wave is folded
# in submission order — the outage streak counts consecutive failures, and
# "consecutive" in completion order would make the trip point flaky.
import threading
first_wave = CELLS[:3]
rendezvous = threading.Barrier(3, timeout=10)
release_first = threading.Event()
fast_finished = threading.Event()
finished: list[tuple[int, str]] = []
result: list[SimpleNamespace] = []
def run(run_idx, arm):
cell = (run_idx, arm)
if cell in first_wave:
rendezvous.wait()
if cell == first_wave[0]:
release_first.wait(timeout=10)
else:
finished.append(cell)
if len(finished) == 2:
fast_finished.set()
return _row()
sweep = threading.Thread(target=lambda: result.append(_sweep(CELLS, workers=3, run=run)))
sweep.start()
try:
assert fast_finished.wait(timeout=10)
assert first_wave[0] not in finished
assert set(finished) == set(first_wave[1:])
finally:
release_first.set()
sweep.join(timeout=10)
assert not sweep.is_alive()
assert result[0].kept == CELLS
assert result[0].started == CELLS
assert result[0].tripped is False
@pytest.mark.parametrize("workers", [1, 2, 3])
def test_sweep_trips_the_breaker_within_one_wave_of_the_serial_point(workers):
# Serial stops after the 5th consecutive systemic failure. Cells already in
# flight when the breaker trips cannot be recalled or erased from the
# evidence, so the overrun is bounded by the wave and every completed row
# is kept. Ten cells make the bound visible rather than hidden by the end.
long_task = [(run_idx, arm) for run_idx in range(5) for arm in ("workflow", "candidate_workflow")]
result = _sweep(long_task, workers=workers, run=lambda *_: _row("session-error"))
assert result.tripped is True
assert 5 <= len(result.started) <= 5 + workers - 1
assert result.kept == result.started
assert len(result.started) < len(long_task)
def test_sweep_reads_a_real_failure_as_signal_rather_than_an_outage():
# resolved=False with no systemic error_kind is the benchmark working, not
# the harness failing; it must reset the streak instead of tripping.
result = _sweep(CELLS, workers=3, run=lambda *_: {"resolved": False, "error_kind": None})
assert result.tripped is False
assert result.streak == 0
assert result.kept == CELLS
def test_sweep_surfaces_an_unexpected_worker_failure_instead_of_dropping_the_cell():
def run(run_idx, arm):
if (run_idx, arm) == (0, "candidate_workflow"):
raise KeyError("harness bug")
return _row()
# A Future holds its exception until read. Unread, this cell would vanish
# from the evidence with no crash and no row — fewer runs in an arm's
# aggregate, silently.
with pytest.raises(KeyError):
_sweep(CELLS, workers=3, run=run)
def test_sweep_runs_cells_of_a_wave_at_the_same_time():
import threading
barrier = threading.Barrier(3, timeout=10)
def run(run_idx, arm):
# Deadlocks unless all three cells of the wave are genuinely in flight
# together — a pool that serialised them would time out here.
barrier.wait()
return _row()
result = _sweep(CELLS, workers=3, run=run)
assert result.kept == CELLS
def test_sweep_of_one_worker_never_leaves_the_calling_thread():
import threading
caller = threading.current_thread()
seen: list[threading.Thread] = []
def run(run_idx, arm):
seen.append(threading.current_thread())
return _row()
# Ctrl-C reaches only the main thread, so the serial default has to stay on
# it: a cell on a worker thread is outside the reach of the cleanup that
# kills its sandboxed process tree.
_sweep(CELLS, workers=1, run=run)
assert seen == [caller] * len(CELLS)
@pytest.mark.parametrize("failure", [KeyError("harness bug"), SystemExit(97)])
def test_sweep_keeps_the_rows_of_cells_that_finished_beside_a_failing_one(failure):
def run(run_idx, arm):
if (run_idx, arm) == (0, "candidate_workflow"):
raise failure
return _row()
progress = _progress()
# The failing cell's two siblings completed and spent their budget before
# the harness bug surfaced. Reading the futures in order and raising on the
# first failure would drop their rows: money spent, no evidence written.
with pytest.raises(type(failure)):
_sweep(CELLS, workers=3, run=run, into=progress)
assert progress.kept == [(0, "workflow"), (1, "workflow")]
def test_sweep_hands_a_ctrl_c_back_without_waiting_for_the_running_cells(monkeypatch):
import threading
in_flight = threading.Barrier(3, timeout=10)
release = threading.Event()
finished: list[tuple[int, str]] = []
def run(run_idx, arm):
in_flight.wait()
release.wait(timeout=10)
finished.append((run_idx, arm))
return _row()
def interrupt_once_the_wave_is_running(_futures, *_args, **_kwargs):
# Stands in for the Ctrl-C an operator types mid-wave: an async
# KeyboardInterrupt is delivered to the main thread, which is the one
# blocked here waiting on the wave.
in_flight.wait()
raise KeyboardInterrupt
monkeypatch.setattr(runner, "wait", interrupt_once_the_wave_is_running)
try:
with runner.cancellation_scope(release), pytest.raises(KeyboardInterrupt):
_sweep(CELLS, workers=2, run=run)
# Cancellation releases active work before joining; no worker can
# outlive the assets the interrupted sweep is about to clean up.
assert set(finished) == set(CELLS[:2])
finally:
release.set()
def test_workers_is_bounded_at_both_ends_before_the_sweep_starts():
base = ["--tasks", "tasks.yaml", "--model", "pinned-model"]
assert runner.build_parser().parse_args(base).workers == 1
at_max = runner.build_parser().parse_args([*base, "--workers", str(runner.MAX_WORKERS)])
assert at_max.workers == runner.MAX_WORKERS
# A mistyped worker count has to fail at the command line: hours later it
# only shows up as timed-out sessions, which the promotion gate throws away.
for rejected in ("0", "-1", str(runner.MAX_WORKERS + 1)):
with pytest.raises(SystemExit):
runner.build_parser().parse_args([*base, "--workers", rejected])
def test_partial_wave_submission_preserves_rows_and_original_interruption(monkeypatch):
original = runner.ThreadPoolExecutor.submit
submissions = 0
def submit(pool, *args, **kwargs):
nonlocal submissions
submissions += 1
if submissions == 2:
raise KeyboardInterrupt("submission interrupted")
return original(pool, *args, **kwargs)
monkeypatch.setattr(runner.ThreadPoolExecutor, "submit", submit)
records = []
with pytest.raises(KeyboardInterrupt, match="submission interrupted"):
runner.sweep_task_cells(
CELLS,
workers=2,
run=lambda *_: _row(),
on_start=lambda *_: None,
on_record=lambda *record: records.append(record),
outage_streak=0,
outage_limit=5,
)
assert len(records) == 2
assert records[1][2]["error_kind"] == "cancelled"
@pytest.mark.parametrize("error_kind", ["infra-error", "cleanup-failure"])
def test_progress_line_reports_an_unmeasured_failure_as_unmeasured_not_as_free(error_kind):
dead = runner.infra_error_record(RuntimeError("bwrap died"))
dead["error_kind"] = error_kind
line = runner.cell_progress_line("task", "workflow", 0, dead)
# The 0.0s are placeholders for numbers no session ever produced; printed
# as numbers they read as a cell that ran instantly for free.
assert "cost=n/a" in line
assert "took=n/a" in line
assert f"error_kind={error_kind}" in line
# results.jsonl is promotion evidence — only the display changes.
assert dead["cost_usd"] == 0.0
assert dead["duration_s"] == 0.0
def test_progress_line_reports_the_numbers_a_real_run_measured():
line = runner.cell_progress_line(
"task",
"workflow",
1,
{
"resolved": True,
"input_tokens": 10,
"output_tokens": 2,
"cost_usd": 0.5,
"duration_s": 12.0,
"error_kind": None,
},
)
assert "cost=$0.5" in line
assert "took=12.0s" in line
assert "error_kind=none" in line
def test_claude_settings_allow_the_review_artifact_directory():
"""The second gate on the artifact path.
The bwrap bind is not the only thing that decides whether the agent can
write: the CLI applies this filesystem policy to its own tools, so a path
missing from allowWrite is unwritable however the mount is shaped. The
artifact lived under /workspace when this list was written, which is why
moving it out needed this entry and nothing caught the omission.
"""
settings = json.loads(proposer_sandbox.build_claude_settings(sandbox_enabled=True))
filesystem = settings["sandbox"]["filesystem"]
assert proposer_sandbox.SANDBOX_REVIEW_OUTPUT in filesystem["allowWrite"]
assert proposer_sandbox.SANDBOX_REVIEW_OUTPUT in filesystem["allowRead"]
assert filesystem["denyRead"] == ["/"]
def test_review_contract_tells_the_agent_the_writable_path():
prompt = runner.REVIEW_PROMPT.format(task="task text")
assert f"{runner.SANDBOX_REVIEW_OUTPUT}/{runner.REVIEW_OUTPUT}" in prompt
assert f"{runner.SANDBOX_WORKSPACE}/{runner.REVIEW_OUTPUT}" not in prompt
# The JSON shape survives .format() with its braces intact.
assert '{"schema_version":1' in prompt
artifact = f"{runner.SANDBOX_REVIEW_OUTPUT}/{runner.REVIEW_OUTPUT}"
assert runner.CE_REVIEW_PROMPT.format(task="task text").count(artifact) == 1
def test_enforce_phase_workspace_can_require_an_untouched_workspace(tmp_path):
(tmp_path / "tracked.py").write_text("original\n")
before = runner_artifacts.workspace_snapshot(tmp_path)
runner_artifacts.enforce_phase_workspace(tmp_path, before, allowed_artifact=None)
(tmp_path / "tracked.py").write_text("the review edited the code it was reviewing\n")
with pytest.raises(ValueError, match="changed the read-only workspace"):
runner_artifacts.enforce_phase_workspace(tmp_path, before, allowed_artifact=None)

46
eval/tests/test_sanitized_graph.py
View file

@ -27,6 +27,32 @@ def test_prebuilt_graph_and_harness_assets_are_rejected(task):
sanitized_graph.validate_no_prebuilt_graph_assets(task)
def test_review_case_patches_are_allowed_sandbox_copy():
sanitized_graph.validate_no_prebuilt_graph_assets(
{"sandbox_copy": ["eval/workflow_bench/review_cases/pr-2718.patch"]}
)
@pytest.mark.parametrize(
"task",
[
{"sandbox_copy": ["eval/workflow_bench"]},
{"sandbox_copy": ["eval/workflow_bench/evolve.py"]},
{
"sandbox_dependencies": [
{
"source": "eval/workflow_bench/review_cases/pr-2718.patch",
"target": "patch",
}
]
},
],
)
def test_non_corpus_harness_paths_stay_rejected(task):
with pytest.raises(SandboxError, match="prebuilt graph or harness"):
sanitized_graph.validate_no_prebuilt_graph_assets(task)
def test_graph_environment_is_offline_deterministic_and_ignores_target_gitignore():
env = sanitized_graph._graph_environment()
@ -34,6 +60,10 @@ def test_graph_environment_is_offline_deterministic_and_ignores_target_gitignore
assert env["GITNEXUS_NO_GITIGNORE"] == "1"
assert env["GITNEXUS_WORKER_POOL_SIZE"] == "1"
assert env["GITNEXUS_PARSE_CHUNK_CONCURRENCY"] == "1"
assert env["GITNEXUS_WORKER_READY_TIMEOUT_MS"] == str(
sanitized_graph.GRAPH_WORKER_READY_TIMEOUT_MS
)
assert int(env["GITNEXUS_WORKER_READY_TIMEOUT_MS"]) >= 60_000
assert "ANTHROPIC_API_KEY" not in env
@ -182,6 +212,22 @@ def test_prepare_sanitized_graph_builds_once_from_parentless_tree_and_caches_onl
assert removed == [seed]
def test_prepare_sanitized_graph_requires_head_when_given_a_template(tmp_path: Path):
with pytest.raises(SandboxError, match="sanitized HEAD"):
sanitized_graph.prepare_sanitized_graph(
{},
repo=tmp_path,
resolved_sha="b" * 40,
parent=tmp_path,
cache=SimpleNamespace(), # type: ignore[arg-type]
claude_bin="claude",
bwrap_bin="bwrap",
runtime_mounts=(),
clone_template=tmp_path,
sanitized_head=None,
)
def test_graph_snapshot_rejects_arm_sanitization_identity_drift(tmp_path: Path):
assets = SimpleNamespace(
digest="digest",

372
eval/tests/test_session_progress.py Normal file
View file

@ -0,0 +1,372 @@
"""Live progress reporting for long headless sessions.
Progress includes event metadata and bounded, redacted tool argument/result
previews. Model prose and raw event streams are never echoed. These tests pin
that boundary along with the signals that distinguish work from a wedged run.
"""
from __future__ import annotations
import io
import json
import time
from workflow_bench.runner_sessions import SessionProgress, neutralize_ci_log_text
def _drain_lines(stream: io.StringIO) -> list[str]:
return [line for line in stream.getvalue().splitlines() if line.strip()]
def _observe(progress: SessionProgress, chunk: bytes) -> None:
progress.observe(chunk)
progress._emit_pending()
def test_progress_bounds_unanswered_tools_and_undrained_messages() -> None:
progress = SessionProgress("bounded", stream=io.StringIO())
for index in range(2000):
progress.observe(
(
json.dumps(
{
"type": "assistant",
"message": {
"content": [
{"type": "tool_use", "id": str(index), "name": "Bash", "input": {"command": "true"}}
]
},
}
)
+ "\n"
).encode()
)
assert len(progress._pending_tools) <= 256
assert len(progress._pending_messages) <= 256
assert "1999" in progress._pending_tools
assert "0" not in progress._pending_tools
progress.observe(
(
json.dumps(
{
"type": "user",
"message": {
"content": [{"type": "tool_result", "tool_use_id": "1999", "content": "recent result"}]
},
}
)
+ "\n"
).encode()
)
progress._emit_pending()
assert "recent result" in progress._stream.getvalue()
assert "1999" not in progress._pending_tools
def test_progress_reports_bounded_redacted_tool_io_but_never_model_prose() -> None:
stream = io.StringIO()
progress = SessionProgress(
"gen 0 proposer",
stream=stream,
heartbeat_s=3600,
secrets=("SECRET-TOKEN-abc123",),
)
events = [
{"type": "system", "subtype": "init"},
{
"type": "assistant",
"message": {
"content": [
{"type": "text", "text": "SECRET-REASONING-abc123"},
{
"type": "tool_use",
"id": "t1",
"name": "Grep",
"input": {
"pattern": "TODO",
"path": "/workspace",
"token": "SECRET-TOKEN-abc123",
},
},
]
},
},
{
"type": "user",
"message": {
"content": [
{
"type": "tool_result",
"tool_use_id": "t1",
"is_error": False,
"content": "src/a.py:1: TODO " + "x" * 1000,
}
]
},
},
{"type": "result", "num_turns": 1, "is_error": False, "total_cost_usd": 1.5},
]
for event in events:
_observe(progress, (json.dumps(event) + "\n").encode())
output = stream.getvalue()
assert "SECRET-REASONING-abc123" not in output
assert "SECRET-TOKEN-abc123" not in output
assert "[REDACTED]" in output
assert "session initialized" in output
assert "turn 1 · Grep" in output
assert 'tool Grep input={"pattern":"TODO","path":"/workspace","token":"[REDACTED]"}' in output
assert "tool Grep result=ok output=" in output
assert "truncated" in output
assert "finished · 1 turns · ok · $1.50" in output
def test_progress_reports_errors_and_mcp_io_but_skips_other_tool_payloads() -> None:
stream = io.StringIO()
progress = SessionProgress("flow", stream=stream, heartbeat_s=3600)
events = [
{
"type": "assistant",
"message": {
"content": [
{
"type": "tool_use",
"id": "m1",
"name": "mcp__gitnexus__query",
"input": {"search_query": "call resolution"},
},
{
"type": "tool_use",
"id": "e1",
"name": "Edit",
"input": {"file_path": "secret.py", "new_string": "do not log"},
},
]
},
},
{
"type": "user",
"message": {
"content": [
{
"type": "tool_result",
"tool_use_id": "m1",
"is_error": True,
"content": "repository is not indexed",
},
{
"type": "tool_result",
"tool_use_id": "e1",
"content": "edited secret.py",
},
]
},
},
]
for event in events:
_observe(progress, (json.dumps(event) + "\n").encode())
output = stream.getvalue()
assert 'tool mcp__gitnexus__query input={"search_query":"call resolution"}' in output
assert 'tool mcp__gitnexus__query result=error output="repository is not indexed"' in output
assert "do not log" not in output
assert "edited secret.py" not in output
def test_progress_distinguishes_mcp_semantic_errors_from_transport_success() -> None:
stream = io.StringIO()
progress = SessionProgress("flow", stream=stream, heartbeat_s=3600)
events = [
{
"type": "assistant",
"message": {
"content": [
{
"type": "tool_use",
"id": "m1",
"name": "mcp__gitnexus__impact",
"input": {"target": "missing", "direction": "upstream"},
}
]
},
},
{
"type": "user",
"message": {
"content": [
{
"type": "tool_result",
"tool_use_id": "m1",
"is_error": False,
"content": [
{
"type": "text",
"text": '{"error":"Target missing not found"}\n\n---\n**Next:** retry',
}
],
}
]
},
},
]
for event in events:
_observe(progress, (json.dumps(event) + "\n").encode())
assert "tool mcp__gitnexus__impact result=semantic-error" in stream.getvalue()
def test_progress_calls_out_api_retries_because_that_is_the_stuck_signature() -> None:
stream = io.StringIO()
progress = SessionProgress("proposer", stream=stream, heartbeat_s=3600)
event = {
"type": "system",
"subtype": "api_retry",
"attempt": 7,
"max_retries": 10,
"retry_delay_ms": 34199.87,
"error": "unknown",
}
_observe(progress, (json.dumps(event) + "\n").encode())
line = _drain_lines(stream)[-1]
assert "API retry 7/10 in 34s" in line
assert "no response from the model endpoint" in line
def test_progress_speaks_up_while_a_session_is_silent() -> None:
stream = io.StringIO()
with SessionProgress("proposer", stream=stream, heartbeat_s=0.05):
time.sleep(0.35)
heartbeats = [line for line in _drain_lines(stream) if "still running" in line]
assert heartbeats, "a silent session must still report that it is alive"
assert "0 turns" in heartbeats[0]
def test_progress_survives_partial_chunks_garbage_and_unbounded_lines() -> None:
stream = io.StringIO()
progress = SessionProgress("proposer", stream=stream, heartbeat_s=3600)
payload = json.dumps(
{"type": "assistant", "message": {"content": [{"type": "tool_use", "id": "t1", "name": "Bash"}]}}
).encode()
# An event split across reads, non-JSON noise, and a huge newline-free run.
_observe(progress, payload[:10])
_observe(progress, payload[10:] + b"\nnot json at all\n")
_observe(progress, b"x" * (4 * 1024 * 1024))
_observe(progress, b'\n{"type":"result","num_turns":2,"is_error":true}\n')
output = stream.getvalue()
assert "turn 1 · Bash" in output
assert "finished · 2 turns · error" in output
def test_progress_sanitizes_a_hostile_tool_name() -> None:
stream = io.StringIO()
progress = SessionProgress("proposer", stream=stream, heartbeat_s=3600)
event = {
"type": "assistant",
"message": {"content": [{"type": "tool_use", "id": "t1", "name": "Bash\nFAKE-LOG-LINE injected"}]},
}
_observe(progress, (json.dumps(event) + "\n").encode())
assert "FAKE-LOG-LINE" not in stream.getvalue()
assert len(_drain_lines(stream)) == 1
def test_progress_redacts_non_ascii_secrets_before_json_escaping() -> None:
stream = io.StringIO()
secret = "tokén-密码"
progress = SessionProgress("flow", stream=stream, heartbeat_s=3600, secrets=(secret,))
event = {
"type": "assistant",
"message": {
"content": [
{"type": "tool_use", "id": "t1", "name": "Grep", "input": {"token": secret}},
]
},
}
_observe(progress, (json.dumps(event, ensure_ascii=False) + "\n").encode())
output = stream.getvalue()
assert secret not in output
assert json.dumps(secret)[1:-1] not in output
assert "[REDACTED]" in output
def test_cell_failure_detail_line_explains_why_a_cell_failed() -> None:
from workflow_bench.runner import cell_failure_detail_line
assert cell_failure_detail_line("t", "workflow", 0, {"error_kind": None}) is None
assert cell_failure_detail_line("t", "workflow", 0, {"error_kind": "x"}) is None
line = cell_failure_detail_line(
"trivial-status-json-alias",
"candidate_workflow",
1,
{
"error_kind": "plan-evidence-invalid",
"error_detail": "unauthorized workspace path\ntoken=sk-secret-value",
},
("sk-secret-value",),
)
assert line is not None
assert line.startswith("[trivial-status-json-alias][candidate_workflow][run 1] detail: ")
assert "unauthorized workspace path" in line
assert "sk-secret-value" not in line
assert "\n" not in line
def test_cell_failure_detail_line_bounds_a_huge_detail() -> None:
from workflow_bench.runner import MAX_CELL_DETAIL_CHARS, cell_failure_detail_line
line = cell_failure_detail_line(
"t", "workflow", 0, {"error_kind": "session-error", "error_detail": {"stdout_tail": "y" * 50_000}}
)
assert line is not None
assert "truncated" in line
assert len(line) < MAX_CELL_DETAIL_CHARS + 200
def test_progress_neutralizes_github_actions_annotation_forms() -> None:
rewritten = neutralize_ci_log_text(
"gitnexus/src/cli/optional-grammars.ts(18,36): error TS2307: Cannot find module "
"'gitnexus-shared'\n::error::Composite projects may not disable incremental compilation.\n"
"##[error]tsc failed"
)
assert "): error TS2307" not in rewritten
assert "): compiler-error TS2307" in rewritten
assert "::error::" not in rewritten
assert "[:]error::" in rewritten
assert "##[error]" not in rewritten
assert "# [error]tsc failed" in rewritten
stream = io.StringIO()
progress = SessionProgress("review-pr-2718-defect-ce_review-run0", stream=stream, heartbeat_s=3600)
events = [
{
"type": "assistant",
"message": {
"content": [{"type": "tool_use", "id": "b1", "name": "Bash", "input": {"command": "npx tsc --noEmit"}}]
},
},
{
"type": "user",
"message": {
"content": [
{
"type": "tool_result",
"tool_use_id": "b1",
"is_error": True,
"content": "gitnexus/src/cli/optional-grammars.ts(18,36): error TS2307: Cannot find module 'gitnexus-shared'",
}
]
},
},
]
for event in events:
_observe(progress, (json.dumps(event) + "\n").encode())
output = stream.getvalue()
assert "): error TS2307" not in output
assert "): compiler-error TS2307" in output
assert "result=error" in output

279
eval/tests/test_sweep_finalization.py Normal file
View file

@ -0,0 +1,279 @@
"""The real sweep must reach the right finalization decision.
`enforce_measurement_health` is unit-tested and the call site is pinned
structurally, but neither shows the guard running inside a sweep. These drive
the real `_run_sweep` with cell execution scripted and everything downstream of
it left alone: folding, aggregation, the artifact writers, the health guard and
the exit selection.
The below-breaker case is the decisive one. A fixture of many unusable cells
aborts through the pre-existing outage breaker instead - `review-evidence-invalid`
is systemic with a limit of 5 - and would pass whether or not the finalization
guard exists. One fresh unusable cell stays under that threshold, so only the
guard can catch it.
"""
from __future__ import annotations
import json
import threading
from collections.abc import Callable
from pathlib import Path
from types import SimpleNamespace
from typing import Any
import pytest
from tests.bench_fixtures import scored_review_row, unusable_review_row
from workflow_bench import runner
TASK = {
"id": "review-pr-2718-defect",
"repo": "~/GitNexus",
"ref": "a" * 40,
"prompt": "review it",
"verify": "true",
"class": "review-defect",
}
def _args(out: Path, **overrides: Any) -> SimpleNamespace:
values: dict[str, Any] = dict(
arms=["review"], claude_bin="claude", effort="xhigh", model="gpt-5.6-sol",
out=out, outage_streak=runner.DEFAULT_OUTAGE_STREAK, promotion_max_task_regression=10.0,
promotion_metric="review_weighted_f1", promotion_min_improvement=1.0,
promotion_min_runs=1, proposer_model=None, reuse_results=None, runs=1, workers=1,
timeout=60, base_url=None, auth_token=None, permission_mode=None,
)
values.update(overrides)
return SimpleNamespace(**values)
def _snapshot(prefix: str) -> SimpleNamespace:
return SimpleNamespace(
digest=f"{prefix}-content", manifest_digest=f"{prefix}-manifest",
dependency_content_digest=f"{prefix}-dep", dependency_manifest_digest=f"{prefix}-depman",
command_digest=f"{prefix}-command", materialize=lambda *a, **k: None,
)
def _sweep(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
record: dict[str, Any] | Callable[[int], dict[str, Any]],
*,
runs: int = 1,
cancel_event: threading.Event | None = None,
candidate_arms: list[str] | None = None,
arms: list[str] | None = None,
after_cell: Callable[[int, str], None] | None = None,
):
"""Drive the real _run_sweep; only cell execution and setup are scripted.
``after_cell`` runs once a cell's record exists, which is how a test sets
cancellation deterministically at a known point instead of racing a sleep.
"""
out = tmp_path / "out"
def scripted_cell(_ctx: Any, run_idx: int, arm: str) -> dict[str, Any]:
row = dict(record(run_idx) if callable(record) else record)
row.update({"task": TASK["id"], "arm": arm, "run": run_idx, "class": TASK["class"]})
if after_cell is not None:
after_cell(run_idx, arm)
return row
monkeypatch.setattr(runner, "run_cell", scripted_cell)
monkeypatch.setattr(runner, "ensure_task_graph", lambda **k: k["env"].graph_snapshots.__setitem__(
k["graph_key"], _snapshot("graph")))
monkeypatch.setattr(runner.TaskAssetCache, "prepare", lambda self, *a, **k: _snapshot("asset"))
# Binding resolution clones the repo and verifies the ref; that is expensive
# setup, and the bindings it would return are supplied directly instead.
monkeypatch.setattr(
runner, "resolve_task_bindings",
lambda tasks, expected, **k: list(expected),
)
return runner._run_sweep(
_args(out, runs=runs, arms=arms or ["review"]),
parser=SimpleNamespace(error=lambda m: (_ for _ in ()).throw(SystemExit(2))),
tasks=[TASK],
skipped_expensive=[],
oracle_snapshots=[_snapshot("oracle")],
expected_task_bindings=[{"repo_identity": str(tmp_path / "repo"), "resolved_sha": "a" * 40}],
ce_plugin_config=None,
bwrap_bin=Path("/bin/true"),
sandbox_backend="test-double",
runtime_mounts=(),
candidate_arms=candidate_arms or [],
candidate_overlay=None,
overlay_digest=None,
promotion_target_bases={},
cancel_event=cancel_event,
), out
def test_one_unusable_cell_below_the_breaker_reaches_the_finalization_guard(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
) -> None:
"""The decisive case: too few failures to trip the breaker, so only the guard can catch it."""
streak = runner.systemic_outage_streak("review-evidence-invalid", 0)
assert streak < runner.DEFAULT_OUTAGE_STREAK, "fixture must stay under the breaker"
unusable = unusable_review_row()
with pytest.raises(SystemExit) as exc:
_sweep(tmp_path, monkeypatch, unusable)
assert exc.value.code == 1
out = capsys.readouterr().out
assert "review: UNUSABLE" in out, "the guard must name the arm and its status"
assert "systemic-outage" not in out, "the breaker must not have tripped"
def test_a_zero_score_stays_a_valid_negative_measurement(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
) -> None:
"""0.0 is a present measurement, not missing evidence.
A truthiness check on the score would misread it as absent and turn a
quality result into an execution-health failure.
"""
zeroed = scored_review_row(
resolved=False, error_kind="oracle-failed",
review_score={"weighted_f1": 0.0}, review_weighted_f1=0.0,
)
_sweep(tmp_path, monkeypatch, zeroed)
out = capsys.readouterr().out
assert "review: OBSERVED_OK" in out
assert "UNUSABLE" not in out
def test_finalization_persists_results_and_report(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Evidence must survive the sweep, and say the same thing the exit does."""
scored = scored_review_row(resolved=False, error_kind="oracle-failed", review_weighted_f1=0.2)
_result, out = _sweep(tmp_path, monkeypatch, scored)
rows = [json.loads(line) for line in (out / "results.jsonl").read_text().splitlines()]
assert len(rows) == 1 and rows[0]["review_weighted_f1"] == 0.2
assert (out / "report.md").is_file()
def test_cancellation_without_an_outage_exits_130(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
) -> None:
"""An interrupted sweep is interrupted, not aborted.
One admissible cell lands first so the measurement-health guard classifies
the arm DEGRADED rather than UNUSABLE - otherwise the guard would supply
exit 1 and this test would pass without ever exercising exit selection.
"""
cancel_event = threading.Event()
with pytest.raises(SystemExit) as exc:
_sweep(
tmp_path, monkeypatch, lambda _run: scored_review_row(),
runs=3, cancel_event=cancel_event,
after_cell=lambda run_idx, _arm: cancel_event.set() if run_idx == 0 else None,
)
stdout = capsys.readouterr().out
report = (tmp_path / "out" / "report.md").read_text()
assert "Sweep cancelled" in report, "an interruption must be reported as one"
assert "systemic-outage" not in stdout, "no breaker trip in this scenario"
assert exc.value.code == 130
def test_an_outage_keeps_exit_1_even_though_the_breaker_cancels(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]
) -> None:
"""Precedence: the breaker sets cancel_event, so order decides the exit.
Testing cancellation first would relabel every outage a Ctrl-C. The first
cell is admissible for the same reason as above, and the failures after it
are consecutive and systemic, which is what the breaker actually counts.
"""
def cell(run_idx: int) -> dict[str, Any]:
return scored_review_row() if run_idx == 0 else unusable_review_row()
cancel_event = threading.Event()
with pytest.raises(SystemExit) as exc:
_sweep(tmp_path, monkeypatch, cell,
runs=1 + runner.DEFAULT_OUTAGE_STREAK, cancel_event=cancel_event)
stdout = capsys.readouterr().out
report = (tmp_path / "out" / "report.md").read_text()
assert "systemic-outage" in stdout, "the real breaker must have tripped"
assert cancel_event.is_set(), "the breaker cancels in-flight work"
assert "Sweep aborted" in report
assert exc.value.code == 1, "an outage must not become the 130 of a Ctrl-C"
def test_an_interrupted_sweep_keeps_the_evidence_it_already_paid_for(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Cancellation must not discard rows that already cost money.
The completed-run persistence test cannot show this: it never interrupts, so
it would pass even if the writer only ran on the clean path.
"""
cancel_event = threading.Event()
with pytest.raises(SystemExit):
_sweep(
tmp_path, monkeypatch, lambda _run: scored_review_row(review_weighted_f1=0.42),
runs=3, cancel_event=cancel_event,
after_cell=lambda run_idx, _arm: cancel_event.set() if run_idx == 0 else None,
)
rows = [
json.loads(line)
for line in (tmp_path / "out" / "results.jsonl").read_text().splitlines()
]
assert len(rows) == 1, "the cell that completed before cancellation must survive"
assert rows[0]["review_weighted_f1"] == 0.42, "its measurement must survive intact"
assert (tmp_path / "out" / "report.md").is_file()
def test_an_interrupted_sweep_emits_nothing_that_authorizes_promotion(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The semantic condition, not the absence of a file.
promotion.json is still written for an aborted run - it is the record of why
nothing was promoted. What must hold is that nothing in it authorizes a
promotion from partial evidence.
"""
cancel_event = threading.Event()
with pytest.raises(SystemExit):
_sweep(
tmp_path, monkeypatch, lambda _run: scored_review_row(),
runs=3, cancel_event=cancel_event,
# The candidate arm has to RUN, not merely appear in promotion
# metadata: _run_sweep builds cells only from args.arms, so naming it
# in candidate_arms alone left the candidate with no results at all -
# and then "insufficient_evidence" would hold because nothing ran,
# not because partial evidence is barred from promoting.
arms=["review", "candidate_review"],
candidate_arms=["candidate_review"],
after_cell=(
lambda run_idx, arm: cancel_event.set()
if run_idx == 0 and arm == "candidate_review"
else None
),
)
rows = [
json.loads(line)
for line in (tmp_path / "out" / "results.jsonl").read_text().splitlines()
]
assert any(r["arm"] == "candidate_review" for r in rows), (
"the candidate must have produced evidence, or insufficient_evidence "
"would hold merely because nothing ran"
)
promotion = json.loads((tmp_path / "out" / "promotion.json").read_text())
assert promotion["run_status"] == "aborted"
assert promotion["decisions"], "an aborted run still has to say what it decided"
for decision in promotion["decisions"]:
assert decision["decision"] == "insufficient_evidence"
assert any("partial evidence" in reason for reason in decision["reasons"])

56
eval/tests/test_task_assets.py
View file

@ -5,15 +5,15 @@ from __future__ import annotations
import os
import stat
import subprocess
from pathlib import Path
from pathlib import Path, PurePosixPath
import pytest
from workflow_bench.proposer_sandbox import VITE_TEMP_DIR, SandboxError
from workflow_bench.oracle_assets import TaskOracleSnapshot
from workflow_bench.runner_tasks import resolve_task_bindings
from workflow_bench.task_assets import TaskAssetCache, stage_task_assets
from workflow_bench import task_assets
from workflow_bench.task_assets import TaskAssetCache, _is_harness_sandbox_copy, stage_task_assets
from workflow_bench import runtime_mounts, task_assets
SHA = "a" * 40
@ -443,3 +443,53 @@ def test_non_node_modules_dependency_snapshot_has_no_vite_temp(tmp_path: Path) -
snapshot = cache.prepare(task, repo=repo, resolved_sha=SHA)
captured = {entry.path.as_posix() for entry in snapshot.dependencies[0].entries}
assert not any(path.endswith(VITE_TEMP_DIR) for path in captured)
def test_review_case_sandbox_copy_is_read_from_the_harness_not_the_task_repo(
monkeypatch, tmp_path: Path
) -> None:
repo = tmp_path / "task-repo"
repo.mkdir()
(repo / "eval" / "workflow_bench").mkdir(parents=True)
harness = tmp_path / "harness"
patch = harness / "eval" / "workflow_bench" / "review_cases" / "pr-2718.patch"
patch.parent.mkdir(parents=True)
patch.write_bytes(b"diff --git a/a b/a\n")
monkeypatch.setattr(runtime_mounts, "HARNESS_ROOT", harness)
task = {
"sandbox_copy": ["eval/workflow_bench/review_cases/pr-2718.patch"],
"sandbox_dependencies": [],
}
with TaskAssetCache(tmp_path / "cache") as cache:
snapshot = cache.prepare(task, repo=repo, resolved_sha=SHA)
copied = snapshot.root / "sandbox-copy" / "eval" / "workflow_bench" / "review_cases" / "pr-2718.patch"
assert copied.read_bytes() == b"diff --git a/a b/a\n"
def test_review_case_sandbox_copy_does_not_fall_back_to_the_task_repo(
monkeypatch, tmp_path: Path
) -> None:
repo = tmp_path / "task-repo"
planted = repo / "eval" / "workflow_bench" / "review_cases" / "pr-2718.patch"
planted.parent.mkdir(parents=True)
planted.write_bytes(b"from-task-repo")
harness = tmp_path / "harness"
harness.mkdir()
monkeypatch.setattr(runtime_mounts, "HARNESS_ROOT", harness)
task = {
"sandbox_copy": ["eval/workflow_bench/review_cases/pr-2718.patch"],
"sandbox_dependencies": [],
}
with TaskAssetCache(tmp_path / "cache") as cache:
with pytest.raises(SandboxError, match="unavailable"):
cache.prepare(task, repo=repo, resolved_sha=SHA)
def test_harness_sandbox_copy_does_not_treat_parent_escapes_as_corpus() -> None:
assert _is_harness_sandbox_copy(PurePosixPath("eval/workflow_bench/review_cases/pr.patch"))
assert not _is_harness_sandbox_copy(
PurePosixPath("eval/workflow_bench/review_cases/../oracles/hidden.json")
)
assert not _is_harness_sandbox_copy(PurePosixPath("eval/workflow_bench/oracles"))

684
eval/tests/test_workflow_bench.py
View file

@ -1,24 +1,39 @@
"""Unit tests for workflow benchmark aggregation, reporting, task, and CI contracts."""
import json
import os
import re
import shlex
import subprocess
import threading
from pathlib import Path
import pytest
import yaml
from typing import Any
from workflow_bench import runner
from workflow_bench.evolution import CANDIDATE_ARMS
from workflow_bench.process_control import _CANCELLATION, cancellation_scope
from workflow_bench.runner import (
aggregate,
GraphBuildEnv,
arm_health,
broken_incumbent_arms,
unhealthy_arms,
unmeasured_arms,
build_parser,
infra_error_record,
next_graph_prefetch_target,
normalized_model_identifier,
parse_shortstat,
prefetch_next_graph,
render_report,
savings,
select_tasks,
systemic_outage_streak,
task_has_planned_paid_cells,
)
@ -61,6 +76,15 @@ def test_aggregate_takes_medians_and_counts_resolved():
"diff_deletions": 5,
"class": "demo",
"resolved": 2,
# None of these are reused, so every resolution was measured this sweep.
"resolved_fresh": 2,
# Health is counted separately from resolution: all three executed and
# produced usable evidence, including the one that resolved nothing.
"fresh_attempts": 3,
"admissible": 3,
"execution_failures": 0,
"evidence_failures": 0,
"health_reasons": [],
"runs": 3,
"valid_runs": 3,
"excluded_runs": 0,
@ -89,7 +113,7 @@ def task_row(task_id: str, **overrides):
"command": "true",
"files": [
{
"source": "trivial-version-alias.oracle.test.ts",
"source": "trivial-status-json-alias.oracle.test.ts",
"target": "oracle.test.ts",
}
],
@ -171,6 +195,11 @@ def test_eval_ci_uses_locked_uv_and_blocking_native_containment_jobs():
assert containment["env"] == {
"GITNEXUS_REQUIRE_BWRAP_CANARY": "1",
"GITNEXUS_REQUIRE_CLAUDE_CANARY": "1",
# This job is the only place with bubblewrap, the pinned runtime and a
# built GitNexus together, so it is where the offline sweep runs with
# nothing provisioning-stubbed. Pinned here so the gate cannot be
# dropped and leave the sweep silently running the stubbed path.
"GITNEXUS_REQUIRE_FULL_SWEEP": "1",
}
assert containment["timeout-minutes"] == 20
assert containment_node_setup["with"] == {
@ -189,11 +218,13 @@ def test_eval_ci_uses_locked_uv_and_blocking_native_containment_jobs():
assert claude_lock["packages"]["node_modules/@anthropic-ai/claude-code"]["integrity"].startswith("sha512-")
assert "if(p.version!=='2.1.214') process.exit(1)" in workflow
assert "'2.1.214 (Claude Code)'" in workflow
assert containment_steps["Build pinned shared runtime"]["working-directory"] == "gitnexus-shared"
assert containment_steps["Build pinned shared runtime"]["run"].splitlines() == [
"npm ci",
"npm run build",
]
# Shared is compiled by gitnexus `npm run build` (scripts/build.js runTsc).
# A dedicated npm ci in gitnexus-shared pulls TypeScript 7 and stalls CI.
assert "Build pinned shared runtime" not in containment_steps
assert not any(
step.get("working-directory") == "gitnexus-shared" and "npm ci" in str(step.get("run", ""))
for step in containment["steps"]
)
assert containment_steps["Install and build pinned GitNexus runtime"]["working-directory"] == "gitnexus"
assert containment_steps["Install and build pinned GitNexus runtime"]["run"].splitlines() == [
"npm ci",
@ -213,6 +244,12 @@ def test_eval_ci_uses_locked_uv_and_blocking_native_containment_jobs():
"tests/test_proposer_sandbox.py",
"tests/test_workflow_bench_sessions.py",
"tests/test_ce_plugin_runtime.py",
# The offline sweep, run here with nothing stubbed: this job is the only
# one carrying bubblewrap, the pinned runtime and a built GitNexus.
"tests/test_offline_sweep_integration.py",
# Carries the real-CLI identity probe, which needs CLAUDE_CANARY_BIN -
# set only on this job. Omitted from this list it skipped everywhere.
"tests/test_mock_provider.py",
"-q",
]
bwrap_canary_marker = re.compile(
@ -234,13 +271,16 @@ def test_shipped_scenarios_opt_out_the_cross_module_cell_and_rebuild_graph_asset
tasks = yaml.safe_load(task_file.read_text())["tasks"]
selected, skipped = select_tasks(tasks, include_expensive=False)
assert [task["id"] for task in selected] == [
"trivial-version-alias",
"inv-bug-pdg-note",
"trivial-status-json-alias",
"inv-bug-c-system-include",
"inv-feature-list-repos-filter",
]
assert skipped == ["cross-module-parse-retry"]
assert all(not task.get("sandbox_copy") for task in tasks)
assert all(task["sandbox_dependencies"] for task in tasks)
assert all(
any(dep.get("source") == "gitnexus-shared/dist" for dep in task["sandbox_dependencies"]) for task in tasks
)
assert all(task["oracle"]["command"] and task["oracle"]["files"] for task in tasks)
assert all("./node_modules/.bin/vitest run" in task["oracle"]["command"] for task in tasks)
assert all("npx vitest" not in task["oracle"]["command"] for task in tasks)
@ -318,6 +358,36 @@ def test_aggregate_excludes_unverified_transcript_evidence():
assert agg["excluded_runs"] == 1
def test_aggregate_excludes_invalid_review_artifacts_from_quality_metrics():
scored = record(
cost_usd=1.0,
review_weighted_f1=0.8,
review_true_positives=2,
review_false_positives=0,
review_false_negatives=1,
review_precision=1.0,
review_recall=0.67,
review_f1=0.8,
review_weighted_precision=0.8,
review_weighted_recall=0.8,
review_blocker_recall=1.0,
review_severity_accuracy=1.0,
review_category_accuracy=1.0,
review_grounded_evidence=1.0,
review_clean_control=False,
)
agg = aggregate(
[
scored,
record(cost_usd=2.0, resolved=False, error_kind="review-evidence-invalid"),
]
)
assert agg["valid_runs"] == 1
assert agg["excluded_runs"] == 1
assert agg["review_weighted_f1"] == 0.8
assert agg["review_true_positives"] == 2
def test_render_report_surfaces_excluded_and_unverified_runs():
results = {
"t": {
@ -425,3 +495,601 @@ def test_outage_streak_flag_defaults_and_disables():
base = ["--tasks", "tasks.yaml", "--model", "claude-sonnet-4-20250514"]
assert build_parser().parse_args(base).outage_streak == 5
assert build_parser().parse_args([*base, "--outage-streak", "0"]).outage_streak == 0
def test_run_evolution_script_is_the_shared_ci_and_local_entrypoint():
eval_dir = Path(__file__).resolve().parents[1]
script = eval_dir / "workflow_bench" / "run-evolution.sh"
workflow = eval_dir.parent / ".github" / "workflows" / "gitnexus-skill-evolution.yml"
assert script.is_file()
assert script.stat().st_mode & 0o111
workflow_text = workflow.read_text()
assert "./workflow_bench/run-evolution.sh --apply" in workflow_text
assert "python -m workflow_bench.evolve" not in workflow_text
env = {
"PATH": os.environ.get("PATH", "/usr/bin"),
"MODEL": "claude-sonnet-5",
"PROPOSER_MODEL": "claude-opus-4-8",
"EFFORT": "xhigh",
"GENERATIONS": "1",
"RUNS": "3",
"WORKERS": "2",
"PROVIDER": "openai",
"INCLUDE_EXPENSIVE": "1",
"SEED_RESULTS": "/tmp/seed-bench",
"CLAUDE_BIN": "/opt/claude",
"OUT_ROOT": "/tmp/wfevolve",
"CE_PLUGIN_DIR": "/tmp/ce-plugin",
"CE_PLUGIN_VERSION": "3.24.0",
"HOME": os.environ.get("HOME", "/tmp"),
}
printed = subprocess.run(
[str(script), "--dry-run", "--apply"],
check=True,
capture_output=True,
text=True,
env=env,
)
argv = shlex.split(printed.stdout)
assert argv[:7] == ["uv", "run", "--locked", "--extra", "dev", "python", "-m"]
assert argv[7] == "workflow_bench.evolve"
assert argv[argv.index("--tasks") + 1] == "workflow_bench/tasks.review.scenarios.yaml"
assert argv[argv.index("--arms") + 1] == "review"
assert argv[argv.index("--ce-plugin-version") + 1] == "3.24.0"
assert argv[argv.index("--model") + 1] == "gpt-5.6-sol"
assert argv[argv.index("--proposer-model") + 1] == "gpt-5.6-sol"
assert argv[argv.index("--effort") + 1] == "xhigh"
assert argv[argv.index("--workers") + 1] == "2"
assert argv[argv.index("--claude-bin") + 1] == "/opt/claude"
assert argv[argv.index("--out-root") + 1] == "/tmp/wfevolve"
assert argv[argv.index("--seed-results") + 1] == "/tmp/seed-bench"
assert "--apply" in argv
assert "--include-expensive" in argv
assert "claude-sonnet-5" not in argv
assert printed.stderr # rewrite notice goes to stderr
def test_planned_paid_cells_treat_missing_reuse_as_paid():
task = {"id": "review-pr-2718-defect"}
assert task_has_planned_paid_cells(
task,
arms=["ce_review", "review", "candidate_review"],
runs=3,
reusable_rows={},
reuse_source=None,
)
reuse_source = Path("/tmp/seed")
rows = {
(task["id"], arm, run_idx): {}
for run_idx in range(3)
for arm in ("ce_review", "review", "candidate_review")
}
assert not task_has_planned_paid_cells(
task,
arms=["ce_review", "review", "candidate_review"],
runs=3,
reusable_rows=rows,
reuse_source=reuse_source,
)
del rows[(task["id"], "candidate_review", 0)]
assert task_has_planned_paid_cells(
task,
arms=["ce_review", "review", "candidate_review"],
runs=3,
reusable_rows=rows,
reuse_source=reuse_source,
)
def test_next_graph_prefetch_skips_ready_shas_and_fully_reused_tasks(tmp_path: Path):
first = {"id": "review-a"}
second = {"id": "review-b"}
third = {"id": "review-c"}
reuse_source = tmp_path / "seed"
reused_second = {
(second["id"], arm, 0): {} for arm in ("ce_review", "review", "candidate_review")
}
target = next_graph_prefetch_target(
[
(first, {"repo_identity": "/repo", "resolved_sha": "aaa"}),
(second, {"repo_identity": "/repo", "resolved_sha": "bbb"}),
(third, {"repo_identity": "/repo", "resolved_sha": "ccc"}),
],
arms=["ce_review", "review", "candidate_review"],
runs=1,
reusable_rows=reused_second,
reuse_source=reuse_source,
ready_keys={("/repo", "aaa")},
)
assert target is not None
task, binding, key = target
assert task["id"] == "review-c"
assert key == ("/repo", "ccc")
assert binding["resolved_sha"] == "ccc"
def test_prefetch_next_graph_runs_ensure_on_a_background_thread(monkeypatch):
started = threading.Event()
seen: list[tuple[str, str]] = []
def fake_ensure(**kwargs):
seen.append(kwargs["graph_key"])
started.set()
monkeypatch.setattr("workflow_bench.runner.ensure_task_graph", fake_ensure)
cancel = threading.Event()
job = prefetch_next_graph(
task={"id": "review-b"},
binding={"repo_identity": "/repo", "resolved_sha": "bbb"},
graph_key=("/repo", "bbb"),
env=GraphBuildEnv(
trees=Path("/tmp"),
task_asset_cache=None,
claude_bin="claude",
bwrap_bin="bwrap",
sandbox_backend="bwrap",
runtime_mounts=(),
clone_templates={},
clone_template_errors={},
graph_snapshots={},
graph_snapshot_errors={},
),
cancel_event=cancel,
)
assert job.key == ("/repo", "bbb")
assert started.wait(timeout=2)
job.join()
assert seen == [("/repo", "bbb")]
def test_a_reused_resolution_does_not_count_as_this_sweeps_health():
"""resolved counts evidence; resolved_fresh counts evidence measured today.
broken_incumbent_arms reads resolved_fresh because a reused row proves last
generation's environment worked. Counting it would make an arm whose cells
were all reused look healthy in exactly the run where a broken environment
should have been caught.
"""
reused = [record(resolved=True, reused=True), record(resolved=True, reused=True)]
agg = aggregate(reused)
assert agg["resolved"] == 2
assert agg["resolved_fresh"] == 0
assert broken_incumbent_arms({"t": {"review": agg}}, {"review"}) == ["review"]
mixed = aggregate([record(resolved=True, reused=True), record(resolved=True)])
assert mixed["resolved_fresh"] == 1
assert broken_incumbent_arms({"t": {"review": mixed}}, {"review"}) == []
def test_graph_build_env_ready_keys_covers_successes_and_failures():
"""A key that failed is attempted, not pending.
next_graph_prefetch_target skips keys already in ready_keys. If a failed
build were omitted, the sweep would prefetch it again every iteration and
pay a full clone and offline index each time for a build that cannot
succeed.
"""
env = GraphBuildEnv(
trees=Path("/tmp"),
task_asset_cache=None,
claude_bin="claude",
bwrap_bin="bwrap",
sandbox_backend="bwrap",
runtime_mounts=(),
clone_templates={("/repo", "aaa"): (Path("/tmp/a"), "aaa")},
clone_template_errors={("/repo", "bbb"): OSError("clone failed")},
graph_snapshots={("/repo", "ccc"): object()},
graph_snapshot_errors={("/repo", "ddd"): OSError("index failed")},
)
assert env.ready_keys() == {
("/repo", "aaa"),
("/repo", "bbb"),
("/repo", "ccc"),
("/repo", "ddd"),
}
def _cell(**overrides) -> dict[str, Any]:
"""One results.jsonl row, healthy unless told otherwise."""
base = record(resolved=True)
base.update({"error_kind": None, "review_evidence_valid": True, "transcript_missing": False})
base.update(overrides)
return base
def _arms(**by_arm) -> dict[str, dict[str, dict[str, Any]]]:
return {"task0": {arm: aggregate(rows) for arm, rows in by_arm.items()}}
def test_a_reviewer_that_scores_badly_is_not_an_unhealthy_harness():
"""Reconstructed from Actions run 33962002890's logged observations.
Every completed cell was resolved=False with error_kind=oracle-failed, at a
median score of 0.212 — the reviews ran, wrote artifacts and were scored.
That is a valid negative for the quality gate to judge. Diagnosing it as a
broken environment is the confusion this classification exists to end.
"""
scored_but_wrong = [_cell(resolved=False, error_kind="oracle-failed") for _ in range(3)]
results = _arms(review=scored_but_wrong, ce_review=list(scored_but_wrong))
assert unhealthy_arms(results, {"review", "ce_review"}) == []
health = arm_health(results, {"review"})["review"]
assert health.admissible == 3 and health.fresh_attempts == 3
assert (health.execution_failures, health.evidence_failures) == (0, 0)
def test_an_all_zero_score_is_still_a_valid_negative():
zeroed = [_cell(resolved=False, error_kind="oracle-failed", review_weighted_f1=0.0) for _ in range(3)]
assert unhealthy_arms(_arms(review=zeroed), {"review"}) == []
def test_artifacts_that_were_never_written_are_an_unhealthy_harness():
"""Reconstructed from Actions run 33912693948.
All 41 artifacts came back 0 bytes because the mount made an atomic write
impossible. The reviews could not produce evidence at all — the opposite of
the case above, and the one a health check must catch. The old caller
excluded review arms entirely, so it could not have.
"""
unwritable = [_cell(resolved=False, ok=False, error_kind="review-evidence-invalid") for _ in range(3)]
flagged = unhealthy_arms(_arms(review=unwritable), {"review"})
assert [h.arm for h in flagged] == ["review"]
assert flagged[0].evidence_failures == 3
assert "review-evidence-invalid" in flagged[0].reasons
def test_one_admissible_cell_leaves_an_arm_degraded_not_healthy():
"""Mixed outcomes are DEGRADED. One usable measurement does not erase two failures.
Not fatal - the sweep still produced evidence - but calling it healthy is
how a partly-broken environment passes review.
"""
mixed = [
_cell(resolved=False, error_kind="oracle-failed"),
_cell(resolved=False, ok=False, error_kind="session-error"),
_cell(resolved=False, ok=False, error_kind="infra-error"),
]
results = _arms(review=mixed)
health = arm_health(results, {"review"})["review"]
assert health.status == "DEGRADED"
assert unhealthy_arms(results, {"review"}) == [], "degraded is diagnostic, not fatal"
assert health.execution_failures == 2, "failures must stay visible, not be erased"
assert health.admissible == 1
def test_a_row_that_fails_both_ways_is_only_subtracted_once():
"""run_arm can produce a row that is an execution AND an evidence failure.
It keeps the first error_kind — a session-error survives — and still sets
review_evidence_valid=False when the artifact will not parse. Counting that
row against admissible twice zeroed an arm that held a real measurement,
which arm_health reports as UNUSABLE and the measurement gate then fails on.
"""
both = _cell(resolved=False, ok=False, error_kind="session-error", review_evidence_valid=False)
results = _arms(review=[both, _cell(resolved=True, error_kind="oracle-failed")])
health = arm_health(results, {"review"})["review"]
assert (health.execution_failures, health.evidence_failures) == (1, 1)
assert health.fresh_attempts == 2
assert health.admissible == 1
assert health.status == "DEGRADED"
assert unhealthy_arms(results, {"review"}) == []
def test_reused_rows_alone_leave_current_health_unknown():
"""Historical success cannot certify this sweep's environment."""
reused = [_cell(reused=True) for _ in range(3)]
results = _arms(review=reused)
assert unmeasured_arms(results, {"review"}) == ["review"]
assert unhealthy_arms(results, {"review"}) == []
assert arm_health(results, {"review"})["review"].measured is False
def test_the_paid_canary_survives_a_prior_run_with_more_run_indices():
"""The canary counts planned cells, not every key reuse selection returned.
Reuse selection accepts any non-negative prior `run`, so a results directory
produced with --runs 5 leaves keys this sweep never plans. Comparing against
those made the "arm is fully reused" test false exactly when it was true,
and the incumbent went a whole sweep without one measured cell.
"""
tasks = [{"id": "task0"}, {"id": "task1"}]
reusable = {(task["id"], "review", run): {} for task in tasks for run in range(5)}
dropped = runner.drop_canary_reuse_key(reusable, arm="review", tasks=tasks, runs=3)
assert dropped == ("task0", "review", 0)
assert dropped not in reusable
# A second call is a no-op: the arm now has its paid cell.
assert runner.drop_canary_reuse_key(reusable, arm="review", tasks=tasks, runs=3) is None
def test_an_arm_with_a_planned_paid_cell_keeps_every_reusable_row():
tasks = [{"id": "task0"}]
reusable = {("task0", "review", 0): {}}
assert runner.drop_canary_reuse_key(reusable, arm="review", tasks=tasks, runs=2) is None
assert len(reusable) == 1
def test_reused_successes_do_not_mask_fresh_execution_failures():
rows = [_cell(reused=True), _cell(reused=True), _cell(ok=False, error_kind="session-error")]
flagged = unhealthy_arms(_arms(review=rows), {"review"})
assert [h.arm for h in flagged] == ["review"]
assert flagged[0].fresh_attempts == 1 and flagged[0].execution_failures == 1
def test_a_parseable_artifact_does_not_excuse_a_failed_session():
"""Artifact parseability must not override an execution failure."""
rows = [_cell(ok=False, error_kind="session-error", review_evidence_valid=True) for _ in range(2)]
flagged = unhealthy_arms(_arms(review=rows), {"review"})
assert [h.arm for h in flagged] == ["review"]
assert flagged[0].execution_failures == 2
def test_a_single_unusable_review_is_caught_below_the_breaker_threshold():
"""The decisive regression for the finalization guard.
A fixture of 41 empty artifacts would abort through the outage breaker -
review-evidence-invalid is systemic and the limit is 5 - so it proves
nothing about this path. One fresh unusable cell is under that threshold,
which leaves the finalization check as the only thing that can catch it.
"""
streak = 0
for _ in range(1):
streak = runner.systemic_outage_streak("review-evidence-invalid", streak)
assert streak < runner.DEFAULT_OUTAGE_STREAK, "fixture must not reach the breaker"
results = _arms(review=[_cell(resolved=False, ok=False, error_kind="review-evidence-invalid")])
with pytest.raises(SystemExit) as exc:
runner.enforce_measurement_health(results, {"review"})
assert exc.value.code == 1
def test_finalization_reports_every_arm_and_names_no_cause(capsys):
"""Status for each arm; an empty artifact does not become an EROFS diagnosis."""
results = _arms(
review=[_cell(resolved=False, ok=False, error_kind="review-evidence-invalid")],
ce_review=[_cell(resolved=False, error_kind="oracle-failed")],
)
with pytest.raises(SystemExit):
runner.enforce_measurement_health(results, {"review", "ce_review"})
out = capsys.readouterr().out
assert "review: UNUSABLE" in out
assert "ce_review: OBSERVED_OK" in out
assert "cause=undetermined" in out
assert "EROFS" not in out and "mount" not in out
def test_valid_negatives_do_not_abort_finalization(capsys):
"""The 16h run's shape must survive the real guard, not just the classifier."""
scored_but_wrong = [_cell(resolved=False, error_kind="oracle-failed") for _ in range(3)]
health = runner.enforce_measurement_health(
_arms(review=scored_but_wrong, ce_review=list(scored_but_wrong)), {"review", "ce_review"}
)
assert {h.status for h in health.values()} == {"OBSERVED_OK"}
assert "UNUSABLE" not in capsys.readouterr().out
def test_reused_only_arm_is_reported_unknown_by_finalization(capsys):
runner.enforce_measurement_health(_arms(review=[_cell(reused=True)]), {"review"})
assert "review: UNKNOWN" in capsys.readouterr().out
def test_run_sweep_calls_the_health_guard_and_not_the_legacy_helper():
"""Pins the wiring the caller correction exposed.
Reads the compiled code object's global references rather than the source
text: deleting the call removes the name and fails this test, which is the
mutation check. It does NOT prove the guard runs end to end - _run_sweep
needs bwrap and a sandbox, so no test here drives it.
"""
referenced = runner._run_sweep.__code__.co_names
assert "enforce_measurement_health" in referenced
assert "broken_incumbent_arms" not in referenced
def test_ce_review_is_classified_even_though_it_is_not_a_candidate_arm():
"""ce_review is a comparator, absent from CANDIDATE_ARMS.
Dropping the `- {"review"}` exclusion alone would have left it unchecked.
"""
assert "ce_review" not in set(CANDIDATE_ARMS.values())
health = arm_health(_arms(ce_review=[_cell()]), {"review", "ce_review"})
assert "ce_review" in health
def _packed_cells(tasks: int, runs: int, arms: tuple[str, ...]) -> list[tuple[str, int, str]]:
return [(f"t{t}", r, a) for t in range(tasks) for r in range(runs) for a in arms]
def test_packed_sweep_runs_every_cell_and_folds_in_submission_order():
"""Fold order is the contract the breaker rests on.
Cells finish in whatever order the pool returns them, but the breaker counts
CONSECUTIVE systemic failures, which only means something in a fixed order.
"""
cells = _packed_cells(3, 2, ("review", "candidate_review"))
folded: list[tuple[str, int, str]] = []
streak, tripped = runner.sweep_packed_cells(
cells,
workers=4,
run=lambda task, run_idx, arm: {"error_kind": None, "review_evidence_valid": True},
on_start=lambda *_: None,
on_record=lambda task, run_idx, arm, _rec: folded.append((task, run_idx, arm)),
outage_streak=0,
outage_limit=0,
)
assert folded == cells
assert (streak, tripped) == (0, False)
def test_packed_sweep_trips_the_breaker_on_the_same_cell_waves_would():
"""Packing must not change WHEN a doomed run aborts, only how it is fed."""
cells = _packed_cells(3, 3, ("review",))
fail_from = 2
folded: list[int] = []
def run(task: str, run_idx: int, arm: str) -> dict[str, Any]:
index = cells.index((task, run_idx, arm))
systemic = index >= fail_from
return {
"error_kind": "session-error" if systemic else None,
"review_evidence_valid": not systemic,
}
streak, tripped = runner.sweep_packed_cells(
cells,
workers=2,
run=run,
on_start=lambda *_: None,
on_record=lambda t, r, a, _rec: folded.append(cells.index((t, r, a))),
outage_streak=0,
outage_limit=runner.DEFAULT_OUTAGE_STREAK,
)
assert tripped is True
assert streak == runner.DEFAULT_OUTAGE_STREAK
# Five consecutive systemic failures starting at index 2 -> trips on index 6.
assert folded[-1] == fail_from + runner.DEFAULT_OUTAGE_STREAK - 1
assert folded == sorted(folded), "records must fold in submission order"
def test_packed_sweep_skips_a_task_whose_assets_never_arrive():
"""A task that cannot be prepared is skipped, not run against nothing."""
cells = _packed_cells(3, 2, ("review",))
ran: list[str] = []
runner.sweep_packed_cells(
cells,
workers=3,
run=lambda task, run_idx, arm: ran.append(task)
or {"error_kind": None, "review_evidence_valid": True},
on_start=lambda *_: None,
on_record=lambda *_: None,
outage_streak=0,
outage_limit=0,
await_ready=lambda task: task != "t1",
)
assert set(ran) == {"t0", "t2"}
assert "t1" not in ran
def test_packed_sweep_workers_inherit_the_runs_cancellation_event():
"""A worker that cannot see the event runs on after the sweep is cancelled.
The cells are submitted from a producer THREAD, and a new thread starts with
an empty context - so copying the context at submission copies the wrong one
unless the caller's is captured first. run_managed falls back to
_CANCELLATION when no event is passed, which is how a cell's subprocesses
learn the run was cancelled at all.
"""
seen: list[threading.Event | None] = []
event = threading.Event()
with cancellation_scope(event):
runner.sweep_packed_cells(
_packed_cells(2, 1, ("review",)),
workers=2,
run=lambda *_: seen.append(_CANCELLATION.get()) or {"error_kind": None},
on_start=lambda *_: None,
on_record=lambda *_: None,
outage_streak=0,
outage_limit=0,
)
assert seen and all(observed is event for observed in seen)
def test_packed_sweep_window_must_keep_the_pool_fed():
with pytest.raises(ValueError, match="window must be at least workers"):
runner.sweep_packed_cells(
_packed_cells(1, 1, ("review",)),
workers=4,
run=lambda *_: {"error_kind": None},
on_start=lambda *_: None,
on_record=lambda *_: None,
outage_streak=0,
outage_limit=0,
window=2,
)
def test_a_raising_packed_cell_still_persists_its_settled_siblings():
"""A crash in one cell must not erase the evidence of cells that finished.
run_cell deliberately lets unexpected harness exceptions propagate, and the
wave scheduler answers that by folding every non-failing sibling before it
re-raises. The packed scheduler has to hold the same contract: the later
cells already ran and already cost money, so losing their rows would mean
paying for evidence the sweep then throws away.
"""
folded: list[tuple[int, str]] = []
started = threading.Event()
def run(task_id: str, run_idx: int, arm: str) -> dict[str, Any]:
if run_idx == 0:
# Let the later cell finish first, so there is settled evidence to
# lose at the moment this one raises.
started.wait(timeout=5)
raise RuntimeError("harness bug in cell 0")
started.set()
return {"error_kind": None}
with pytest.raises(RuntimeError, match="harness bug in cell 0"):
runner.sweep_packed_cells(
_packed_cells(1, 2, ("review",)),
workers=2,
run=run,
on_start=lambda *_: None,
on_record=lambda task_id, run_idx, arm, _rec: folded.append((run_idx, arm)),
outage_streak=0,
outage_limit=0,
)
assert (1, "review") in folded, "the sibling that completed was never recorded"
def test_an_uninvoked_skill_still_counts_toward_the_arm_median():
"""Pins a KNOWN GAP, not a desired behaviour.
A cell whose skill never ran still moves the arm's quality median, even
though an arm exists to measure a SKILL. The narrow fix - filtering those
rows out of the quality metrics - is worse than the gap: valid_runs and
excluded_runs keep counting them, so the promotion gate sees N clean runs
while the median came from fewer. Since the dropped rows are systematically
an arm's worst, that biases toward promoting, and it was measured flipping
keep_incumbent to promote.
Closing it honestly needs a scored-run count and a paired-equality check in
the promotion gate. Pinned here so the half-fix cannot be reapplied without
someone reading why it was reverted.
"""
good = record(review_weighted_f1=1.0, cost_usd=2.0)
uninvoked = record(
review_weighted_f1=0.0, cost_usd=4.0, error_kind="skill-not-invoked", skill_invoked=False
)
agg = aggregate([good, uninvoked])
assert agg["review_weighted_f1"] == 0.5, "the uninvoked row is counted - the known gap"
assert agg["cost_usd"] == 3.0
# The invariant that makes the half-fix unsafe: the median and the run count
# the gate reads must cover the same rows.
assert agg["valid_runs"] == 2

531
eval/tests/test_workflow_bench_evolution.py
View file

@ -8,14 +8,18 @@ from types import SimpleNamespace
import pytest
from workflow_bench.evolution import (
CANDIDATE_SKILLS,
MAX_CANDIDATE_ENTRIES,
apply_candidate_overlay,
candidate_overlay_digest,
evaluate_candidate,
evaluate_review_candidate,
required_candidate_arms,
seed_evaluated_skills,
skill_fingerprint,
unexercised_overlay_skills,
)
from workflow_bench.promotion_apply import mirror_targets
from workflow_bench.process_control import ManagedProcessResult
from workflow_bench.runner import aggregate, build_parser
@ -185,8 +189,7 @@ def test_candidate_overlay_is_skill_only_and_content_addressed(tmp_path):
review_skill = review_overlay / ".claude" / "skills" / "gitnexus-review" / "SKILL.md"
review_skill.parent.mkdir(parents=True)
review_skill.write_text("review candidate\n")
with pytest.raises(ValueError, match="plan,work"):
candidate_overlay_digest(review_overlay)
assert candidate_overlay_digest(review_overlay)
invalid = tmp_path / "invalid"
source = invalid / "gitnexus" / "src" / "cli" / "index.ts"
@ -238,6 +241,116 @@ def test_required_candidate_arms_are_minimal_for_touched_skills(tmp_path):
"candidate_workflow_direct",
]
review = tmp_path / "review"
write_overlay_skill(review, "gitnexus-review")
assert required_candidate_arms(review) == ["candidate_review"]
def test_review_gate_is_quality_first_and_requires_repeated_evidence():
def arm(score, blocker=1.0, false_positives=0, runs=3):
return {
"runs": runs,
"valid_runs": runs,
"excluded_runs": 0,
"class": "review-defect",
"review_weighted_f1": score,
"review_blocker_recall": blocker,
"review_false_positives": false_positives,
"review_clean_control": False,
"review_verdict_correct": True,
}
decision = evaluate_review_candidate(
{
"case-a": {
"review": arm(0.6),
"candidate_review": arm(0.8),
}
},
incumbent_arm="review",
candidate_arm="candidate_review",
model="pinned-model",
)
assert decision["decision"] == "promote"
regression = evaluate_review_candidate(
{
"case-a": {
"review": arm(0.6, blocker=1.0),
"candidate_review": arm(0.8, blocker=0.0),
}
},
incumbent_arm="review",
candidate_arm="candidate_review",
model="pinned-model",
)
assert regression["decision"] == "keep_incumbent"
assert any("blocker recall" in reason for reason in regression["reasons"])
def test_review_gate_rejects_added_false_positives_on_clean_controls():
base = {
"runs": 3,
"valid_runs": 3,
"excluded_runs": 0,
"class": "review-clean",
"review_weighted_f1": 1.0,
"review_blocker_recall": 1.0,
"review_clean_control": True,
"review_clean_pass": True,
"review_verdict_correct": True,
}
decision = evaluate_review_candidate(
{
"clean": {
"review": {**base, "review_false_positives": 0},
"candidate_review": {**base, "review_false_positives": 1, "review_clean_pass": False},
}
},
incumbent_arm="review",
candidate_arm="candidate_review",
model="pinned-model",
)
assert decision["decision"] == "keep_incumbent"
assert any("clean control" in reason for reason in decision["reasons"])
@pytest.mark.parametrize("wrong_verdict,bad_blocker", [(True, False), (False, True), (False, False)])
def test_review_gate_preserves_every_repeat_safeguard(wrong_verdict, bad_blocker):
from workflow_bench.runner import aggregate
def row(score, verdict=True, blocker=1.0):
return {
"resolved": True,
"review_weighted_f1": score,
"review_blocker_recall": blocker,
"review_false_positives": 0,
"review_verdict_correct": verdict,
"review_clean_control": False,
"review_clean_pass": False,
}
incumbent = aggregate([row(0.5) for _ in range(3)])
candidate = aggregate([row(0.8), row(0.8), row(0.8, not wrong_verdict, 0.0 if bad_blocker else 1.0)])
decision = evaluate_review_candidate(
{"case": {"review": incumbent, "candidate_review": candidate}},
incumbent_arm="review",
candidate_arm="candidate_review",
model="pinned-model",
)
assert decision["decision"] == ("keep_incumbent" if wrong_verdict or bad_blocker else "promote")
def test_review_gate_treats_an_empty_corpus_as_insufficient_evidence():
decision = evaluate_review_candidate(
{},
incumbent_arm="review",
candidate_arm="candidate_review",
model="pinned-model",
)
assert decision["decision"] == "insufficient_evidence"
assert any("no paired review task results" in reason for reason in decision["reasons"])
@pytest.mark.skipif(os.name == "nt", reason="candidate overlays require the Linux outer sandbox")
def test_apply_candidate_overlay_creates_a_clean_ephemeral_commit(tmp_path):
@ -314,6 +427,12 @@ def test_apply_candidate_overlay_creates_a_clean_ephemeral_commit(tmp_path):
) == candidate_overlay_digest(overlay)
assert incumbent.read_text() == "candidate\n"
git_commands = [command for command in sandbox.commands if command[0] == "/usr/bin/git"]
assert git_commands[0][-4:] == [
"add",
"-f",
"--",
".claude/skills/gitnexus-work/SKILL.md",
]
assert [command[-1] for command in git_commands[:2]] == [
".claude/skills/gitnexus-work/SKILL.md",
"--",
@ -349,6 +468,181 @@ def test_candidate_overlay_rejects_linked_destination_parents(tmp_path):
apply_candidate_overlay(overlay, repo, sandbox=sandbox)
@pytest.mark.skipif(os.name == "nt", reason="candidate overlays require the Linux outer sandbox")
def test_apply_candidate_overlay_force_adds_historically_ignored_skill(tmp_path):
repo = tmp_path / "repo"
repo.mkdir()
subprocess.run(["git", "init", "--quiet", str(repo)], check=True)
(repo / ".gitignore").write_text(".claude/skills/*\n")
(repo / "README").write_text("subject\n")
subprocess.run(["git", "-C", str(repo), "add", "."], check=True)
subprocess.run(
[
"git",
"-C",
str(repo),
"-c",
"user.name=test",
"-c",
"user.email=test@invalid",
"commit",
"--quiet",
"-m",
"historical checkout that ignores skills",
],
check=True,
)
overlay = tmp_path / "candidate"
write_overlay_skill(overlay, "gitnexus-review")
class LocalSandbox:
def __init__(self):
self.clone = repo
def run(self, command, **kwargs):
if command[0] == "/bin/mkdir":
return ManagedProcessResult(
state="exited",
returncode=0,
stdout_tail="",
stderr_tail="",
duration_s=0.0,
)
translated = [str(repo) if item == "/workspace" else item for item in command]
completed = subprocess.run(
translated,
cwd=repo,
env=dict(kwargs["env"]),
capture_output=True,
text=True,
check=False,
)
return ManagedProcessResult(
state="exited",
returncode=completed.returncode,
stdout_tail=completed.stdout,
stderr_tail=completed.stderr,
duration_s=0.0,
)
apply_candidate_overlay(overlay, repo, sandbox=LocalSandbox())
assert (repo / ".claude" / "skills" / "gitnexus-review" / "SKILL.md").read_text() == (
"gitnexus-review candidate\n"
)
status = subprocess.run(
["git", "-C", str(repo), "status", "--porcelain"],
check=True,
capture_output=True,
text=True,
)
assert status.stdout == ""
@pytest.mark.skipif(os.name == "nt", reason="skill seeds require the Linux outer sandbox")
def test_seed_evaluated_skills_installs_missing_review_skill_and_is_idempotent(tmp_path):
repo = tmp_path / "clone"
repo.mkdir()
subprocess.run(["git", "init", "--quiet", str(repo)], check=True)
# Historical review SHAs ignore the whole skill tree and lack today's
# `!.claude/skills/gitnexus-review/` allowlist. Seeding must still commit.
(repo / ".gitignore").write_text(".claude/skills/*\n")
(repo / "README").write_text("subject\n")
subprocess.run(["git", "-C", str(repo), "add", "."], check=True)
subprocess.run(
[
"git",
"-C",
str(repo),
"-c",
"user.name=test",
"-c",
"user.email=test@invalid",
"commit",
"--quiet",
"-m",
"historical checkout without review skill",
],
check=True,
)
before = subprocess.run(
["git", "-C", str(repo), "rev-parse", "HEAD"],
check=True,
capture_output=True,
text=True,
).stdout.strip()
source = tmp_path / "harness"
skill = source / ".claude" / "skills" / "gitnexus-review" / "SKILL.md"
persona = source / ".claude" / "skills" / "gitnexus-review" / "ci-personas" / "lens.md"
persona.parent.mkdir(parents=True)
skill.write_text("current incumbent review skill\n")
persona.write_text("persona\n")
class LocalSandbox:
def __init__(self):
self.clone = repo
def run(self, command, **kwargs):
if command[0] == "/bin/mkdir":
return ManagedProcessResult(
state="exited",
returncode=0,
stdout_tail="",
stderr_tail="",
duration_s=0.0,
)
translated = [str(repo) if item == "/workspace" else item for item in command]
completed = subprocess.run(
translated,
cwd=repo,
env=dict(kwargs["env"]),
capture_output=True,
text=True,
check=False,
)
return ManagedProcessResult(
state="exited",
returncode=completed.returncode,
stdout_tail=completed.stdout,
stderr_tail=completed.stderr,
duration_s=0.0,
)
sandbox = LocalSandbox()
seed_evaluated_skills(source, repo, sandbox=sandbox, arm="review")
assert skill_fingerprint(repo, "review") is not None
assert (repo / ".claude" / "skills" / "gitnexus-review" / "SKILL.md").read_text() == (
"current incumbent review skill\n"
)
assert (
repo / ".claude" / "skills" / "gitnexus-review" / "ci-personas" / "lens.md"
).read_text() == "persona\n"
status = subprocess.run(
["git", "-C", str(repo), "status", "--porcelain"],
check=True,
capture_output=True,
text=True,
)
assert status.stdout == ""
after = subprocess.run(
["git", "-C", str(repo), "rev-parse", "HEAD"],
check=True,
capture_output=True,
text=True,
).stdout.strip()
assert after != before
seed_evaluated_skills(source, repo, sandbox=sandbox, arm="review")
again = subprocess.run(
["git", "-C", str(repo), "rev-parse", "HEAD"],
check=True,
capture_output=True,
text=True,
).stdout.strip()
assert again == after
@pytest.mark.skipif(os.name == "nt", reason="skill links are rejected by the Linux sandbox harness")
def test_skill_fingerprint_rejects_linked_skill_roots(tmp_path):
outside = tmp_path / "outside"
@ -424,9 +718,7 @@ def test_candidate_gate_refuses_promotion_on_unmeasured_cost():
results = {
"task-a": {
"workflow_direct": aggregate([record(cost_usd=1.0) for _ in range(3)]),
"candidate_workflow_direct": aggregate(
[record(cost_usd=0.1), record(cost_usd=None), record(cost_usd=0.1)]
),
"candidate_workflow_direct": aggregate([record(cost_usd=0.1), record(cost_usd=None), record(cost_usd=0.1)]),
}
}
decision = evaluate_candidate(
@ -510,8 +802,23 @@ def test_candidate_gate_rejects_a_partial_candidate_even_with_a_resolution_edge(
assert any("oracle-backed quality floor" in reason for reason in decision["reasons"])
@pytest.mark.parametrize("resolved", [0, 2])
def test_candidate_gate_never_promotes_zero_or_partial_success_for_efficiency(resolved):
@pytest.mark.parametrize(
("resolved", "expected_decision", "expected_reason"),
[
# Nothing resolved anywhere: the task is ungated, which leaves the
# generation with no quality signal at all — refuse outright rather
# than rank a 100x cost win across runs that all failed the oracle.
(0, "insufficient_evidence", "no task supplied quality signal"),
# Partial success on a task the incumbent also partly resolves stays a
# quality-floor rejection: the candidate has to be reliable, not lucky.
(2, "keep_incumbent", "oracle-backed quality floor"),
],
)
def test_candidate_gate_never_promotes_zero_or_partial_success_for_efficiency(
resolved,
expected_decision,
expected_reason,
):
incumbent_records = [record(cost_usd=1.0, resolved=index < resolved) for index in range(3)]
candidate_records = [record(cost_usd=0.01, resolved=index < resolved) for index in range(3)]
decision = evaluate_candidate(
@ -526,9 +833,184 @@ def test_candidate_gate_never_promotes_zero_or_partial_success_for_efficiency(re
model="pinned-model",
)
assert decision["decision"] == "keep_incumbent"
assert decision["decision"] == expected_decision
assert decision["tasks"][0]["candidate_quality_floor_met"] is False
assert any("oracle-backed quality floor" in reason for reason in decision["reasons"])
assert any(expected_reason in reason for reason in decision["reasons"])
def test_a_task_no_arm_can_resolve_is_reported_but_does_not_veto_promotion():
# inv-feature-list-repos-filter fails its hidden oracle on every run of
# both arms. Gating on it made promotion unreachable for as long as it
# stayed in the set, while saying nothing about the candidate.
solvable = {
"workflow": aggregate([record(cost_usd=1.0, resolved=index > 1) for index in range(3)]),
"candidate_workflow": aggregate([record(cost_usd=1.0) for _ in range(3)]),
}
unsolvable = {
"workflow": aggregate([record(cost_usd=1.0, resolved=False) for _ in range(3)]),
"candidate_workflow": aggregate([record(cost_usd=1.3, resolved=False) for _ in range(3)]),
}
decision = evaluate_candidate(
{"task-a": solvable, "task-impossible": unsolvable},
incumbent_arm="workflow",
candidate_arm="candidate_workflow",
model="pinned-model",
)
assert decision["decision"] == "promote"
assert decision["ungated_tasks"] == ["task-impossible"]
assert decision["gated_tasks"] == ["task-a"]
assert [row["gated"] for row in decision["tasks"]] == [True, False]
# The ungated task's 30% cost regression stays under the failed-task cap
# but must not reach the median or the (tighter) gated per-task cap.
assert decision["median_improvement_pct"] == 0.0
assert not any("above the" in reason for reason in decision["reasons"])
# One aggregate line, so a growing set of unsolvable tasks cannot crowd the
# real verdict out of the three reasons the proposer is shown — and it
# discloses how much of the set the verdict actually rests on.
assert [reason for reason in decision["reasons"] if "not gated on" in reason] == [
"not gated on 1 task(s) neither arm resolved: task-impossible (evidence base: 1/2 paired tasks gated)"
]
def test_an_ungated_task_still_ranks_against_the_failed_task_cost_cap():
# Leaving the quality gate is not leaving the spend gate: burning 9x the
# incumbent's cost to fail the same oracle is a regression the gate has to
# see, or a candidate can hide unbounded waste inside "task health".
solvable = {
"workflow": aggregate([record(cost_usd=1.0, resolved=index > 1) for index in range(3)]),
"candidate_workflow": aggregate([record(cost_usd=1.0) for _ in range(3)]),
}
unsolvable = {
"workflow": aggregate([record(cost_usd=1.0, resolved=False) for _ in range(3)]),
"candidate_workflow": aggregate([record(cost_usd=9.0, resolved=False) for _ in range(3)]),
}
decision = evaluate_candidate(
{"task-a": solvable, "task-impossible": unsolvable},
incumbent_arm="workflow",
candidate_arm="candidate_workflow",
model="pinned-model",
)
assert decision["decision"] == "keep_incumbent"
assert decision["ungated_tasks"] == ["task-impossible"]
assert any("failed-task cap" in reason for reason in decision["reasons"])
def test_a_mutually_failed_task_stays_gated_when_the_skill_never_loaded():
# skill-not-invoked is prompt evidence, not task health: the skill under
# test never ran, so the task cannot be written off as beyond both arms.
results = {
"task-a": {
"workflow": aggregate([record(cost_usd=1.0, resolved=False) for _ in range(3)]),
"candidate_workflow": aggregate(
[record(cost_usd=0.01, resolved=False, error_kind="skill-not-invoked") for _ in range(3)]
),
}
}
decision = evaluate_candidate(
results,
incumbent_arm="workflow",
candidate_arm="candidate_workflow",
model="pinned-model",
)
assert decision["ungated_tasks"] == []
assert decision["tasks"][0]["gated"] is True
assert decision["tasks"][0]["skill_attributable_failure"] is True
# Gated with teeth: the 99% cost "win" must not carry a candidate whose
# skill never loaded.
assert decision["decision"] == "keep_incumbent"
assert any("never invoked the skill under test" in reason for reason in decision["reasons"])
def test_a_mutually_failed_task_stays_gated_when_its_metric_was_never_measured():
# Ungating is a claim about spend as well as quality. With no measured
# cost there is nothing to claim, so the task stays in the gate and the
# missing measurement is named instead of silently skipped.
results = {
"task-a": {
"workflow": aggregate([record(cost_usd=1.0, resolved=False) for _ in range(3)]),
"candidate_workflow": aggregate(
[record(cost_usd=None, resolved=False), *(record(cost_usd=0.1, resolved=False) for _ in range(2))]
),
}
}
decision = evaluate_candidate(
results,
incumbent_arm="workflow",
candidate_arm="candidate_workflow",
model="pinned-model",
)
assert decision["ungated_tasks"] == []
assert decision["decision"] == "insufficient_evidence"
assert any("was not measured on every run" in reason for reason in decision["reasons"])
def test_partial_progress_on_a_task_the_incumbent_never_resolves_is_not_punished():
# Resolving 1 of 3 runs where the incumbent resolves none is strictly
# better than resolving none — which the gate ungates and forgives. Holding
# the partial run to the quality floor made improvement score worse than
# inaction.
def outcome(candidate_resolved: int) -> dict[str, object]:
return evaluate_candidate(
{
"task-a": {
"workflow": aggregate([record(cost_usd=1.0) for _ in range(3)]),
"candidate_workflow": aggregate([record(cost_usd=0.5) for _ in range(3)]),
},
"task-hard": {
"workflow": aggregate([record(cost_usd=1.0, resolved=False) for _ in range(3)]),
"candidate_workflow": aggregate(
[record(cost_usd=1.0, resolved=index < candidate_resolved) for index in range(3)]
),
},
},
incumbent_arm="workflow",
candidate_arm="candidate_workflow",
model="pinned-model",
)
no_progress = outcome(0)
some_progress = outcome(1)
assert no_progress["decision"] == "promote"
assert no_progress["ungated_tasks"] == ["task-hard"]
# The partial run gives the task quality signal, so it is gated — but as
# improvement, not as a floor failure the zero-progress candidate escapes.
assert some_progress["decision"] == "promote"
assert some_progress["ungated_tasks"] == []
assert some_progress["tasks"][1]["quality_floor_enforced"] is False
assert not any("quality floor" in reason for reason in some_progress["reasons"])
def test_promotion_requires_a_gated_majority_of_the_paired_tasks():
# Two of three tasks written off as task health leaves one task deciding
# the whole promotion. Ungating keeps promotion reachable; it must not
# hollow out the evidence base that makes a promotion mean anything.
solvable = {
"workflow": aggregate([record(cost_usd=1.0, resolved=index > 1) for index in range(3)]),
"candidate_workflow": aggregate([record(cost_usd=0.1) for _ in range(3)]),
}
unsolvable = {
"workflow": aggregate([record(cost_usd=1.0, resolved=False) for _ in range(3)]),
"candidate_workflow": aggregate([record(cost_usd=1.0, resolved=False) for _ in range(3)]),
}
decision = evaluate_candidate(
{"task-a": solvable, "task-impossible": unsolvable, "task-impossible-2": dict(unsolvable)},
incumbent_arm="workflow",
candidate_arm="candidate_workflow",
model="pinned-model",
)
assert decision["decision"] == "insufficient_evidence"
assert decision["gated_tasks"] == ["task-a"]
assert any("evidence base is too thin" in reason for reason in decision["reasons"])
def test_candidate_gate_promotes_on_a_two_run_resolution_margin():
@ -553,3 +1035,34 @@ def test_overlay_skills_must_be_exercised_by_selected_candidate_arms(tmp_path):
write_overlay_skill(plan_overlay, "gitnexus-plan")
assert unexercised_overlay_skills(plan_overlay, ["candidate_workflow_direct"]) == ["gitnexus-plan"]
assert unexercised_overlay_skills(plan_overlay, ["candidate_workflow"]) == []
@pytest.mark.parametrize("skill", sorted(CANDIDATE_SKILLS))
def test_a_promoted_skill_is_visible_to_git_status_in_every_shipped_tree(skill):
"""A promotion the repository cannot see is a promotion that never happens.
The workflow detects an applied promotion with `git status --porcelain`,
which is blind to ignored paths, and `.claude/skills/*` is ignored with a
hand-maintained per-skill allowlist. A candidate skill missing from that
allowlist would leave the run reporting "No promotion this run" after the
gate had already said promote — silently, and only after a full generation
of benchmark spend.
"""
repo_root = Path(__file__).resolve().parents[2]
from pathlib import PurePosixPath
targets = [
str(path.parent)
for path in mirror_targets(PurePosixPath(".claude/skills") / skill / "SKILL.md")
]
ignored = [
target
for target in targets
if subprocess.run(
["git", "check-ignore", "-q", f"{target}/SKILL.md"],
cwd=repo_root,
check=False,
).returncode
== 0
]
assert ignored == []

343
eval/tests/test_workflow_bench_sessions.py
View file

@ -4,6 +4,7 @@ import argparse
import hashlib
import json
import os
import shutil
import subprocess
import sys
from pathlib import Path
@ -11,9 +12,10 @@ from types import SimpleNamespace
import pytest
from workflow_bench import evolve, runner, runner_sessions, runtime_mounts
from workflow_bench import evolve, runner, runner_artifacts, runner_sessions, runtime_mounts
from workflow_bench.evolution import skill_fingerprint
from workflow_bench.process_control import ManagedProcessResult
from workflow_bench.process_control import ManagedProcessError, ManagedProcessResult
from workflow_bench.proposer_sandbox import SandboxError
from workflow_bench.runner import snapshot_plan_docs
@ -71,6 +73,7 @@ def bench_args(**overrides):
"claude_bin": "claude",
"timeout": 5,
"model": None,
"effort": "xhigh",
"base_url": None,
"auth_token": None,
"permission_mode": None,
@ -117,10 +120,16 @@ def skill_events(skill_input: dict, *, tool_id: str = "skill-1", is_error: bool
def fake_sandbox(root: Path) -> SimpleNamespace:
# private_root is NOT the clone. Conflating them puts the review artifact
# directory inside the workspace, which the real sandbox never does and
# which hides whether the workspace was left untouched.
private_root = root.parent / f"{root.name}-sandbox-private"
private_root.mkdir(exist_ok=True)
return SimpleNamespace(
backend="test-double",
claude_bin="claude",
clone=root,
private_root=root,
private_root=private_root,
command_prefix=[],
command_prefix_for=lambda **_kwargs: [],
settings_json="{}",
@ -167,6 +176,25 @@ def test_run_claude_forwards_the_named_model_to_every_session(monkeypatch, tmp_p
assert captured[captured.index("--model") + 1] == "claude-sonnet-4-20250514"
def test_run_claude_forwards_xhigh_effort_to_every_session(monkeypatch, tmp_path):
captured: list[str] = []
def fake_run(command, **kwargs):
captured.extend(command)
return fake_cli_result(VALID_REPORT)
monkeypatch.setattr(runner_sessions, "run_managed", fake_run)
runner.run_claude(
"task",
tmp_path,
claude_bin="claude",
timeout=5,
model="gpt-5.6-sol",
effort="xhigh",
)
assert captured[captured.index("--effort") + 1] == "xhigh"
def test_run_claude_restricts_tools_via_tools_flag_outside_bare(monkeypatch, tmp_path):
# Outside --bare, the built-in toolset defaults to everything (subagents,
# WebFetch, Task, ...) and --allowedTools only pre-approves within that —
@ -295,10 +323,18 @@ def test_run_arm_keeps_session_error_kind_over_verify(monkeypatch, tmp_path):
def test_agent_tool_grants_are_exact_and_nomcp_has_no_graph_tools(monkeypatch, tmp_path):
read_only = runner.allowed_agent_tools(implementation=False)
review_tools = runner.allowed_agent_tools(implementation=False, allow_edit=False)
implementation = runner.allowed_agent_tools(implementation=True)
no_mcp = runner.allowed_agent_tools(implementation=True, include_mcp=False)
assert read_only == [*runner.BUILTIN_AGENT_TOOLS, *runner.GITNEXUS_READ_ONLY_TOOLS]
assert review_tools == [
tool
for tool in [*runner.BUILTIN_AGENT_TOOLS, *runner.GITNEXUS_READ_ONLY_TOOLS]
if tool != "Edit"
]
assert "Write" in review_tools
assert "Edit" not in review_tools
assert implementation == [
*runner.BUILTIN_AGENT_TOOLS,
*runner.GITNEXUS_READ_ONLY_TOOLS,
@ -326,7 +362,7 @@ def test_agent_tool_grants_are_exact_and_nomcp_has_no_graph_tools(monkeypatch, t
)
assert captured[0]["allowed_tools"] == read_only # planning
assert captured[1]["allowed_tools"] == read_only # review
assert captured[1]["allowed_tools"] == review_tools # review
assert captured[2]["allowed_tools"] == implementation
assert captured[3]["allowed_tools"] == list(runner.BUILTIN_AGENT_TOOLS)
assert captured[3]["mcp_config_json"] == '{"mcpServers":{}}'
@ -355,7 +391,7 @@ def test_mcp_config_uses_only_the_minimal_pinned_harness_runtime(monkeypatch, tm
directory.mkdir(parents=True)
(runtime / "dist" / "cli" / "index.js").write_text("")
(runtime / "hooks" / "claude" / "resolve-analyze-cmd.cjs").write_text("")
(runtime / "package.json").write_text(json.dumps({"version": runner.PINNED_GITNEXUS_VERSION}))
(runtime / "package.json").write_text(json.dumps({"version": "9.9.9-test"}))
(runtime / "node_modules" / "gitnexus-shared").symlink_to(shared, target_is_directory=True)
(shared / "package.json").write_text(json.dumps({"name": "gitnexus-shared"}))
monkeypatch.setattr(runtime_mounts, "HARNESS_ROOT", tmp_path)
@ -379,9 +415,6 @@ def test_mcp_config_uses_only_the_minimal_pinned_harness_runtime(monkeypatch, tm
(shared / "package.json", f"{runner.SANDBOX_GITNEXUS_SHARED}/package.json"),
(runtime / "hooks" / "claude", f"{runner.SANDBOX_GITNEXUS}/hooks/claude"),
]
package = json.loads((runtime / "package.json").read_text())
assert package["version"] == runner.PINNED_GITNEXUS_VERSION
mounted_sources = {mount.source for mount in mounts}
mounted_targets = {mount.target for mount in mounts}
assert runtime not in mounted_sources
@ -399,6 +432,118 @@ def test_mcp_config_uses_only_the_minimal_pinned_harness_runtime(monkeypatch, tm
assert f"{runner.SANDBOX_GITNEXUS}/hooks" not in mounted_targets
def _install_pinned_runtime(root: Path) -> None:
runtime = root / "gitnexus"
shared = root / "gitnexus-shared"
for directory in (
runtime / "dist" / "cli",
runtime / "node_modules",
runtime / "vendor",
runtime / "hooks" / "claude",
shared / "dist",
):
directory.mkdir(parents=True)
(runtime / "dist" / "cli" / "index.js").write_text("")
(runtime / "hooks" / "claude" / "resolve-analyze-cmd.cjs").write_text("")
(runtime / "package.json").write_text(json.dumps({"version": "9.9.9-test"}))
(runtime / "node_modules" / "gitnexus-shared").symlink_to(shared, target_is_directory=True)
(shared / "package.json").write_text(json.dumps({"name": "gitnexus-shared"}))
def test_runtime_mounts_reuse_primary_checkout_node_modules_from_a_worktree(
monkeypatch, tmp_path
) -> None:
primary = tmp_path / "primary"
worktree = tmp_path / "worktree"
_install_pinned_runtime(primary)
(primary / ".git" / "worktrees" / "wt").mkdir(parents=True)
_install_pinned_runtime(worktree)
shutil.rmtree(worktree / "gitnexus" / "node_modules")
(worktree / "gitnexus" / "node_modules").symlink_to(
primary / "gitnexus" / "node_modules",
target_is_directory=True,
)
(worktree / ".git").write_text(f"gitdir: {primary / '.git' / 'worktrees' / 'wt'}\n")
monkeypatch.setattr(runtime_mounts, "HARNESS_ROOT", worktree)
mounts = runner.trusted_gitnexus_runtime_mounts()
by_target = {mount.target: mount.source for mount in mounts}
assert by_target[f"{runner.SANDBOX_GITNEXUS}/node_modules"] == (
primary / "gitnexus" / "node_modules"
)
assert by_target[f"{runner.SANDBOX_GITNEXUS_SHARED}/package.json"] == (
primary / "gitnexus-shared" / "package.json"
)
assert by_target[f"{runner.SANDBOX_GITNEXUS}/dist"] == worktree / "gitnexus" / "dist"
def test_runtime_mounts_reuse_primary_shared_when_only_the_inner_link_points_there(
monkeypatch, tmp_path
) -> None:
primary = tmp_path / "primary"
worktree = tmp_path / "worktree"
_install_pinned_runtime(primary)
(primary / ".git" / "worktrees" / "wt").mkdir(parents=True)
_install_pinned_runtime(worktree)
linked = worktree / "gitnexus" / "node_modules" / "gitnexus-shared"
linked.unlink()
linked.symlink_to(primary / "gitnexus-shared", target_is_directory=True)
(worktree / ".git").write_text(f"gitdir: {primary / '.git' / 'worktrees' / 'wt'}\n")
monkeypatch.setattr(runtime_mounts, "HARNESS_ROOT", worktree)
mounts = runner.trusted_gitnexus_runtime_mounts()
by_target = {mount.target: mount.source for mount in mounts}
assert by_target[f"{runner.SANDBOX_GITNEXUS}/node_modules"] == (
worktree / "gitnexus" / "node_modules"
)
assert by_target[f"{runner.SANDBOX_GITNEXUS_SHARED}/package.json"] == (
primary / "gitnexus-shared" / "package.json"
)
def test_runtime_mounts_reject_a_node_modules_symlink_outside_the_primary_checkout(
monkeypatch, tmp_path
) -> None:
primary = tmp_path / "primary"
worktree = tmp_path / "worktree"
outsider = tmp_path / "outsider"
_install_pinned_runtime(primary)
_install_pinned_runtime(outsider)
(primary / ".git" / "worktrees" / "wt").mkdir(parents=True)
_install_pinned_runtime(worktree)
shutil.rmtree(worktree / "gitnexus" / "node_modules")
(worktree / "gitnexus" / "node_modules").symlink_to(
outsider / "gitnexus" / "node_modules",
target_is_directory=True,
)
(worktree / ".git").write_text(f"gitdir: {primary / '.git' / 'worktrees' / 'wt'}\n")
monkeypatch.setattr(runtime_mounts, "HARNESS_ROOT", worktree)
with pytest.raises(SandboxError, match="primary checkout"):
runner.trusted_gitnexus_runtime_mounts()
def test_runtime_mounts_reject_a_node_modules_symlink_in_a_regular_checkout(
monkeypatch, tmp_path
) -> None:
checkout = tmp_path / "checkout"
other = tmp_path / "other"
_install_pinned_runtime(checkout)
_install_pinned_runtime(other)
(checkout / ".git").mkdir()
shutil.rmtree(checkout / "gitnexus" / "node_modules")
(checkout / "gitnexus" / "node_modules").symlink_to(
other / "gitnexus" / "node_modules",
target_is_directory=True,
)
monkeypatch.setattr(runtime_mounts, "HARNESS_ROOT", checkout)
with pytest.raises(SandboxError, match="must be a real directory"):
runner.trusted_gitnexus_runtime_mounts()
@pytest.mark.skipif(
os.environ.get("GITNEXUS_REQUIRE_BWRAP_CANARY") != "1",
reason="real Bubblewrap canary is mandatory in the named Ubuntu CI job",
@ -458,7 +603,11 @@ def test_real_bubblewrap_runtime_mount_imports_cli_without_exposing_checkout(tmp
assert visibility.ok, visibility.stderr_tail
assert imported.ok, imported.stderr_tail
assert analyze_imported.ok, analyze_imported.stderr_tail
assert imported.stdout_tail.strip() == runner.PINNED_GITNEXUS_VERSION
# The runtime the sandbox sees must be the one this checkout built —
# compared against the checkout itself rather than a constant, so a release
# bump cannot fail a benchmark that is running exactly what it should.
built = json.loads((runtime_mounts.HARNESS_ROOT / "gitnexus" / "package.json").read_text())
assert imported.stdout_tail.strip() == built["version"]
def test_isolated_mcp_registry_contains_only_the_sandbox_clone(tmp_path):
@ -641,6 +790,23 @@ def test_skill_invocation_parses_supported_exact_identifier_fields(skill_input):
)
def test_skill_invocation_accepts_plugin_qualified_identifier():
assert (
runner_sessions.skill_was_invoked_events(
skill_events({"skill": "compound-engineering:ce-code-review"}),
"ce-code-review",
)
is True
)
assert (
runner_sessions.skill_was_invoked_events(
skill_events({"skill": "compound-engineering:ce-plan"}),
"ce-code-review",
)
is False
)
@pytest.mark.parametrize(
"skill_input",
[
@ -967,6 +1133,35 @@ def test_final_result_event_must_be_last(monkeypatch, tmp_path):
assert "not the last event" in rec["error_detail"]["event_stream_error"]
def test_background_task_teardown_after_the_result_stays_valid_evidence(monkeypatch, tmp_path):
# Claude Code drains background-task bookkeeping after the final result
# event. Those `system` events carry no tool or usage payload, so they must
# not invalidate an otherwise complete session (run 29907431284 lost three
# runs this way, and the gate demands zero excluded runs).
teardown = [
{"type": "system", "subtype": "background_tasks_changed", "tasks": []},
{"type": "system", "subtype": "task_updated", "task_id": "bdw43oy7j", "patch": {"status": "killed"}},
{"type": "system", "subtype": "task_notification", "task_id": "bdw43oy7j", "status": "stopped"},
]
stream = event_stream(*skill_events({"skill": "gitnexus-work"})) + "".join(
json.dumps(event) + "\n" for event in teardown
)
monkeypatch.setattr(runner_sessions, "run_managed", lambda *a, **k: fake_cli_result(stream))
rec = runner.run_claude(
"task",
tmp_path,
claude_bin="claude",
timeout=5,
expected_skill="gitnexus-work",
)
assert rec["ok"] is True
assert rec["error_kind"] is None
assert rec["skill_invoked"] is True
assert rec["transcript_missing"] is False
assert "evidence_diagnostics" not in rec
def test_snapshot_plan_docs_detects_one_modified_plan_and_rejects_ambiguous_output(tmp_path):
plans = tmp_path / "docs" / "plans"
plans.mkdir(parents=True)
@ -1059,7 +1254,7 @@ def test_planning_cannot_change_source_tests_or_downstream_skill(monkeypatch, tm
@pytest.mark.parametrize(
("attack", "expected_detail"),
[
("workspace", "unauthorized workspace path"),
("workspace", "changed the read-only workspace"),
("skill", "changed the evaluated skill fingerprint"),
],
)
@ -1075,7 +1270,11 @@ def test_review_phase_rejects_workspace_or_skill_mutation(
expected_skill_digest = "expected-skill-fingerprint"
def adversarial_review(prompt, *args, **kwargs):
(tmp_path / "review-output.md").write_text("review findings")
# Write where the contract now says: the artifact directory outside the
# workspace, which is the only place the agent can write atomically.
artifact = runner.review_output_path(sandbox, runner.REVIEW_OUTPUT)
artifact.parent.mkdir(parents=True, exist_ok=True)
artifact.write_text('{"schema_version":1,"verdict":"approve","findings":[]}')
if attack == "workspace":
source.write_text("review silently changed source")
return session_record()
@ -1106,6 +1305,97 @@ def test_review_phase_rejects_workspace_or_skill_mutation(
assert expected_detail in rec["error_detail"]
def test_a_cancelled_clone_copy_does_not_fall_back_to_an_uncancellable_copytree(monkeypatch, tmp_path):
"""The reflink fallback is for a filesystem, not for a teardown.
run_managed reports cancellation as a non-OK result rather than raising, so
the fallback treated it like an unsupported reflink and started a copytree
that cannot be cancelled — waiting out exactly the full copy the outage
breaker set the cancellation event to avoid.
"""
source = tmp_path / "template"
(source / ".git").mkdir(parents=True)
parent = tmp_path / "clones"
parent.mkdir()
copied: list[object] = []
monkeypatch.setattr(
runner_artifacts,
"run_managed",
lambda *_a, **_k: ManagedProcessResult(
state="cancelled",
returncode=None,
stdout_tail="",
stderr_tail="",
duration_s=0.1,
),
)
monkeypatch.setattr(runner_artifacts.shutil, "copytree", lambda *a, **k: copied.append(a))
with pytest.raises(ManagedProcessError):
runner.copy_isolated_tree(source, parent)
assert copied == []
assert list(parent.iterdir()) == [], "the partial target must be cleaned up"
@pytest.mark.parametrize("arm", ["review", "ce_review"])
def test_run_arm_mounts_the_review_artifact_directory_outside_the_workspace(monkeypatch, tmp_path, arm):
"""A writable FILE inside a read-only directory is not a writable path.
The Write tool creates `<target>.tmp.<n>.<hex>` beside the target and
renames it, so a read-only parent fails the temp create with EROFS and the
artifact stays 0 bytes. The mount target must be the directory, and it must
sit outside the read-only workspace.
Driven through run_arm rather than rebuilt here: an expected tuple assembled
in the test passes whatever run_arm actually mounts, which is the one thing
this needs to prove.
"""
assert not runner.SANDBOX_REVIEW_OUTPUT.startswith(runner.SANDBOX_WORKSPACE + "/")
assert runner.SANDBOX_REVIEW_OUTPUT != runner.SANDBOX_WORKSPACE
verify_calls: list[dict] = []
sandbox = fake_sandbox(tmp_path)
sandbox.command_prefix_for = lambda **kwargs: verify_calls.append(kwargs) or []
def review_session(prompt, *args, **kwargs):
artifact = runner.review_output_path(sandbox, runner.REVIEW_OUTPUT)
artifact.parent.mkdir(parents=True, exist_ok=True)
artifact.write_text('{"schema_version":1,"verdict":"approve","findings":[]}')
return session_record()
monkeypatch.setattr(runner, "run_claude", review_session)
monkeypatch.setattr(runner, "skill_fingerprint", lambda *_a, **_k: "skill-digest")
monkeypatch.setattr(runner, "run_verify", lambda *a, **k: (True, "ok"))
runner.run_arm(
arm,
{"prompt": "p", "verify": "true"},
tmp_path,
bench_args(),
sandbox=sandbox,
expected_skill_digest="skill-digest",
)
review_output = runner.review_output_path(sandbox, runner.REVIEW_OUTPUT)
expected = (runner.ReadOnlyMount(source=review_output.parent, target=runner.SANDBOX_REVIEW_OUTPUT),)
# The EROFS bug is about the AGENT's write, so the mount that has to be the
# directory is the writable one on the review session — not the read-only
# exposure the verify command gets afterwards. Assert both: they are
# separate arguments to separate command prefixes.
writable = [call["extra_writable_mounts"] for call in verify_calls if "extra_writable_mounts" in call]
assert writable, "the review session must be given a writable artifact mount"
assert writable[-1] == expected, "mount the directory, not the file"
read_only = [call["extra_read_only_mounts"] for call in verify_calls if "extra_read_only_mounts" in call]
assert read_only, "the verify invocation must be given the artifact mount"
assert read_only[-1] == expected, "mount the directory, not the file"
assert not expected[0].target.startswith(f"{runner.SANDBOX_WORKSPACE}/")
# The artifact the harness later reads is the one inside that mount.
assert review_output.parent in review_output.parents
def _git(repo, *args, check=True):
return subprocess.run(["git", "-C", str(repo), *args], check=check, capture_output=True, text=True)
@ -1156,3 +1446,34 @@ def test_make_worktree_clone_has_no_tags_but_keeps_all_branches(tmp_path):
current = _git(target, "rev-parse", "HEAD").stdout.strip()
assert current == other_sha
def test_copy_isolated_tree_does_not_share_git_objects_or_refs(tmp_path):
repo = tmp_path / "repo"
repo.mkdir()
_git(repo, "init", "--quiet")
_git(repo, "checkout", "--quiet", "-b", "main")
sha = _git_commit(repo, "base")
clones = tmp_path / "clones"
clones.mkdir()
template = runner.make_worktree(repo, sha, clones)
(template / "marker.txt").write_text("template\n")
copy = runner.copy_isolated_tree(template, clones)
assert copy != template
assert (copy / "marker.txt").read_text() == "template\n"
(copy / "marker.txt").write_text("copy\n")
assert (template / "marker.txt").read_text() == "template\n"
copy_head = _git(copy, "rev-parse", "HEAD").stdout.strip()
template_head = _git(template, "rev-parse", "HEAD").stdout.strip()
assert copy_head == template_head == sha
# An equal initial HEAD is also what a shared ref namespace looks like, so
# write a ref and prove the template cannot see it. A linked worktree would
# pass every assertion above, including the alternates check — its `.git` is
# a file, so the directory inspected below simply does not exist.
_git(copy, "branch", "copy-only")
assert _git(copy, "show-ref", "--verify", "refs/heads/copy-only").returncode == 0
assert _git(template, "show-ref", "--verify", "refs/heads/copy-only", check=False).returncode != 0
assert (copy / ".git").is_dir()
alternates = copy / ".git" / "objects" / "info" / "alternates"
assert not alternates.exists()

1323
eval/uv.lock generated
View file

File diff suppressed because it is too large Load diff

298
eval/workflow_bench/README.md
View file

@ -1,4 +1,4 @@
# Workflow benchmark — observe the token savings
# Skill benchmark — evolve review quality, measure workflow cost
Measures whether the `gitnexus-plan` → `gitnexus-work` engineering workflow
actually saves tokens versus a baseline agent on the same tasks, using real
@ -8,18 +8,19 @@ report.
## What it compares
| Arm | Sessions | Notes |
| --- | --- | --- |
| `workflow` | `gitnexus-plan` on the task, then `gitnexus-work` on the produced plan | The skills must be installed (`gitnexus setup`, or repo-local `.claude/skills/`) |
| `candidate_workflow` | same sessions as `workflow`, with a candidate skill overlay | Paired with `workflow` on the same task/ref/model |
| `workflow_direct` | one `gitnexus-work` direct-mode session | The middle option — execution discipline without a planning pass |
| `candidate_workflow_direct` | same session as `workflow_direct`, with a candidate skill overlay | Paired with `workflow_direct` on the same task/ref/model |
| `ce_workflow` | `ce-plan` on the task, then `ce-work` on the produced plan | External comparator: the explicitly supplied, pinned compound-engineering plugin's plan→work family |
| `ce_workflow_direct` | one `ce-work` direct-mode session | External comparator paired with `workflow_direct` |
| `review` | one `gitnexus-review` session over local uncommitted changes | The task's `setup` applies the diff under review; the review is written to `review-output.md` so `verify` can gate on it |
| `ce_review` | one `ce-code-review` session over the same changes | External comparator paired with `review` |
| `baseline` | one session with the identical task text | `--disallowedTools Skill` so it cannot borrow the workflow; same repo, same MCP tools |
| `baseline_nomcp` | like baseline, graph tools also disallowed | Separates the workflow-discipline question from the GitNexus-tools question (off by default) |
| Arm | Sessions | Notes |
| --------------------------- | ---------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| `workflow` | `gitnexus-plan` on the task, then `gitnexus-work` on the produced plan | The skills must be installed (`gitnexus setup`, or repo-local `.claude/skills/`) |
| `candidate_workflow` | same sessions as `workflow`, with a candidate skill overlay | Paired with `workflow` on the same task/ref/model |
| `workflow_direct` | one `gitnexus-work` direct-mode session | The middle option — execution discipline without a planning pass |
| `candidate_workflow_direct` | same session as `workflow_direct`, with a candidate skill overlay | Paired with `workflow_direct` on the same task/ref/model |
| `ce_workflow` | `ce-plan` on the task, then `ce-work` on the produced plan | External comparator: the explicitly supplied, pinned compound-engineering plugin's plan→work family |
| `ce_workflow_direct` | one `ce-work` direct-mode session | External comparator paired with `workflow_direct` |
| `review` | one `gitnexus-review` session over an immutable historical PR snapshot | Emits strict `review-output.json`; hidden human labels score quality after the session |
| `candidate_review` | the same review with a `gitnexus-review` candidate overlay | Paired with `review` on the same case/ref/model/runtime |
| `ce_review` | one pinned `ce-code-review` session over the same changes | External comparator paired with both review arms |
| `baseline` | one session with the identical task text | `--disallowedTools Skill` so it cannot borrow the workflow; same repo, same MCP tools |
| `baseline_nomcp` | like baseline, graph tools also disallowed | Separates the workflow-discipline question from the GitNexus-tools question (off by default) |
Every arm runs in a fresh detached git worktree of the task's `ref`, once per
`--runs`. The model-visible `verify` command is recorded as
@ -36,7 +37,7 @@ lfg's gate and work's direct-mode triage should encode.
```bash
cd eval
export GITNEXUS_BENCH_AUTH_TOKEN="$ANTHROPIC_API_KEY"
export GITNEXUS_BENCH_ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY"
uv run --locked --extra dev python -m workflow_bench.runner \
--tasks workflow_bench/tasks.scenarios.yaml --runs 3 \
--model claude-sonnet-4-20250514
@ -120,10 +121,15 @@ digest. Files written beneath the agent's `$HOME` are never trusted as
evidence.
Bare mode is deliberately non-interactive: it does not consult a stored
Claude login/keychain or `ANTHROPIC_AUTH_TOKEN`. Supply one explicit API or
proxy key through `GITNEXUS_BENCH_AUTH_TOKEN` (preferred) or `--auth-token`;
Claude login/keychain or `ANTHROPIC_AUTH_TOKEN`. Supply an Anthropic API key
through `GITNEXUS_BENCH_ANTHROPIC_API_KEY` (preferred) or `--anthropic-api-key`;
the harness maps it to `ANTHROPIC_API_KEY` only for the trusted Claude parent
and scrubs it from agent-launched tools.
and scrubs it from agent-launched tools. `GITNEXUS_BENCH_AUTH_TOKEN` and
`--auth-token` remain as aliases. OpenAI keys are not a drop-in
replacement: pass `--openai-api-key` / `GITNEXUS_BENCH_OPENAI_API_KEY` with
`gpt-*` / `o*` / `openai/*` model ids and the harness starts a loopback
LiteLLM proxy. The OpenAI key stays on that host process; Claude still sees
only a minted `ANTHROPIC_API_KEY` plus `ANTHROPIC_BASE_URL`.
The trusted Claude CLI still needs outbound access to the explicitly supplied
model endpoint. This is not a network broker, so the CLI itself retains that
@ -133,6 +139,30 @@ Native benchmark execution is therefore Linux/WSL2-only. Evidence assembly
and hand-authored overlay preparation can happen elsewhere, but
`--initial-overlay` does not bypass containment.
For a local diagnostic inside a container that blocks user namespaces, an
operator may explicitly choose the non-containment host backend:
```bash
UNSAFE_NO_BWRAP=1 RUNS=1 ./workflow_bench/run-evolution.sh
```
This mode runs review sessions directly in disposable host worktrees and is
**not** a security boundary: it does not isolate the network or create a PID
namespace, and a session that can `chmod` can undo the workspace lock. The
harness drops write bits on the whole clone, with no carve-out, so accidental
`npm install` / analyze writes cannot invalidate review evidence. The review
artifact is not in the clone at all: it lives in a writable directory bound at
`/review-output`, outside the workspace.
Sandbox cleanup restores owner write bits before deleting the private TMPDIR,
because a session that `copytree`s the locked clone would otherwise leave
non-empty 0555 directories that `rmtree` cannot remove. Historical review
SHAs that gitignore `.claude/skills/*` are force-added when the harness seeds
or overlays the evaluated `gitnexus-review` skill.
Treat model and verifier processes as able to access host files and
credentials available to the invoking user. It is restricted to the review
benchmark, forbidden with `--apply` and whenever `CI` is set;
promotion-capable and CI runs must use Bubblewrap.
## Prompt and skill evolution loop
Prompts age as models and tool harnesses change. Treat the current skills and
@ -140,6 +170,36 @@ router thresholds as an incumbent policy, not permanent truth. Candidate
changes run offline in the same throwaway clones as the incumbent; production
skills never rewrite themselves from a live task.
On the self-hosted evolution box, `run-evolution.sh` passes
`--max-runtime-from-instance-window` and the CLI derives its own cap from
`/proc/uptime` at startup (24h EventBridge window minus a 90-minute upload
reserve), in the same breath as it starts the clock that cap is measured
against — a budget computed anywhere earlier is spent by the seconds between. A `workflow_dispatch` that lands on an
already-running instance therefore exits in-process instead of vanishing when
the box stops — a cancelled GitHub job skips even `if: always()`, which is
how run 33962002890 lost 51 finished sessions. Local runs are uncapped.
A review generation is 6 tasks × 3 arms × 3 runs. Serial workers=1 at ~19
minutes per session is a 16-hour job (run 33962002890). Two harness changes
cut that without shrinking the gate:
- **Comparator reuse.** `evolve.py` forwards the seed / prior generation as
`--reuse-results`. Incumbent `review` and `ce_review` rows are copied into
the new `results.jsonl` when model, effort, task SHA, prompt digest, oracle
bytes, incumbent skill digest, CE plugin digest, and sandbox backend still
match. Candidate arms always run. A weekly generation with an unchanged
incumbent therefore pays 18 sessions, not 54. A promotion, model change,
task-corpus change, or harness `RUNTIME_DIGEST` change invalidates the
lock and re-runs the comparators.
- **Sanitized clone templates.** Each unique task SHA is cloned and
sanitized once. Cells copy that parentless snapshot (reflink when the
filesystem allows) instead of `git clone --no-local` plus repack/prune/fsck
54 times. Isolation is a private `.git`, not a second copy of full history.
Dispatch defaults to `--workers 3` so those 18 paid cells can overlap. Size
workers to the host: a cell that loses CPU and hits the session ceiling is
an excluded run the gate refuses.
Build an overlay that mirrors only the canonical repo-local skill paths:
```text
@ -161,7 +221,7 @@ paid work. For a work overlay:
cd eval
uv run --locked --extra dev python -m workflow_bench.runner \
--tasks workflow_bench/tasks.scenarios.yaml \
--runs 3 --model claude-sonnet-4-20250514 \
--runs 3 --workers 1 --model claude-sonnet-4-20250514 \
--arms workflow candidate_workflow \
workflow_direct candidate_workflow_direct \
--candidate-overlay /tmp/gn-skill-candidate
@ -177,7 +237,7 @@ artifacts. Those artifacts are the trajectory evidence: cluster failures and
expensive detours, propose one bounded prompt change, and feed it back as the
next overlay.
When candidate arms are present the runner also writes schema-3
When candidate arms are present the runner also writes schema-6
`promotion.json`. It
binds the immutable overlay digest, benchmark model, truthful candidate origin
(a named proposer model or `manual-initial-overlay`), selected
@ -186,9 +246,36 @@ immutable dependency bytes, committed base digest of every apply
destination, exact required arms, thresholds, and evidence expiry. Its default
deterministic gate is deliberately conservative:
Schema 6 binds a separate policy to each required candidate arm and records
whether the sweep completed. Apply validates the paired metrics and recomputes
each decision. Historical schema 5 reports remain readable; regenerate their
benchmark evidence before applying an overlay. Editing a schema number does
not supply the missing evidence.
Review candidates optimize weighted F1 with a minimum improvement of 0.01,
complete paired evidence on every selected task, and no per-task quality
regression. Complete misses score zero. Matching uses maximum cardinality
throughout the 100-finding limit. Downgraded findings receive at most their
reported severity's weight; only blocking-severity matches count toward blocker
recall. Every valid candidate repeat must have the correct verdict, and the
minimum blocker recall across repeats must not regress. Clean controls retain
their false-positive and verdict safeguards. Implementation candidates retain
the efficiency policy below:
- at least 3 paired VALID runs per task, zero excluded runs in either arm
(session/infra-error rows therefore block promotion), and a named model;
- the candidate must pass the hidden oracle on every valid run for every task;
- a fully measured task that neither arm ever resolves remains reported but is
ungated from the quality comparison — only if its metric was measured in both
arms and no run hit `skill-not-invoked` (a skill that never loaded is prompt
evidence, not task health). An ungated task still ranks against a looser 100%
failed-task regression cap on the promotion metric;
- at least half the paired tasks must stay gated, and `promotion.json` discloses
the gated/ungated split per decision; a set with no gated task at all is
`insufficient_evidence`;
- the candidate must pass the hidden oracle on every valid run of every gated
task the incumbent resolves at least once — on a task the incumbent never
resolves, partial candidate progress counts as improvement instead of failing
the floor, so making some progress is never scored worse than making none;
- no per-task resolution-rate regression (quality is lexicographically first);
- promotion by resolution needs a margin of at least 2 resolved runs —
a 1-run difference is noise at this run count and falls through to the
@ -219,28 +306,85 @@ without weakening today's deterministic promotion boundary.
### Closing the loop automatically (`evolve.py`)
The evolution workflow runs an offline containment preflight with the pinned
Claude Code 2.1.214 binary before starting a paid proposer or benchmark. The
review canary seals the workspace read-only and writes nothing into it: the
artifact directory is bound at `/review-output` outside the workspace, and the
file itself is deliberately absent until the session creates it, so its absence
distinguishes "never written" from "written badly". Runtime mount placeholders
are prepared in the disposable clone before sealing it; existing config bytes
are preserved.
Any pre-existing result entry, including a symlink, is rejected. Required
canaries fail when their runtime or Bubblewrap is unavailable.
The default outage limit is five consecutive unusable results, across task
boundaries. Invalid review JSON advances this limit even when a skill or session
error was recorded first. A valid zero-quality review resets it. Concurrent
waves can exceed the limit by at most `workers - 1` completed cells; no further
wave starts after a trip. Completed rows and redacted diagnostics remain in the
partial report, the runner exits nonzero, and the evolution driver stops without
applying or starting another generation.
SIGINT and SIGTERM propagate one cancellation event through managed commands,
including clone, setup, Claude, and verification. Executor submissions copy
the run context so indirect subprocess helpers receive the same event. Active
process groups or Windows Job Objects are terminated and workers joined before
shared assets or the gateway are released. Controlled cancellation tests require
cleanup within 15 seconds. Cancellation remains distinct from timeout and
quality failure in recorded evidence.
The gateway runs under a private supervisor watching a pipe owned only by the
harness. Parent exit, including SIGKILL, closes that pipe and stops the proxy
group; Windows also retains kill-on-close Job Object ownership. Keep completed
JSONL rows, transcripts, the partial report, and gateway diagnostics when
investigating an interrupted run. A subsequent paid comparison needs fresh
evidence from all arms under the same dependency lock. LiteLLM pricing comes
from that locked release's local cost map; compare no old/new-lock costs as
quality evidence.
`workflow_bench.evolve` automates the three manual arrows — propose,
benchmark, apply — without moving the trust boundary:
```bash
cd eval
uv run --locked --extra dev python -m workflow_bench.evolve \
--tasks workflow_bench/tasks.scenarios.yaml \
--model claude-sonnet-4-20250514 --generations 2 \
--seed-results results/wfbench-<prior-run> # optional gen-0 evidence
./workflow_bench/run-evolution.sh # local; no working-tree apply
./workflow_bench/run-evolution.sh --apply # CI; same argv the workflow uses
./workflow_bench/run-evolution.sh --dry-run # print the evolve command
```
Each generation: a confined **proposer** session reads the incumbent plan/work
skills, the prior generation's `results.jsonl`
loser rows, their session transcripts and patches, and the learning queue,
The GitHub skill-evolution job calls this script. Do not invoke
`python -m workflow_bench.evolve` directly for a full loop. Environment knobs
match the workflow: `MODEL`, `PROPOSER_MODEL`, `GENERATIONS`, `RUNS`,
`WORKERS`, `PROVIDER`, `EFFORT`, `SEED_RESULTS`, `INCLUDE_EXPENSIVE`. The
checked-in production defaults are `PROVIDER=openai`, `MODEL=gpt-5.6-sol`,
`PROPOSER_MODEL=gpt-5.6-sol`, and `EFFORT=xhigh`.
The scheduled/default profile is read-only review evolution. Set
`EVOLUTION_PROFILE=implementation` explicitly to run the legacy plan/work
benchmark. Review mode requires `CE_PLUGIN_DIR` and `CE_PLUGIN_VERSION`.
Each review generation: a confined **proposer** session reads only the incumbent
`gitnexus-review` skill, normalized CE/incumbent/candidate result rows, bounded
review artifacts and session transcripts, and the rejected
`proposal.md` when available (including a workflow seed from a prior run), and
the learning queue,
then writes ONE bounded candidate overlay plus a reviewer-facing
`proposal.md`. The overlay is re-validated by `candidate_overlay_files`
(same boundary: Markdown under the plan/work trees, nothing else), frozen,
`proposal.md`. The proposer's clone is sanitized exactly like an arm's before
its session starts: it authors the artifact the arms are scored with, so
letting it read `eval/workflow_bench` would hand it the task prompts and the
hidden oracles it is about to be graded against, and a proposal could win the
gate by encoding the expected behavior into a skill rather than by being a
better skill. The overlay is re-validated by `candidate_overlay_files`
(same boundary: Markdown under `gitnexus-review`, including exercised
`ci-personas/`, nothing else), frozen,
and exercised only by its exact required pairs. Task refs are resolved once
before generation zero and the immutable task bindings are forwarded to every
generated runner invocation, so a moving branch cannot change later evidence.
The deterministic gate then decides. Promotion application rejects older
pre-oracle evidence schemas. `promote` stops the loop; with `--apply`
The deterministic quality-first gate rejects blocker-recall regressions,
new false positives on clean controls, and any weighted-score regression.
Repeated evidence (`RUNS>=3`) is required for promotion; `RUNS=1` is
diagnostic-only. CE is the external comparator. Cost and latency are
tiebreakers and never compensate for quality loss. `promote` stops the loop; with `--apply`
the authorized frozen bytes
are transactionally applied to the canonical
`.claude/skills/` trees and their shipped mirrors as an ordinary
@ -250,17 +394,19 @@ generation's trajectories to the next proposer. `--initial-overlay` skips
the generation-0 proposer to benchmark a hand-written candidate;
`--proposer-model` upgrades only the diagnosis session.
**Learning queue.** Live plan/work skill runs never self-edit (see each
**Learning queue.** Live skill runs never self-edit (see each
skill's "Skill feedback" section) — instead they may append one-line JSON notes to
`workflow_bench/learnings.jsonl` (gitignored, machine-local like the
transcripts they complement). The proposer reads the queue as hints, not
ground truth: a learning only reaches a shipped skill by surviving the same
paired benchmark as any other candidate. Legacy review/LFG rows are ignored;
those skills do not yet have honest candidate lanes or promotion gates.
paired benchmark as any other candidate.
Run the driver on the existing re-evaluation triggers (model/harness change,
90-day staleness), not on a tight schedule — every generation costs ≥3 paired
runs per task, and `--generations` is the only loop bound.
For ad-hoc use, run the driver on the existing re-evaluation triggers
(model/harness change or 90-day staleness). The repository workflow runs a
deliberate weekly drift check: dispatch defaults to three concurrent cells
of one task; scheduled concurrency still requires
`GITNEXUS_EVOLUTION_WORKERS=3` after a clean proof. `--workers` is bounded
to 1–8 before paid work starts. `--generations` remains the only loop bound.
## Free-model setup (no paid tokens)
@ -279,12 +425,30 @@ uv run --locked --with 'litellm[proxy]' litellm --config workflow_bench/free-mod
# 2. Point the benchmark at it
uv run --locked --extra dev python -m workflow_bench.runner \
--tasks workflow_bench/tasks.scenarios.yaml --runs 3 \
--base-url http://localhost:4000 --auth-token "$LITELLM_MASTER_KEY" --model free-coder
--base-url http://localhost:4000 --anthropic-api-key "$LITELLM_MASTER_KEY" --model free-coder
```
## OpenAI API keys
Claude Code still speaks Anthropic `/v1/messages`. For a paid OpenAI backend,
do not point `--anthropic-api-key` at an `sk-...` OpenAI key. Export the OpenAI key
and use OpenAI model ids; the driver starts the proxy itself:
```bash
export GITNEXUS_BENCH_OPENAI_API_KEY="$OPENAI_API_KEY"
PROVIDER=openai ./workflow_bench/run-evolution.sh
```
The GitHub skill-evolution workflow accepts `GITNEXUS_BENCH_OPENAI_API_KEY` on
the `gitnexus-evolution` environment. Dispatch with `provider=openai` to force
that backend even when an Anthropic token is also configured (otherwise `auto`
keeps using Anthropic whenever that secret exists). Claude default model
inputs are then rewritten to `gpt-5.6-sol`; every proposer and benchmark
session receives `--effort xhigh`.
Caveats, honestly:
- Both arms run on the same model, so the *comparison* stays fair at any
- Both arms run on the same model, so the _comparison_ stays fair at any
quality level — but small free models follow skills less reliably, so
expect lower resolve rates and noisier savings than on frontier models.
Treat free-model runs as directional; confirm headline numbers with a
@ -311,16 +475,16 @@ Three task classes × three arms, single-repo (GitNexus itself). **Every arm
resolved every task** — at this difficulty, pass/fail quality is saturated
and the comparison is pure cost:
| task (class) | arm | resolved | cost $ | wall | turns | vs baseline cost |
| --- | --- | --- | --- | --- | --- | --- |
| trivial-version-alias | workflow | 1/1 | 9.16 | 16m | 63 | −333% |
| trivial-version-alias | baseline | 1/1 | 2.11 | 2.8m | 16 | — |
| inv-bug-pdg-note | workflow | 1/1 | 14.56 | 21m | 83 | −331% |
| inv-bug-pdg-note | workflow_direct | 1/1 | 5.23 | 7.5m | 32 | −55% |
| inv-bug-pdg-note | baseline | 1/1 | 3.38 | 4.7m | 22 | — |
| inv-feature-list-repos-filter | workflow | 1/1 | 13.22 | 19m | 84 | −211% |
| inv-feature-list-repos-filter | workflow_direct | 1/1 | 4.87 | 4.8m | 38 | −15% (wall +14% faster) |
| inv-feature-list-repos-filter | baseline | 1/1 | 4.25 | 5.5m | 32 | — |
| task (class) | arm | resolved | cost $ | wall | turns | vs baseline cost |
| ----------------------------- | --------------- | -------- | ------ | ---- | ----- | ----------------------- |
| trivial-version-alias | workflow | 1/1 | 9.16 | 16m | 63 | −333% |
| trivial-version-alias | baseline | 1/1 | 2.11 | 2.8m | 16 | — |
| inv-bug-pdg-note | workflow | 1/1 | 14.56 | 21m | 83 | −331% |
| inv-bug-pdg-note | workflow_direct | 1/1 | 5.23 | 7.5m | 32 | −55% |
| inv-bug-pdg-note | baseline | 1/1 | 3.38 | 4.7m | 22 | — |
| inv-feature-list-repos-filter | workflow | 1/1 | 13.22 | 19m | 84 | −211% |
| inv-feature-list-repos-filter | workflow_direct | 1/1 | 4.87 | 4.8m | 38 | −15% (wall +14% faster) |
| inv-feature-list-repos-filter | baseline | 1/1 | 4.25 | 5.5m | 32 | — |
What the ground base says, honestly:
@ -335,7 +499,7 @@ What the ground base says, honestly:
detect_changes-before-commit) is cheap. It produced noticeably more test
coverage than baseline for near-equal cost on the feature task.
- **Quality didn't differentiate because nothing failed.** The regime where
the workflow should win on *resolve rate* — cross-module tasks where
the workflow should win on _resolve rate_ — cross-module tasks where
baselines flail — is the unmeasured cell (`cross-module-parse-retry`), and
the next thing to measure, ideally with `--runs 3+` on a free backend.
- Caveats: n=1 per cell, one repo, one model; churn numbers from this run
@ -353,11 +517,11 @@ If a future run shows the workflow flattering itself here, distrust the run.
The hardest class — retry-with-backoff across the worker-pool/pipeline
seams, transient-vs-deterministic classification:
| arm | resolved | cost $ | wall | turns | churn |
| --- | --- | --- | --- | --- | --- |
| workflow | 1/1 | 18.32 | 37m | 107 | 4/+373/−17 |
| **workflow_direct** | 1/1 | **9.53** | **15m** | **52** | 11/+244/−66 |
| baseline | 1/1 | 18.03 | 34m | 98 | 6/+345/−69 |
| arm | resolved | cost $ | wall | turns | churn |
| ------------------- | -------- | -------- | ------- | ------ | ----------- |
| workflow | 1/1 | 18.32 | 37m | 107 | 4/+373/−17 |
| **workflow_direct** | 1/1 | **9.53** | **15m** | **52** | 11/+244/−66 |
| baseline | 1/1 | 18.03 | 34m | 98 | 6/+345/−69 |
(The workflow_direct row is the clean re-run under clone isolation — the
original was contaminated, see the integrity note below.)
@ -391,14 +555,14 @@ category-priced freshness (`accept` for compact classes), per-category turn
budgets, and the work-phase HEAD==pin fast path, the same
`inv-bug-pdg-note` workflow cell re-measured (n=1):
| | ground base | optimized | delta |
| --- | --- | --- | --- |
| resolved | ✅ | ✅ | — |
| cost $ | 14.56 | 11.70 | **−20%** |
| turns | 83 | 72 | −13% |
| output tokens | 59,789 | 53,345 | −11% |
| cache_read | 6.64M | 5.07M | −24% |
| wall | 21m | 25m | +15% |
| | ground base | optimized | delta |
| ------------- | ----------- | --------- | -------- |
| resolved | ✅ | ✅ | — |
| cost $ | 14.56 | 11.70 | **−20%** |
| turns | 83 | 72 | −13% |
| output tokens | 59,789 | 53,345 | −11% |
| cache_read | 6.64M | 5.07M | −24% |
| wall | 21m | 25m | +15% |
Verified in-transcript: the compact form fired (115-line plan vs 209 for a
simpler task pre-optimization), the plan session dropped 72→49 turns, and
@ -411,8 +575,8 @@ this task class, so the routing rule above stands unchanged.
## Writing good tasks
See `tasks.scenarios.yaml`. Small enough to finish headless, real enough to
require investigation — the workflow's savings come from *not re-reading and
not re-investigating*, which trivial tasks never exercise. Keep `verify` as a
require investigation — the workflow's savings come from _not re-reading and
not re-investigating_, which trivial tasks never exercise. Keep `verify` as a
model-visible authored-test quality signal, and add an independent `oracle`
whose source files live under `workflow_bench/oracles/`. Oracle commands must
run only files staged beneath `$GITNEXUS_BENCH_ORACLE_ROOT`; for Vitest, include
@ -422,7 +586,7 @@ carry build pre-hooks).
## Relation to the SWE-bench harness
The rest of `eval/` benchmarks GitNexus *tools* inside a litellm agent loop
(baseline vs graph-enhanced). This module benchmarks the *skill workflow*
The rest of `eval/` benchmarks GitNexus _tools_ inside a litellm agent loop
(baseline vs graph-enhanced). This module benchmarks the _skill workflow_
inside the real CLI harness those skills ship for. Different question, same
spirit: measure, don't assume.

604
eval/workflow_bench/comparator_reuse.py Normal file
View file

@ -0,0 +1,604 @@
"""Reuse frozen comparator cells when the current sweep is still the same experiment.
Weekly skill evolution re-runs incumbent ``review`` / ``ce_review`` (and the
implementation incumbents) even when the model, effort, tasks, oracles,
incumbent skill bytes, and CE plugin have not changed. Those arms are the
baseline the gate compares a *new* candidate against — they are not the
thing being evolved. Replaying them burns two-thirds of a generation.
This module selects prior ``results.jsonl`` rows that are safe to carry
forward. Candidate arms are never reused. A mismatch on any bound field
falls through to a paid cell. Missing artifacts also fall through: a reused
row that the proposer cannot read is worse than spending the tokens again.
"""
from __future__ import annotations
import hashlib
import json
import os
import re
import stat
from collections.abc import Iterator, Mapping, Sequence
from contextlib import contextmanager
from dataclasses import dataclass
from datetime import UTC, datetime, timedelta
from pathlib import Path, PurePosixPath
from typing import Any
from .evolution import CANDIDATE_ARMS, EVIDENCE_MAX_AGE_DAYS
from .proposer_sandbox import SandboxError
from .runner_sessions import MAX_TRANSCRIPT_BYTES, PARENT_EVENT_STREAM_SOURCE
from .runtime_mounts import CE_ARMS
from .task_assets import COPY_CHUNK_BYTES, _write_all
REUSABLE_COMPARATOR_ARMS = frozenset(
{
"review",
"ce_review",
"workflow",
"workflow_direct",
"ce_workflow",
"ce_workflow_direct",
"baseline",
"baseline_nomcp",
}
)
# Must stay aligned with runner.EXCLUDED_ERROR_KINDS plus review-invalid.
# A reused row becomes promotion evidence; excluded kinds cannot enter that set.
REUSE_EXCLUDED_ERROR_KINDS = frozenset(
{
"session-error",
"infra-error",
"evidence-unverified",
"cleanup-failure",
"review-evidence-invalid",
"cancelled",
}
)
_TRANSCRIPT_NAME = re.compile(r"[A-Za-z0-9._-]{1,200}")
CellKey = tuple[str, str, int]
@dataclass(frozen=True)
class TaskReuseBinding:
"""Per-task identity the prior row must still match."""
task_base_sha: str
task_prompt_digest: str
oracle_digest: str
oracle_command_digest: str
oracle_manifest_digest: str
# The cell's environment is part of its identity: a comparator measured
# against different task assets or different sandbox dependencies is a
# measurement of a different machine, not a baseline for this sweep.
task_asset_manifest_digest: str | None = None
sandbox_dependency_manifest_digest: str | None = None
@dataclass(frozen=True)
class ComparatorReuseExpectation:
"""Sweep-wide lock for comparator reuse. Any drift pays for a fresh cell."""
model: str
effort: str
sandbox_backend: str
runtime_digest: str | None
now: datetime
max_age: timedelta
tasks: Mapping[str, TaskReuseBinding]
skill_digests: Mapping[str, str | None]
ce_plugin_version: str | None
ce_plugin_manifest_digest: str | None
def load_result_rows(path: Path) -> list[dict[str, Any]]:
"""Load ``results.jsonl``; skip malformed lines the same way evolve does."""
rows: list[dict[str, Any]] = []
for line in path.read_text().splitlines():
if not line.strip():
continue
try:
row = json.loads(line)
except json.JSONDecodeError:
continue
if isinstance(row, dict):
rows.append(row)
return rows
def current_runtime_digest() -> str | None:
"""Harness lockfile digest exported by ``run-evolution.sh``, if present."""
value = os.environ.get("RUNTIME_DIGEST", "").strip()
return value or None
def row_is_reusable_comparator(row: Mapping[str, Any], expected: ComparatorReuseExpectation) -> bool:
"""True when ``row`` is a complete, still-valid comparator measurement."""
arm = row.get("arm")
if not isinstance(arm, str) or arm in CANDIDATE_ARMS or arm not in REUSABLE_COMPARATOR_ARMS:
return False
if row.get("error_kind") in REUSE_EXCLUDED_ERROR_KINDS:
return False
if row.get("error_kind") not in (None, ""):
return False
if row.get("ok") is not True:
return False
if row.get("transcript_missing") is True:
return False
if row.get("candidate_overlay_digest") not in (None, ""):
return False
# Age against the ORIGINAL measurement, not the copy time: materialize_reused_row
# restamps recorded_at, so a chained row would otherwise refresh its own clock
# and never expire. Bound both directions - a future stamp is corrupt, not fresh.
recorded = _parse_recorded_at(row.get("reused_from_recorded_at") or row.get("recorded_at"))
if recorded is None:
return False
age = expected.now - recorded
if age > expected.max_age or age < timedelta(0):
return False
if row.get("model") != expected.model and row.get("benchmark_model") != expected.model:
return False
if row.get("effort") != expected.effort:
return False
if row.get("sandbox_backend") != expected.sandbox_backend:
return False
# Fail closed. A row with no runtime_digest was measured by a harness that
# did not record one, which is exactly the drift this lock exists to catch;
# treating the absence as agreement made every legacy row reusable forever.
prior_runtime = row.get("runtime_digest")
if not isinstance(prior_runtime, str) or not prior_runtime:
return False
if not expected.runtime_digest or prior_runtime != expected.runtime_digest:
return False
task_id = row.get("task")
binding = expected.tasks.get(task_id) if isinstance(task_id, str) else None
if binding is None:
return False
if row.get("task_base_sha") != binding.task_base_sha:
return False
if row.get("task_prompt_digest") != binding.task_prompt_digest:
return False
if row.get("oracle_digest") != binding.oracle_digest:
return False
if row.get("oracle_command_digest") != binding.oracle_command_digest:
return False
if row.get("oracle_manifest_digest") != binding.oracle_manifest_digest:
return False
# Fail closed on both sides, as the runtime digest does: an unbound
# expectation means this sweep could not determine its own environment, and
# a row without the field was measured before it was recorded.
for field, bound in (
("task_asset_manifest_digest", binding.task_asset_manifest_digest),
("sandbox_dependency_manifest_digest", binding.sandbox_dependency_manifest_digest),
):
prior = row.get(field)
if not isinstance(prior, str) or not prior or not bound or prior != bound:
return False
if arm in CE_ARMS:
if row.get("ce_plugin_version") != expected.ce_plugin_version:
return False
if row.get("ce_plugin_manifest_digest") != expected.ce_plugin_manifest_digest:
return False
else:
expected_skill = expected.skill_digests.get(arm)
if not expected_skill or row.get("skill_digest") != expected_skill:
return False
if arm in {"review", "ce_review"}:
if row.get("review_evidence_valid") is not True:
return False
# The artifact, not just the score derived from it. materialize_reused_row
# copies it only when the name is present, so without this a row whose
# artifact copy never happened could be carried forward as a scored
# review that a proposer then cannot read - evidence by assertion.
review_artifact = row.get("review_artifact")
if not isinstance(review_artifact, str) or not review_artifact:
return False
if not isinstance(row.get("review_score"), dict):
return False
if row.get("review_weighted_f1") is None:
return False
artifacts = row.get("transcript_artifacts")
if not isinstance(artifacts, list) or not artifacts:
return False
try:
for artifact in artifacts:
_transcript_metadata(artifact)
except SandboxError:
return False
return True
def select_reusable_comparator_rows(
rows: Sequence[Mapping[str, Any]],
*,
expected: ComparatorReuseExpectation,
) -> dict[CellKey, dict[str, Any]]:
"""Index reusable rows by ``(task, arm, run)``. Conflicting duplicates drop the key."""
chosen: dict[CellKey, dict[str, Any]] = {}
blocked: set[CellKey] = set()
for row in rows:
if not row_is_reusable_comparator(row, expected):
continue
task_id = row["task"]
arm = row["arm"]
run = row.get("run")
if not isinstance(run, int) or isinstance(run, bool) or run < 0:
continue
key = (str(task_id), str(arm), run)
if key in blocked:
continue
previous = chosen.get(key)
if previous is None:
chosen[key] = dict(row)
continue
if _row_identity(previous) != _row_identity(row):
blocked.add(key)
chosen.pop(key, None)
return chosen
def materialize_reused_row(
row: Mapping[str, Any],
*,
source_dir: Path,
dest_dir: Path,
) -> dict[str, Any]:
"""Copy digest-bound artifacts into this sweep's evidence dir and stamp reuse."""
source, _ = _resolved_directory(source_dir, label="reuse source")
dest, _ = _resolved_directory(dest_dir, label="reuse destination")
if source == dest:
raise SandboxError("comparator reuse cannot read and write the same results directory")
materialized = dict(row)
materialized["reused"] = True
# Keep the FIRST measurement time across a chain. Overwriting it with the
# previous copy's stamp let a row refresh its own clock every generation and
# outlive the max_age bound entirely.
materialized["reused_from_recorded_at"] = row.get("reused_from_recorded_at") or row.get("recorded_at")
materialized["recorded_at"] = datetime.now(UTC).isoformat()
artifacts = row.get("transcript_artifacts")
if not isinstance(artifacts, list) or not artifacts:
raise SandboxError("reused row is missing transcript_artifacts")
# Every path below is resolved against a held descriptor, never re-walked
# from a name. Both roots are already symlink-free (_resolved_directory
# resolved them), and pinning them here means the components under them
# cannot be swapped out from under a check that already passed.
with (
_open_pinned_root(source_dir, label="reuse source") as source_fd,
_open_pinned_root(dest_dir, label="reuse destination") as dest_fd,
):
copied_artifacts: list[dict[str, Any]] = []
for artifact in artifacts:
copied_artifacts.append(_copy_transcript_artifact(source_fd, dest_fd, artifact))
materialized["transcript_artifacts"] = copied_artifacts
review_name = row.get("review_artifact")
if isinstance(review_name, str) and review_name:
_copy_named_artifact(source_fd, dest_fd, review_name, label="review artifact")
task = row.get("task")
arm = row.get("arm")
run = row.get("run")
if isinstance(task, str) and isinstance(arm, str) and isinstance(run, int) and not isinstance(run, bool):
patch_name = f"{task}-{arm}-run{run}.patch"
if _is_regular_at(patch_name, dir_fd=source_fd):
_copy_named_artifact(source_fd, dest_fd, patch_name, label="patch artifact")
return materialized
def default_reuse_max_age() -> timedelta:
return timedelta(days=EVIDENCE_MAX_AGE_DAYS)
def _row_identity(row: Mapping[str, Any]) -> tuple[Any, ...]:
return (
row.get("skill_digest"),
row.get("oracle_digest"),
row.get("review_weighted_f1"),
row.get("ce_plugin_manifest_digest"),
row.get("recorded_at"),
)
def _parse_recorded_at(value: Any) -> datetime | None:
if not isinstance(value, str) or not value:
return None
try:
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError:
return None
if parsed.tzinfo is None:
parsed = parsed.replace(tzinfo=UTC)
return parsed.astimezone(UTC)
def _transcript_metadata(metadata: Any) -> tuple[str, str, int]:
if not isinstance(metadata, dict) or set(metadata) != {"path", "sha256", "bytes", "source"}:
raise SandboxError("transcript artifact metadata must contain only path, sha256, bytes, and source")
relative = metadata["path"]
digest = metadata["sha256"]
size = metadata["bytes"]
if metadata["source"] != PARENT_EVENT_STREAM_SOURCE:
raise SandboxError("transcript artifact source is not the parent event stream")
if not isinstance(relative, str) or not isinstance(digest, str) or not re.fullmatch(r"[0-9a-f]{64}", digest):
raise SandboxError("transcript artifact metadata is malformed")
if not isinstance(size, int) or isinstance(size, bool) or size < 0 or size > MAX_TRANSCRIPT_BYTES:
raise SandboxError("transcript artifact byte count is out of range")
relative_path = PurePosixPath(relative)
if (
relative_path.is_absolute()
or len(relative_path.parts) != 2
or relative_path.parts[0] != "transcripts"
or any(part in {"", ".", ".."} for part in relative_path.parts)
or _TRANSCRIPT_NAME.fullmatch(relative_path.parts[1]) is None
):
raise SandboxError(f"unsafe transcript artifact path: {relative!r}")
return relative, digest, size
def _resolved_directory(path: Path, *, label: str) -> tuple[Path, tuple[int, int]]:
"""An existing, non-symlink directory, resolved through its parents.
Deliberately weaker than proposer_sandbox's same-shaped helper, which
refuses every symlink hop in the path. That one guards a MOUNT ROOT, where
a hop changes what an untrusted session is handed. This one guards a DATA
directory whose contents are validated individually anyway - every file
read goes through ``_regular_file`` (lstat, symlinks rejected) and every
write through ``O_NOFOLLOW`` - so a symlinked parent grants nothing those
guards do not already cover, while refusing one would reject ordinary
setups such as a symlinked artifacts directory or macOS's /var.
Separately named because they make different promises. Do not merge them
without first deciding which promise the reuse path should make.
"""
resolved = path.expanduser()
try:
metadata = resolved.lstat()
except OSError as exc:
raise SandboxError(f"{label} is unavailable: {resolved}: {exc}") from exc
if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISDIR(metadata.st_mode):
raise SandboxError(f"{label} must be a real directory: {resolved}")
return resolved.resolve(), (metadata.st_dev, metadata.st_ino)
@contextmanager
def _open_pinned_root(path: Path, *, label: str) -> Iterator[int]:
"""Open a checked root and prove it is still the directory that was checked.
The symlink POLICY above is deliberate and unchanged: parent hops stay
allowed, so a symlinked artifacts directory or macOS's /var still works.
What is closed here is separate from that policy - the gap between checking
a name and using it. lstat names one directory and resolve() re-walks the
same name afterwards, so a prior sweep that renames its results root and
drops a symlink in its place is resolved to somewhere else entirely, and
O_NOFOLLOW on the open cannot see a link that resolve() already followed.
Comparing the opened descriptor's identity to the checked one costs an
fstat and rejects nothing that holds still: a stable directory always
matches itself. It matters for reuse specifically because the failure is
silent - rows would be copied out of the wrong directory and folded into a
comparator baseline as though they were this sweep's own evidence.
"""
resolved, expected = _resolved_directory(path, label=label)
with _open_real_directory(resolved, label=label) as fd:
opened = os.fstat(fd)
if (opened.st_dev, opened.st_ino) != expected:
raise SandboxError(f"{label} was replaced between the check and the open: {resolved}")
yield fd
def _copy_transcript_artifact(source_fd: int, dest_fd: int, metadata: Mapping[str, Any]) -> dict[str, Any]:
relative, expected_digest, expected_size = _transcript_metadata(metadata)
name = PurePosixPath(relative).name
# Both `transcripts` components are opened as descriptors, not checked as
# names. An lstat that passes and a pathname that is used afterwards are two
# different directories whenever a concurrent writer renames the first one
# away — which the reuse directory, written by a prior sweep, invites.
with (
_open_real_directory("transcripts", dir_fd=dest_fd, label="transcript destination", create=True) as dest_dir_fd,
_open_real_directory("transcripts", dir_fd=source_fd, label="transcript source") as source_dir_fd,
):
os.fchmod(dest_dir_fd, 0o700)
# One descriptor for the whole transfer, and ONE read of it. Hashing the
# source and then reading it again to copy leaves the recorded digest
# describing bytes that are not the bytes written: the descriptor stops
# the pathname being substituted, not the inode being rewritten, and
# this directory belongs to a sweep that may still be writing. Digest
# what is copied, then judge it.
with _open_regular(name, dir_fd=source_dir_fd, label="transcript") as artifact_fd:
digest, copied_bytes = _copy_owner_only(
artifact_fd, name, dir_fd=dest_dir_fd, max_bytes=expected_size
)
if copied_bytes != expected_size or digest != expected_digest:
# The destination now holds bytes no expectation vouches for.
os.unlink(name, dir_fd=dest_dir_fd)
drift = "size" if copied_bytes != expected_size else "digest"
raise SandboxError(f"reused transcript {drift} drifted: {relative}")
return {"path": relative, "sha256": digest, "bytes": expected_size, "source": PARENT_EVENT_STREAM_SOURCE}
def _copy_named_artifact(source_fd: int, dest_fd: int, name: str, *, label: str) -> None:
relative = PurePosixPath(name)
if relative.is_absolute() or len(relative.parts) != 1 or relative.parts[0] in {"", ".", ".."}:
raise SandboxError(f"unsafe {label} path: {name!r}")
with _open_regular(name, dir_fd=source_fd, label=label) as artifact_fd:
# No expectation is recorded for these, so the digest is discarded - but
# "no recorded size" is not "no limit". The source is a prior sweep
# directory that can change between sweeps, so a replaced artifact could
# be arbitrarily large; MAX_TRANSCRIPT_BYTES is the ceiling the capture
# path already enforces on evidence of this kind.
_, copied = _copy_owner_only(artifact_fd, name, dir_fd=dest_fd, max_bytes=MAX_TRANSCRIPT_BYTES)
if copied > MAX_TRANSCRIPT_BYTES:
os.unlink(name, dir_fd=dest_fd)
raise SandboxError(f"reused {label} exceeds {MAX_TRANSCRIPT_BYTES} bytes: {name}")
def _require_openat() -> None:
"""openat is what makes a checked directory and a used directory the same one.
Without it the only alternative is to re-walk the name after the check,
which is exactly the race this module is guarding. Refusing is safe: the
caller in runner treats a SandboxError from reuse as "run a paid cell", so
a platform without openat pays for the cells rather than copying through a
directory nobody verified. The sweep itself is Linux-only anyway (bwrap,
/proc/uptime); this is about the unit tests and about failing loudly.
"""
if os.open not in os.supports_dir_fd or os.lstat not in os.supports_dir_fd:
raise SandboxError("comparator reuse requires POSIX openat support (os.supports_dir_fd)")
def _is_regular_at(name: str, *, dir_fd: int) -> bool:
"""True when `name` under the pinned directory is a regular non-symlink file."""
try:
metadata = os.lstat(name, dir_fd=dir_fd)
except OSError:
return False
return stat.S_ISREG(metadata.st_mode)
@contextmanager
def _open_real_directory(
path: Path | str,
*,
dir_fd: int | None = None,
label: str,
create: bool = False,
) -> Iterator[int]:
"""Open one directory that is not a symlink, and hold it for every use below.
``O_DIRECTORY | O_NOFOLLOW`` makes the check and the open a single syscall,
so unlike an ``lstat`` followed by a path, there is no window in which the
directory can be replaced. ``_resolved_directory`` still tolerates a
symlinked reuse ROOT — it hands this function the already-resolved path —
but every component below it is pinned.
"""
_require_openat()
if create:
try:
os.mkdir(path, 0o700, dir_fd=dir_fd)
except FileExistsError:
# Already there is the ordinary case — a second artifact from the
# same row. What it already IS still has to be proven, and the
# O_DIRECTORY|O_NOFOLLOW open below is what proves it, so there is
# nothing to do here.
pass
except OSError as exc:
raise SandboxError(f"{label} cannot be created: {path}: {exc}") from exc
try:
descriptor = os.open(
path,
os.O_RDONLY | getattr(os, "O_DIRECTORY", 0) | getattr(os, "O_NOFOLLOW", 0),
dir_fd=dir_fd,
)
except FileNotFoundError as exc:
# Absent is a different fact from present-but-not-a-real-directory, and
# the caller falls through to a paid cell on either.
raise SandboxError(f"{label} is missing: {path}") from exc
except OSError as exc:
raise SandboxError(f"{label} must be a real directory: {path}: {exc}") from exc
try:
# O_DIRECTORY is the check on Linux; the fstat covers a platform whose
# os module does not define it, where the flag degrades to 0.
if not stat.S_ISDIR(os.fstat(descriptor).st_mode):
raise SandboxError(f"{label} must be a real directory: {path}")
yield descriptor
finally:
os.close(descriptor)
@contextmanager
def _open_regular(name: str, *, dir_fd: int, label: str) -> Iterator[int]:
"""Open a regular non-symlink file under a pinned directory, and hold it.
Checking a name and then re-opening it is a race the reuse directory is
exposed to: it is written by a previous sweep and read by this one, so a
concurrent writer can replace a validated file with a symlink in between.
Resolving against ``dir_fd`` removes the directory half, ``O_NOFOLLOW``
refuses the leaf link, and the fstat comparison proves the open descriptor
is the inode that was checked — the same guarantee
evolution._bounded_regular_bytes makes for evidence files.
"""
_require_openat()
try:
before = os.lstat(name, dir_fd=dir_fd)
except OSError as exc:
raise SandboxError(f"{label} is missing: {name}: {exc}") from exc
if stat.S_ISLNK(before.st_mode) or not stat.S_ISREG(before.st_mode):
raise SandboxError(f"{label} must be a regular non-symlink file: {name}")
try:
descriptor = os.open(name, os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0), dir_fd=dir_fd)
except OSError as exc:
raise SandboxError(f"{label} is unreadable: {name}: {exc}") from exc
try:
opened = os.fstat(descriptor)
if not stat.S_ISREG(opened.st_mode) or (opened.st_dev, opened.st_ino) != (before.st_dev, before.st_ino):
raise SandboxError(f"{label} changed while opening: {name}")
yield descriptor
finally:
os.close(descriptor)
def _copy_owner_only(source: int, name: str, *, dir_fd: int, max_bytes: int | None = None) -> tuple[str, int]:
"""Copy one open file into the pinned directory; return what was written.
The digest is taken from the same buffers that are written, so it describes
the copy rather than a state the source was in at some earlier read.
``max_bytes`` bounds the copy itself. The source is a prior sweep directory
this module already treats as concurrently writable, so a transcript
appended to after its metadata was recorded would otherwise be streamed to
EOF and only then compared against its declared size - filling the
destination, or never reaching EOF at all, long before the drift check could
reject it. Stopping one byte past the ceiling keeps that comparison
meaningful while bounding the work.
"""
# O_CREAT|O_EXCL is the existence check, and unlike a stat beforehand it is
# atomic: a file appearing between check and open cannot slip through.
try:
descriptor = os.open(
name,
os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, "O_NOFOLLOW", 0),
0o600,
dir_fd=dir_fd,
)
except FileExistsError as exc:
raise SandboxError(f"reuse destination already exists: {name}") from exc
try:
os.fchmod(descriptor, 0o600)
os.lseek(source, 0, os.SEEK_SET)
digest = hashlib.sha256()
written = 0
limit = None if max_bytes is None else max_bytes + 1
while True:
want = COPY_CHUNK_BYTES if limit is None else min(COPY_CHUNK_BYTES, limit - written)
if want <= 0:
break
chunk = os.read(source, want)
if not chunk:
break
digest.update(chunk)
written += len(chunk)
_write_all(descriptor, chunk)
os.fsync(descriptor)
return digest.hexdigest(), written
finally:
os.close(descriptor)

505
eval/workflow_bench/evolution.py
View file

@ -7,6 +7,7 @@ import os
import secrets
import stat
import statistics
from collections.abc import Sequence
from pathlib import Path, PurePosixPath
from typing import Any
@ -21,9 +22,94 @@ from .proposer_sandbox import (
CANDIDATE_ARMS = {
"candidate_workflow": "workflow",
"candidate_workflow_direct": "workflow_direct",
"candidate_review": "review",
}
PROMOTION_SCHEMA_VERSION = 6
def promotion_policy(
candidate_arms: Sequence[str],
*,
metric: str = "cost_usd",
min_runs: int = 3,
min_improvement_pct: float = 5.0,
max_task_regression_pct: float = 20.0,
) -> dict[str, dict[str, Any]]:
"""The exact per-arm policy shared by evidence production and application."""
if not candidate_arms or len(set(candidate_arms)) != len(candidate_arms):
raise ValueError("promotion policy requires unique candidate arms")
policies = {}
for arm in candidate_arms:
if arm not in CANDIDATE_ARMS:
raise ValueError(f"unsupported candidate arm: {arm}")
policies[arm] = (
{
"metric": "review_weighted_f1",
"min_runs": min_runs,
"min_improvement": 0.01,
"quality_rule": "correct verdict on every repeat; minimum blocker recall; no clean-control regression",
}
if arm == "candidate_review"
else {
"metric": metric,
"min_runs": min_runs,
"min_improvement_pct": min_improvement_pct,
"max_task_regression_pct": max_task_regression_pct,
"max_failed_task_regression_pct": MAX_FAILED_TASK_REGRESSION_PCT,
"min_gated_task_ratio": MIN_GATED_TASK_RATIO,
"quality_rule": "no per-task resolution-rate regression",
}
)
return policies
def promotion_evidence(
results: dict[str, dict[str, dict[str, Any]]],
*,
policy: dict[str, dict[str, Any]],
model: str | None,
complete: bool,
) -> dict[str, Any]:
"""Produce discriminated, recomputable decisions, including partial reports."""
decisions = []
for candidate, rules in policy.items():
common = {
"incumbent_arm": CANDIDATE_ARMS[candidate],
"candidate_arm": candidate,
"model": model,
"min_runs": rules["min_runs"],
}
if candidate == "candidate_review":
decision = evaluate_review_candidate(results, **common, min_improvement=rules["min_improvement"])
else:
decision = evaluate_candidate(
results,
**common,
**{
key: rules[key]
for key in (
"metric",
"min_improvement_pct",
"max_task_regression_pct",
"max_failed_task_regression_pct",
)
},
)
if not complete:
decision["decision"] = "insufficient_evidence"
decision["reasons"].append("sweep aborted; partial evidence cannot promote")
decisions.append(decision)
return {
"schema_version": PROMOTION_SCHEMA_VERSION,
"run_status": "complete" if complete else "aborted",
"policy": policy,
"decisions": decisions,
}
CANDIDATE_SKILLS = {
"gitnexus-plan",
"gitnexus-review",
"gitnexus-work",
}
# Skills each incumbent arm actually loads in its sessions. An overlay that
@ -32,6 +118,7 @@ CANDIDATE_SKILLS = {
ARM_SKILLS = {
"workflow": ("gitnexus-plan", "gitnexus-work"),
"workflow_direct": ("gitnexus-work",),
"review": ("gitnexus-review",),
}
# Repo-local prompts whose bytes are evidence for each executed arm. Keep this
# distinct from ``ARM_SKILLS``: that mapping defines which skills a promotable
@ -54,6 +141,19 @@ MAIN_LOOP_ONLY_WARNING = (
"each run output, deduplicating events "
"that share one message.id."
)
# Failure kinds the prompts under test cause, not the task: the skill never
# ran at all. Both arms failing a task this way is evidence about the skills,
# so such a task stays inside the gate however unresolvable it looks.
SKILL_ATTRIBUTABLE_ERROR_KINDS = frozenset({"skill-not-invoked"})
# Leaving the quality gate is not leaving the spend gate. A candidate may fail
# the same oracle the incumbent fails, but not at a multiple of its cost — an
# ungated task is still real money and still ranks on the metric.
MAX_FAILED_TASK_REGRESSION_PCT = 100.0
# Promotion must rest on a real evidence base. Half the paired tasks is the
# loosest rule the three-task production set can carry: it tolerates the one
# scenario neither arm resolves and refuses a generation that has quietly
# decayed to a single gated task deciding everything.
MIN_GATED_TASK_RATIO = 0.5
EVIDENCE_MAX_AGE_DAYS = 90
MAX_CANDIDATE_OVERLAY_BYTES = 4 * 1024 * 1024
MAX_SKILL_FINGERPRINT_BYTES = 4 * 1024 * 1024
@ -325,7 +425,7 @@ def candidate_overlay_files(overlay: Path) -> list[Path]:
):
raise ValueError(
"candidate overlays may only contain Markdown files under "
".claude/skills/gitnexus-{plan,work}: "
".claude/skills/gitnexus-{plan,review,work}: "
f"{relative}"
)
return entries
@ -345,6 +445,8 @@ def required_candidate_arms(overlay: Path) -> list[str]:
required.append("candidate_workflow")
if "gitnexus-work" in touched:
required.append("candidate_workflow_direct")
if "gitnexus-review" in touched:
required.append("candidate_review")
return required
@ -365,6 +467,69 @@ def candidate_overlay_digest(overlay: Path) -> str:
return digest
def _commit_sandbox_paths(
sandbox: SandboxSession,
relative_paths: Sequence[str],
*,
message: str,
require_change: bool,
) -> bool:
"""Stage and commit paths inside the outer sandbox.
Returns True when a commit was created. ``require_change`` keeps the
candidate-overlay contract: a no-op overlay is an error, while an
incumbent skill seed may already match the historical tree.
"""
mkdir_command = ["/bin/mkdir", "-p", f"{SANDBOX_TMP}/wfbench-empty-hooks"]
mkdir_result = sandbox.run(
mkdir_command,
timeout=60,
env=build_sandbox_environment(),
)
if not mkdir_result.ok:
raise ManagedProcessError(mkdir_command, mkdir_result)
if not relative_paths:
if require_change:
raise ValueError("candidate overlay is byte-identical to the incumbent skills")
return False
# Historical review SHAs gitignore `.claude/skills/*` and lack the current
# per-skill allowlist. Force-add so a seed or overlay of harness-owned
# skill bytes is not rejected as an ignored path.
command, added = _sandbox_overlay_git(sandbox, ["add", "-f", "--", *relative_paths])
if not added.ok:
raise ManagedProcessError(command, added)
command, changed = _sandbox_overlay_git(
sandbox,
["diff", "--cached", "--quiet", "--no-ext-diff", "--no-textconv", "--"],
)
if changed.returncode == 0:
if require_change:
raise ValueError("candidate overlay is byte-identical to the incumbent skills")
return False
if changed.returncode != 1:
raise ManagedProcessError(command, changed)
command, committed = _sandbox_overlay_git(
sandbox,
[
"commit",
"--quiet",
"--no-verify",
"-m",
message,
],
extra_config=(
"user.name=workflow-bench",
"user.email=workflow-bench@invalid",
),
)
if not committed.ok:
raise ManagedProcessError(command, committed)
return True
def apply_candidate_overlay(
overlay: Path,
worktree: Path,
@ -383,47 +548,96 @@ def apply_candidate_overlay(
for relative, content in payload:
_replace_regular_file(worktree, relative, content)
relative_paths.append(relative.as_posix())
mkdir_command = ["/bin/mkdir", "-p", f"{SANDBOX_TMP}/wfbench-empty-hooks"]
mkdir_result = sandbox.run(
mkdir_command,
timeout=60,
env=build_sandbox_environment(),
)
if not mkdir_result.ok:
raise ManagedProcessError(mkdir_command, mkdir_result)
command, added = _sandbox_overlay_git(sandbox, ["add", "--", *relative_paths])
if not added.ok:
raise ManagedProcessError(command, added)
command, changed = _sandbox_overlay_git(
_commit_sandbox_paths(
sandbox,
["diff", "--cached", "--quiet", "--no-ext-diff", "--no-textconv", "--"],
relative_paths,
message="benchmark candidate skill overlay",
require_change=True,
)
if changed.returncode == 0:
raise ValueError("candidate overlay is byte-identical to the incumbent skills")
if changed.returncode != 1:
raise ManagedProcessError(command, changed)
command, committed = _sandbox_overlay_git(
sandbox,
[
"commit",
"--quiet",
"--no-verify",
"-m",
"benchmark candidate skill overlay",
],
extra_config=(
"user.name=workflow-bench",
"user.email=workflow-bench@invalid",
),
)
if not committed.ok:
raise ManagedProcessError(command, committed)
return digest
def seed_evaluated_skills(
source_repo: Path,
worktree: Path,
*,
sandbox: SandboxSession,
arm: str,
) -> None:
"""Install the current evaluated skill tree into a historical clone.
Review evolution scores the current (or overlay) ``gitnexus-review`` skill
against a historical PR checkout. Older SHAs predate that skill, and
using whatever prose happened to exist at the reviewed commit would make
the incumbent arm a moving target. Copy the harness checkout's skill
bytes and commit them before setup so ``git status`` still shows only
the task patch.
"""
skill_names = EVALUATED_ARM_SKILLS.get(arm)
if not skill_names:
return
source_repo = source_repo.expanduser().absolute()
expected_clone = Path(os.path.abspath(worktree.expanduser()))
sandbox_clone = Path(os.path.abspath(sandbox.clone.expanduser()))
if sandbox_clone != expected_clone:
raise ValueError("skill seed sandbox does not bind the requested clone")
_require_real_directory(source_repo, label="incumbent skill repository")
if source_repo.resolve(strict=True) != source_repo:
raise ValueError(f"incumbent skill repository cannot traverse symlinks: {source_repo}")
relative_paths: list[str] = []
total = 0
for skill_name in skill_names:
_require_directory_chain(
source_repo,
Path(".claude") / "skills" / skill_name,
label="incumbent skill root",
)
skill_root = source_repo / ".claude" / "skills" / skill_name
pending = [skill_root]
while pending:
directory = pending.pop()
try:
children = list(os.scandir(directory))
except OSError as exc:
raise ValueError(f"incumbent skill directory is unreadable: {directory}: {exc}") from exc
for item in children:
path = Path(item.path)
if item.is_symlink():
raise ValueError(
"incumbent skill seed cannot contain symlinks: "
f"{path.relative_to(source_repo)}"
)
if item.is_dir(follow_symlinks=False):
pending.append(path)
continue
if not item.is_file(follow_symlinks=False):
raise ValueError(
"incumbent skill seed entries must be regular files: "
f"{path.relative_to(source_repo)}"
)
total += item.stat(follow_symlinks=False).st_size
if total > MAX_SKILL_FINGERPRINT_BYTES:
raise ValueError("incumbent skill seed exceeds the bounded evidence limit")
relative = Path(".claude") / "skills" / skill_name / path.relative_to(skill_root)
content = _bounded_regular_bytes(
path,
limit=MAX_SKILL_FINGERPRINT_BYTES,
label="incumbent skill file",
)
_replace_regular_file(worktree, relative, content)
relative_paths.append(PurePosixPath(relative.as_posix()).as_posix())
_commit_sandbox_paths(
sandbox,
relative_paths,
message="benchmark incumbent review skill",
require_change=False,
)
def unexercised_overlay_skills(overlay: Path, candidate_arms: list[str]) -> list[str]:
"""Overlay skills that no selected candidate arm would ever load.
@ -475,6 +689,133 @@ def skill_fingerprint(worktree: Path, arm: str) -> str | None:
return fingerprint_files(worktree, files)
def evaluate_review_candidate(
results: dict[str, dict[str, dict[str, Any]]],
*,
incumbent_arm: str,
candidate_arm: str,
model: str | None,
min_runs: int = 3,
min_improvement: float = 0.01,
) -> dict[str, Any]:
"""Quality-first promotion gate for paired read-only review arms."""
reasons: list[str] = []
task_rows: list[dict[str, Any]] = []
insufficient = not model
regression = False
improvement = False
if not model:
reasons.append("a named --model is required so review evidence cannot drift")
for task_id, arms in sorted(results.items()):
if incumbent_arm not in arms or candidate_arm not in arms:
insufficient = True
reasons.append(f"{task_id}: both {incumbent_arm} and {candidate_arm} are required")
continue
incumbent = arms[incumbent_arm]
candidate = arms[candidate_arm]
incumbent_runs = int(incumbent.get("valid_runs", 0))
candidate_runs = int(candidate.get("valid_runs", 0))
incumbent_score = incumbent.get("review_weighted_f1")
candidate_score = candidate.get("review_weighted_f1")
incumbent_blockers = incumbent.get("review_blocker_recall")
candidate_blockers = candidate.get("review_blocker_recall")
incumbent_fp = incumbent.get("review_false_positives")
candidate_fp = candidate.get("review_false_positives")
clean = bool(incumbent.get("review_clean_control", candidate.get("review_clean_control", False)))
incumbent_clean_pass = incumbent.get("review_clean_pass")
candidate_clean_pass = candidate.get("review_clean_pass")
task_rows.append(
{
"task": task_id,
"class": incumbent.get("class", ""),
"incumbent_weighted_f1": incumbent_score,
"incumbent": dict(incumbent),
"candidate": dict(candidate),
"gated": True,
"candidate_weighted_f1": candidate_score,
"incumbent_blocker_recall": incumbent_blockers,
"candidate_blocker_recall": candidate_blockers,
"incumbent_false_positives": incumbent_fp,
"candidate_false_positives": candidate_fp,
"clean_control": clean,
"incumbent_clean_pass": incumbent_clean_pass,
"candidate_clean_pass": candidate_clean_pass,
}
)
if (
incumbent_runs < min_runs
or candidate_runs < min_runs
or incumbent_runs != candidate_runs
or incumbent.get("excluded_runs")
or candidate.get("excluded_runs")
):
insufficient = True
reasons.append(
f"{task_id}: needs {min_runs} valid paired runs with zero exclusions "
f"(got {incumbent_runs}/{candidate_runs})"
)
required_values = (
(incumbent_fp, candidate_fp, incumbent_clean_pass, candidate_clean_pass)
if clean
else (incumbent_score, candidate_score, incumbent_fp, candidate_fp)
)
if any(value is None for value in required_values):
insufficient = True
reasons.append(f"{task_id}: structured review quality metrics are incomplete")
continue
if candidate.get("review_verdict_correct") is None:
insufficient = True
reasons.append(f"{task_id}: candidate verdict evidence is incomplete")
elif candidate["review_verdict_correct"] is not True:
regression = True
reasons.append(f"{task_id}: candidate verdict was incorrect on a valid repeat")
if (incumbent_blockers is None) != (candidate_blockers is None):
insufficient = True
reasons.append(f"{task_id}: blocker recall evidence is incomplete")
if incumbent_blockers is not None and candidate_blockers is not None and float(candidate_blockers) < float(
incumbent_blockers
):
regression = True
reasons.append(f"{task_id}: blocker recall regressed")
if clean and float(candidate_fp) > float(incumbent_fp):
regression = True
reasons.append(f"{task_id}: false positives increased on a clean control")
if clean and bool(incumbent_clean_pass) and not bool(candidate_clean_pass):
regression = True
reasons.append(f"{task_id}: clean-control verdict regressed")
if not clean and float(candidate_score) + 1e-9 < float(incumbent_score):
regression = True
reasons.append(f"{task_id}: weighted review score regressed")
if not clean and float(candidate_score) >= float(incumbent_score) + min_improvement:
improvement = True
if not task_rows:
insufficient = True
reasons.append("no paired review task results were found")
if insufficient:
decision = "insufficient_evidence"
elif regression:
decision = "keep_incumbent"
elif not improvement:
decision = "keep_incumbent"
reasons.append("candidate did not improve weighted review quality on any corpus case")
else:
decision = "promote"
reasons.append("candidate improved weighted review quality without blocker or clean-control regression")
return {
"candidate_arm": candidate_arm,
"incumbent_arm": incumbent_arm,
"decision": decision,
"metric": "review_weighted_f1",
"model": model,
"tasks": task_rows,
"ungated_tasks": [],
"reasons": reasons,
}
def evaluate_candidate(
results: dict[str, dict[str, dict[str, Any]]],
*,
@ -485,12 +826,18 @@ def evaluate_candidate(
min_runs: int = 3,
min_improvement_pct: float = 5.0,
max_task_regression_pct: float = 20.0,
max_failed_task_regression_pct: float = MAX_FAILED_TASK_REGRESSION_PCT,
) -> dict[str, Any]:
"""Deterministically decide whether a prompt candidate is promotable.
Resolution is lexicographically primary: a cheaper candidate that fails
more tasks never wins. With equal quality, the candidate must clear the
configured median efficiency gain without a large per-task regression.
A task neither arm can resolve leaves the quality gate, but only on
evidence: a comparable metric, no skill-not-invoked run, and enough tasks
left inside the gate to decide anything. It still ranks against the
failed-task spend cap.
"""
if metric not in PROMOTION_METRICS:
raise ValueError(f"unsupported promotion metric: {metric}")
@ -536,20 +883,66 @@ def evaluate_candidate(
if (not metric_unavailable and incumbent_metric)
else None
)
# A task that both arms measured cleanly and neither ever resolved sits
# outside both arms' current capability. It carries no quality signal
# about the candidate, and its metric compares who spent more while
# failing the same oracle — so gating on it measures the task, not the
# candidate, and one such task vetoes every future promotion for as
# long as it stays in the set. Keep it in the evidence, out of the gate,
# and name it as task health instead.
#
# Ungating is itself a claim, so it needs evidence: the failures must
# be the task's (not a skill that never loaded) and the metric must be
# comparable, otherwise the task stays gated and the checks below name
# what is missing.
fully_measured = (
incumbent_runs >= min_runs
and candidate_runs >= min_runs
and incumbent_runs == candidate_runs
and not incumbent_excluded
and not candidate_excluded
)
skill_attributable = bool(
(set(incumbent.get("error_kinds", {})) | set(candidate.get("error_kinds", {})))
& SKILL_ATTRIBUTABLE_ERROR_KINDS
)
mutually_unresolved = fully_measured and not incumbent["resolved"] and not candidate["resolved"]
gated = not (mutually_unresolved and not skill_attributable and improvement is not None)
# The floor asks the candidate to be reliable where the incumbent is.
# On a task the incumbent never resolves there is no reliability to
# match, and holding partial candidate progress to it punished a
# candidate for resolving 1 of 3 runs while excusing it for resolving
# none — the strictly worse result. A skill that never loaded is the
# exception: those failures belong to the prompts, so the floor applies
# even with nothing on the incumbent's side to match.
quality_floor_enforced = bool(incumbent["resolved"]) or skill_attributable
task_rows.append(
{
"task": task_id,
"class": incumbent.get("class", ""),
"incumbent_resolved": f"{incumbent['resolved']}/{incumbent_runs}",
"incumbent": dict(incumbent),
"candidate": dict(candidate),
"candidate_resolved": f"{candidate['resolved']}/{candidate_runs}",
"incumbent_excluded_runs": incumbent_excluded,
"candidate_excluded_runs": candidate_excluded,
"candidate_quality_floor_met": candidate_runs > 0 and candidate["resolved"] == candidate_runs,
"quality_floor_enforced": quality_floor_enforced,
"incumbent_metric": incumbent_metric,
"candidate_metric": candidate_metric,
"improvement_pct": improvement,
"gated": gated,
"skill_attributable_failure": skill_attributable,
}
)
if not gated:
if improvement < -max_failed_task_regression_pct:
efficiency_regression = True
reasons.append(
f"{task_id}: {metric} regressed {-improvement:.1f}% on a task neither arm resolved, "
f"above the {max_failed_task_regression_pct:.1f}% failed-task cap"
)
continue
if incumbent_runs < min_runs or candidate_runs < min_runs:
insufficient = True
@ -571,11 +964,16 @@ def evaluate_candidate(
if candidate_rate < incumbent_rate:
quality_regression = True
reasons.append(f"{task_id}: resolution regressed from {incumbent_rate:.0%} to {candidate_rate:.0%}")
if candidate_runs > 0 and candidate["resolved"] != candidate_runs:
if quality_floor_enforced and candidate_runs > 0 and candidate["resolved"] != candidate_runs:
quality_floor_failed = True
floor_trigger = (
"a run never invoked the skill under test"
if skill_attributable
else f"the incumbent resolves {incumbent['resolved']}/{incumbent_runs}"
)
reasons.append(
f"{task_id}: candidate must resolve every valid run for the oracle-backed quality floor "
f"(got {candidate['resolved']}/{candidate_runs})"
f"({floor_trigger}; got {candidate['resolved']}/{candidate_runs})"
)
if metric_unavailable:
insufficient = True
@ -592,11 +990,36 @@ def evaluate_candidate(
f"{task_id}: {metric} regressed {-improvement:.1f}%, above the {max_task_regression_pct:.1f}% task cap"
)
ungated_tasks = [row["task"] for row in task_rows if not row["gated"]]
gated_tasks = [row["task"] for row in task_rows if row["gated"]]
if ungated_tasks:
# One line, not one per task: `reasons` is truncated to three entries
# when it is fed back to the proposer (evolve.summarize_gate), and a
# growing set of unsolvable tasks must not crowd out the reason the
# candidate actually won or lost. The full list ships structurally.
reasons.append(
f"not gated on {len(ungated_tasks)} task(s) neither arm resolved: {', '.join(ungated_tasks)} "
f"(evidence base: {len(gated_tasks)}/{len(task_rows)} paired tasks gated)"
)
if not task_rows:
insufficient = True
reasons.append("no paired task results were found")
elif not gated_tasks:
# Every paired task was ungated, so nothing in this generation says
# anything about candidate quality. Refuse rather than fall through to
# an efficiency-only verdict on runs that all failed their oracle.
insufficient = True
reasons.append("no task supplied quality signal: neither arm resolved a run anywhere in the set")
elif len(gated_tasks) < MIN_GATED_TASK_RATIO * len(task_rows):
# Ungating one unsolvable task keeps promotion reachable; ungating most
# of the set turns "promote" into a verdict from whatever is left.
insufficient = True
reasons.append(
f"promotion evidence base is too thin: {len(gated_tasks)}/{len(task_rows)} paired tasks are gated "
f"(at least {MIN_GATED_TASK_RATIO:.0%} required)"
)
improvements = [row["improvement_pct"] for row in task_rows if row["improvement_pct"] is not None]
improvements = [row["improvement_pct"] for row in task_rows if row["gated"] and row["improvement_pct"] is not None]
median_improvement = round(statistics.median(improvements), 1) if improvements else None
incumbent_resolved = sum(
arms[incumbent_arm]["resolved"] for arms in results.values() if incumbent_arm in arms and candidate_arm in arms
@ -640,6 +1063,8 @@ def evaluate_candidate(
"metric": metric,
"metric_warning": (MAIN_LOOP_ONLY_WARNING if metric in MAIN_LOOP_ONLY_METRICS else None),
"median_improvement_pct": median_improvement,
"ungated_tasks": ungated_tasks,
"gated_tasks": gated_tasks,
"reasons": reasons,
"tasks": task_rows,
}

1003
eval/workflow_bench/evolve.py
View file

File diff suppressed because it is too large Load diff

4
eval/workflow_bench/free-model.litellm.yaml
View file

@ -9,7 +9,7 @@
#
# uv run python -m workflow_bench.runner \
# --tasks workflow_bench/tasks.scenarios.yaml \
# --base-url http://localhost:4000 --auth-token "$LITELLM_MASTER_KEY" \
# --base-url http://localhost:4000 --anthropic-api-key "$LITELLM_MASTER_KEY" \
# --model free-coder
#
# Keep the proxy on loopback (litellm's default host). Anyone who can reach
@ -39,5 +39,5 @@ model_list:
general_settings:
# No static default — export LITELLM_MASTER_KEY before starting the proxy
# and pass the same value as --auth-token (see header).
# and pass the same value as --anthropic-api-key (see header).
master_key: os.environ/LITELLM_MASTER_KEY

45
eval/workflow_bench/gateway_supervisor.py Normal file
View file

@ -0,0 +1,45 @@
"""Private gateway owner. EOF on stdin means the harness no longer exists.
Executed by absolute script path so the isolated gateway environment needs no
PYTHONPATH. The gateway receives DEVNULL, never the owner-liveness descriptor.
"""
from __future__ import annotations
import os
import signal
import sys
import threading
from process_control import run_managed
def main() -> int:
cancelled = threading.Event()
def watch_owner() -> None:
try:
# A buffered stdin lock held by a daemon aborts CPython shutdown
# when the proxy exits while the owner is still alive.
os.read(sys.stdin.fileno(), 1)
finally:
cancelled.set()
threading.Thread(target=watch_owner, daemon=True).start()
for signum in (signal.SIGTERM, signal.SIGINT):
signal.signal(signum, lambda *_: cancelled.set())
result = run_managed(
sys.argv[1:],
timeout=7 * 24 * 60 * 60,
cancel_event=cancelled,
echo_stdout=True,
)
if result.stderr_tail:
print(result.stderr_tail, file=sys.stderr, flush=True)
if result.detail:
print(result.detail, file=sys.stderr, flush=True)
return 0 if result.ok or result.state == "cancelled" else 1
if __name__ == "__main__":
raise SystemExit(main())

Some files were not shown because too many files have changed in this diff Show more