mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-08-28 05:25:25 +00:00
0ce7880290
* test(scope-resolution): audit the consumers of file-scoped node ids (#2699 part A)
#2699 item 4 — "audit consumers that assume file-scoped ids" — after #2695/#2714
gave function-local CALLABLES position-bearing ids. Tests and findings only; no
production change. That split is deliberate: `impact` reports
`resolveDefGraphId` at CRITICAL with 23 DIRECT dependents across 7 modules
(every language MRO builder, both Spring attachers, C++ member lookup,
tryEmitEdge, emitReferencesViaLookup, buildGraphTargetIndex, emitFreeCallFallback,
emitReceiverBoundCalls, preEmitInheritanceEdges, emitDetectedInterfaceImplementations,
phpEmitUnresolvedReceiverEdges, emitRubyMixinEdges, emitRustTraitImplEdges,
emitDartHeritageEdges), so changing that key chain is its own change, not a
rider on an audit.
A2 — detect_changes: CONCERN RESOLVED, now pinned. The worry was that an id
containing `@row:col` re-keys whenever a declaration MOVES, making every edit
look like symbol churn. It cannot: `local-backend.ts` maps diff hunks to
symbols by LINE-RANGE OVERLAP (`n.startLine`/`n.endLine`) and merely REPORTS
`n.id`. Node identity never participates in the match. New structural test
asserts the WHERE clause never gains `n.id =` or `n.id IN`, keeps the one
legitimate id-shaped predicate (the `BasicBlock:` prefix exclusion, #2082 U7),
and confirms the id is returned rather than matched. Structural in the same
idiom as `detect-changes-worktree.test.ts`, and labelled as not proving runtime
behaviour.
A1 — ANSWERED, and the answer is that #2699 is NOT fully closed by items 1-3.
The fail-closed guard is gated on `isOverloadableCallable`
(Function | Method | Constructor), so a function-local VALUE never reaches it.
Measured on a fixture: a top-level `const handler` and a function-local
`const handler` still produce ONE node, `Const:v.ts:handler`. That is the
residual half of the issue's original complaint. Pinned as a KNOWN LIMIT with
its reason (widening identity to values re-keys ~14,700 build-time nodes to
change ~800 persisted ones — the decision recorded in `parse-worker.ts`), and
deliberately NOT fixed here.
A3 — id-persisting consumers, classified:
- detect_changes ................ SAFE (position-keyed; pinned by A2)
- MCP impact/context/trace ...... SAFE (resolve by name/uid at query time)
- bench fingerprints ............ SAFE (digest capture shape, not node ids)
- rust-captures golden .......... SAFE (digests captures, not ids)
- cfg pipeline-pdg snapshot ..... AT RISK by design — pins exact edge ids, so
it trips whenever attribution changes. That is the gate working; #2714
already exercised it.
- wiki / group-contract links ... NOT id-keyed on locals (locals are never
cross-file addressable, per the document-scoped contract of item 2).
Verified: tsc clean; 14/14 across the two touched files; `detect_changes`
reports 0 changed symbols (tests only).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0184RmD24KFJidYqpM7v3XjR
* fix(php): a closure binding is a call SOURCE, not only a TARGET (#2699 part B, S1)
A call made inside a closure binding was attributed to the ENCLOSING scope, so
the closure was a call TARGET but never a call SOURCE: impact(handler,
direction:"downstream") reported nothing even though the closure calls out.
Root cause, probe-measured rather than inferred. Instrumenting
pickCallerCallableDef (graph-bridge/ids.ts) to log every rejection reason shows
the closure's own scope EXISTS and its range DOES contain the call site, but its
ownedDefs is EMPTY, so the ":94" owned-callable filter drops it and attribution
falls through to the ":97" enclosing-scope fallback.
The reason is one missing query rule. javascript/query.ts pairs the binding name
with the closure via @declaration.function anchored on the INNER arrow node, so
anchor.range equals the @scope.function range and pass2AttachDeclarations
attaches the declaration to the CLOSURE's scope. No other language had that
rule — PHP, Rust, Kotlin, Ruby and Dart all captured named function
declarations only. That single omission is the entire empty-ownedDefs cause.
This ports the rule to PHP with the same anchor discipline (@declaration.function
on the inner anonymous_function / arrow_function, NOT on the
assignment_expression wrapper). PHP needs nothing else: it already declares
(anonymous_function) and (arrow_function) as @scope.function, so the rule alone
completes it.
Measured on a fixture: `$handler = function ($x) { return target($x); }` inside
outer() now emits
Function:src/a.php:outer.$handler@3:2 -> Function:src/a.php:target
where it previously emitted `outer -> target`.
The pinned test in closure-binding-labels.test.ts asserted the OLD, wrong
behaviour by design ("to catch that asymmetry changing in EITHER direction"), so
it is INVERTED here rather than deleted, per its own instruction. Its block
comment is corrected to record the measured root cause, including that Kotlin
and Ruby will need BOTH this rule AND a relaxed kind gate (their lambda_literal
/ do_block is @scope.block deliberately, #1757), and that Dart has no closure
scope at all.
Verification: closure-binding-labels 50/50; PHP resolver suites 221/221
(php, php-coverage, php-response-shapes). detect_changes {staged}: 1 changed
symbol (PHP_SCOPE_QUERY), 0 affected processes, risk LOW.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0184RmD24KFJidYqpM7v3XjR
* fix(rust): emit a node for a closure binding and make it a call SOURCE (#2699 part B, S3)
Rust was the one exception to #2687's "a closure bound to a name is a Function
node in every language": `let handler = || target(1);` produced NO graph node at
all, so the closure could be neither a call target nor a call source.
Needed BOTH query channels, which is the finding worth recording. Porting only
the scope-resolution rule (as S1 did for PHP) changed nothing measurable here,
because there was no node to attribute anything to:
- languages/rust/query.ts — closure-binding declaration, @declaration.function
on the INNER closure_expression so anchor.range aligns with the existing
(closure_expression) @scope.function. This is what gives the closure's own
scope a callable in ownedDefs, which is what stops pickCallerCallableDef
falling through to the enclosing fn.
- tree-sitter-queries.ts — @definition.function on the OUTER let_declaration.
This emits the Function NODE that Rust never had.
Note the deliberate anchor asymmetry between the two channels: the graph-node
channel anchors the WRAPPER (matching the existing
(lexical_declaration (variable_declarator ... (arrow_function))) rule), while
the scope-resolution channel anchors the INNER closure (to align with
@scope.function). Getting these backwards silently produces either no node or
an unattributable one, so both sites carry a comment saying so.
Measured on a fixture — `let handler = || target(1);` inside outer():
Function:src/a.rs:outer CALLS Function:src/a.rs:outer.handler@2:4
Function:src/a.rs:outer.handler@2:4 CALLS Function:src/a.rs:target
Previously the whole binding was absent and the call read as `outer -> target`.
The rule also covers `move` closures: the closure_expression node spans the
`move` keyword.
Verification: closure-binding-labels 50/50; rust.test.ts 192/192;
rust-coverage, rust-f70, rust-scope all pass; rust-captures-golden passes
UNCHANGED, so no golden regeneration was required. detect_changes {staged}:
2 changed symbols (RUST_SCOPE_QUERY, RUST_QUERIES), 0 affected processes,
risk LOW.
One caveat on the suite runs: this host times out `beforeAll` hooks at the
default 60s under load — rust.test.ts needed --hookTimeout=600000 to complete,
and a concurrent second vitest run starves worker startup entirely (every test
fails at ~5001ms). Both are host artifacts, not signal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0184RmD24KFJidYqpM7v3XjR
* fix(kotlin,ruby): a closure binding is a call SOURCE, via a Block-scope callable boundary (#2699 part B, S2)
Kotlin and Ruby anchor a closure on a Block-kind scope — Kotlin lambda_literal
and Ruby do_block/block are @scope.block DELIBERATELY (#1757 smart casts), so
they must not be re-kinded. pickCallerCallableDef gated its child-scope walk on
kind === 'Function', so a closure there could never become a call SOURCE.
Both halves are required; neither alone changes anything:
1. kotlin/query.ts and ruby/query.ts gain the closure-binding declaration rule,
with @declaration.function on the INNER lambda_literal / block so its range
aligns with the @scope.block range (the anchor discipline documented in
javascript/query.ts). Without this the closure scope owns no callable def.
2. pickCallerCallableDef accepts a Block-kind child as a callable boundary when
the scope IS that callable's body. Without this the kind gate still rejects.
The alignment test in (2) is the part worth scrutiny. Relaxing the kind gate to
accept ANY Block owning a callable would be a real regression: a nested
`fun foo()` declared inside a block is owned by that block, so a call made at
BLOCK level — outside foo — would be misattributed to foo. Comparing the def's
declaration position against the scope's start position discriminates them: for
a closure the declaration and the scope sit on the SAME node, so the positions
match; for a nested function the block starts at `{` while the def starts at the
declaration, so they do not. Existing Function-kind behaviour is untouched, so
every already-working language is unaffected by construction.
The comparison is base-safe: scope-extractor.ts builds a def id as
`def:<filePath>#<startLine>:<startCol>:<type>:<name>` from the same Range a
scope carries, so both sides share one coordinate base. This is called out in
the helper's docblock because `defStartLine` nearby documents its own output as
1-based, which invites a wrong "fix" (#2377 is exactly this class of hazard).
Ruby's call forms are restricted to lambda/proc by name: an unrestricted
(call block: (block)) would match ANY method call taking a block, so
`mapped = items.map { |i| ... }` would wrongly declare `mapped` a callable.
Verified against the parser: 3 matches (->, lambda, proc), map excluded.
Separate #eq? patterns rather than one #match? alternation, which is a known
hazard on this tree-sitter line.
Measured on fixtures:
Kotlin Function:src/A.kt:outer.handler@2:4 CALLS Function:src/A.kt:target
Ruby Function:src/a.rb:outer.handler@4:2 CALLS Method:src/a.rb:target#1
previously `outer -> target` and `outer#0 -> target#1`.
The pinned Kotlin test asserted the old behaviour by design and is INVERTED, not
deleted. Ruby had NO pinned case, so a new one is added rather than inverted.
The describe title no longer claimed something false ("not yet a call SOURCE"
now holds only for Dart) and was retitled.
Verification: closure-binding-labels 51/51; kotlin.test.ts, kotlin-coverage,
ruby.test.ts, ruby-scope, ruby-namespaced all pass (478 passed / 1 expected
inversion before the test was flipped). impact on pickCallerCallableDef:
CRITICAL, 191 impacted, ONE d=1 (resolveCallerGraphId) — the return contract is
unchanged, so that dependent is unaffected. detect_changes {staged}: 5 changed
symbols, 2 affected processes (both EmitReferencesViaLookup, one of them the new
ScopeIsCallableBody step), risk medium.
Dart remains the last failing language: dart/query.ts declares no
@scope.function at all, and dart/captures.ts synthesizes one only from a
declaration WITH a body node, which an expression-bodied closure lacks.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0184RmD24KFJidYqpM7v3XjR
* fix(dart): give a closure binding a scope and a distinct identity (#2699 part B, S4)
Dart was the last language where a closure binding could not be a call SOURCE,
and fixing only that would have made the graph WORSE, not better. This lands
both halves together for that reason.
## The attribution half
A Dart closure had no scope at all. `dart/query.ts` declares no
@scope.function anywhere — Dart's function scopes are SYNTHESIZED in
`dart/captures.ts` from `declNode` + `findFunctionBody(declNode)`, and
`findFunctionBody` looked only at the next named SIBLING for a `function_body`.
A closure literal carries its body as a CHILD (`function_expression_body`), so
it matched nothing and no scope was produced.
`query.ts` gains the closure-binding declaration rule and `findFunctionBody`
understands the child form. Deliberately NO @scope.function is added to the
query: it would collide at identical range with the synthesized one, and
duplicate scope ids make `buildScopeTree` throw, which DROPS THE WHOLE FILE.
## The identity half, and why it is not optional
With attribution alone, two same-named closures in one file both keyed to the
bare `Function:a.dart:handler`. One node then appeared to call BOTH targets —
a CALLS edge present nowhere in the source. That is worse than the missing edge
it replaced, so S4 could not ship without this.
Root cause is not Dart-specific. `enclosingCallablePrefix` derives a SEMANTIC
relation — what encloses this callable — by SYNTACTIC ancestor walk. Dart parses
`int outer() { … }` as `function_signature` followed by `function_body` as
SIBLINGS, so the enclosing callable is never an ancestor of code inside it and
no membership set can fix that; the walk looks in the wrong direction.
This is what SCIP and real compilers avoid by construction. SCIP keeps a local
symbol opaque (`local <id>` — no name, no position, no chain) and models
containment as a SEPARATE `enclosing_symbol` field; its spec says the local/global
choice should follow ACCESSIBILITY, not the ability to name an enclosure. Dart's
own analyzer answers this from `Element.enclosingElement` in the element model,
never from AST ancestry. clang uses `name@offset` for a function-local; Kythe
uses a document-scoped VName plus a `childof` edge. Identity is positional and
opaque; enclosure is a relation.
`findSplitBodyCallableAncestor` is the narrow fix at that seam: a fallback used
ONLY when the ancestor walk finds nothing, recovering the callable from the
body's preceding sibling.
The sibling must be a BARE SIGNATURE, and that restriction is load-bearing —
"any preceding callable sibling" is WRONG and was caught regressing PHP during
this work. In `<?php function target($x) {…} $handler = function ($x) {…};` the
closure is at FILE level, so the ancestor walk correctly finds nothing, the
fallback runs, and an unrestricted version mis-qualified the file-level
`$handler` as `target.$handler`. A preceding sibling is only an ENCLOSING
callable when it cannot hold its own body.
`SPLIT_SIGNATURE_NODE_TYPES` is exactly that set and is DERIVED, not listed:
`LOCAL_SCOPE_BODY_NODE_TYPES` is already `FUNCTION_NODE_TYPES` minus the bare
signature types, so the difference between them IS the split-signature set
(`function_signature`, `method_signature` — verified at runtime). PHP's
`function_definition` carries a body and is in both, so it is excluded. No
language is named in shared code, and any future split-grammar language is
covered for free.
## Verification
Full resolver sweep — the gate that caught #2714's Rust regression — 2926
passed / 1 skipped / 0 failed across 51 files. closure-binding-labels 52/52;
dart.test.ts, dart-coverage, callable-id-lockstep, function-local-identity,
caller-identity-regression all pass (156/156 across 6 files).
impact on `enclosingCallablePrefix`: LOW, 5 impacted, 3 d=1 all inside
parse-worker. detect_changes {staged}: 5 changed symbols, 0 affected processes,
risk LOW.
Three existing Dart expectations FLIPPED rather than being deleted: Dart locals
now carry the same enclosing-callable + position identity every other language
got in #2695, so `local.dart:handler` became `local.dart:caller.handler@1:2`.
A new test pins the actual defect — two same-named closures staying DISTINCT
nodes — because the qualification assertions alone would not fail if the
fabricated edge returned.
One note for future work: an id-shape assertion here carries a call-site suffix
on indirect invocations (`…handler@3:2:5:9`) but not on direct calls. That is
the callable-value-flow pass keying its edge by invocation position, not part of
the node id.
Part B is now complete: PHP (S1), Rust (S3), Kotlin + Ruby (S2), Dart (S4).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0184RmD24KFJidYqpM7v3XjR
* fix(scope-resolution): close every deferred item on #2699 (A1 values, twin-list guard, schema bumps)
Clears the limitations this PR had been carrying rather than leaving them as
follow-ups.
## A1 — function-local VALUES now carry their own identity
This was #2699's ORIGINAL complaint and the one a callable-only gate could never
reach: a top-level `const handler` and a function-local `const handler`
collapsed onto ONE `Const:v.ts:handler`. #2695 restricted position-qualified
identity to Function|Method|Constructor because the collision that produced
wrong CALLS edges was between callables, and widening churned ids for symbols
the pruner mostly deletes. The churn is real and is accepted here deliberately.
Widening needed THREE gates aligned, not one:
- id-building — `parse-worker.ts` nestedCallablePrefix
- resolution — `ids.ts` position key
- registration — `node-lookup.ts` position-key registration
Missing the third would register no position key for values, so every lookup
misses and falls through silently. That is the #2714 failure mode: the caller
attaches to a node that does not exist and the edge is DROPPED, which looks like
"zero dangling edges" from outside. All three now route through ONE predicate,
`isPositionQualifiedLocalLabel`, rather than repeating the label set a third
time.
Only LOCALS move. The prefix comes from `enclosingCallablePrefix`, which returns
undefined when nothing encloses the declaration, so top-level and class-member
ids are untouched — verified by the full resolver sweep, where a leak onto class
members would have broken assertions in every language. `Property` is included
on purpose: a class field stays unqualified because the prefix walk boundaries
on class-likes, while an object-literal property inside a function is genuinely
local and would otherwise keep the old collision.
Measured: `Const:v.ts:handler` + `Const:v.ts:run.handler@3:2`, two distinct
nodes. The KNOWN LIMIT test is FLIPPED per its own former instruction ("this
test should be updated as part of it rather than deleted").
## Schema bumps — required by Part B, not just by A1
INCREMENTAL_SCHEMA_VERSION 20 -> 21, parse-cache SCHEMA_BUMP 27 -> 29.
SCHEMA_BUMP is 29, not 28, and that is the point of re-checking it against
origin/main at MERGE time rather than branch time. This branch cut at 27 and
bumped to 28; #2415 also bumped 27 -> 28 and merged first. The automated
main-merge onto this branch surfaced the collision — leaving it at 28 would have
shipped this whole change with NO parse-cache invalidation, so every warm cache
keeps replaying the pre-fix captures and ids. This is the third instance of that
collision recorded in parse-cache.ts (#2632/#2653 hit it at v21, and
#2653/#2654 hit INCREMENTAL_SCHEMA_VERSION the same way).
Part B already changed emitted node ids AND edges on files that did not
themselves change (Dart locals re-keyed, Rust gained a node it never emitted,
five languages gained closure-source attribution). A v20 index topped up
incrementally keeps serving the old attribution, and a warm parse cache replays
the old captures and ids verbatim. Shipping S1-S4 without these would have let
every existing index silently keep the pre-fix graph.
## Twin-list drift guard — the sixth instance in this family
`IMPLICIT_RECEIVERS` (gitnexus-shared lookup-core.ts) and `THIS_RECEIVERS`
(type-env.ts) spell the same concept in two packages, and nothing enforced
agreement — `$this` was added to the shared list in #2714 only because it was
already in the other. New structural test asserts set equality plus the ONE
deliberate asymmetry (`Me`, Visual Basic spelling, absent from the shared list
because no SupportedLanguages entry uses it) in BOTH directions, so re-adding it
there or dropping it here each fail loudly.
Structural rather than value-imported: both constants are module-private, and
exporting them purely to be testable would widen two public surfaces to satisfy
a test.
## Two false comments corrected
- `lookup-core.ts` said "see the drift guard noted in #2714", implying a guard
existed when it was only a deferred follow-up. It exists now, and the
comment points at it.
- `callable-id-lockstep.test.ts` claimed its regex "fails if any site
reconstructs the id". It matches ONE template spelling; a hand-rolled
concatenation still slips past. Now stated as a tripwire for the known
shape, not a proof.
## Skill learnings
Four entries appended to eval/workflow_bench/learnings.jsonl from this run: the
v9fs safe-writer failure, backticks silently terminating a query template
literal (hit three times), a module-level TDZ const that passes tsc and then
presents as N file failures with ZERO failing assertions, and concurrent vitest
runs starving worker startup so a whole suite fails at ~5001ms.
## Verification
Full resolver sweep 2926 passed / 1 skipped / 0 failed (51 files) — identical to
pre-A1, which is the evidence that only locals moved. All EIGHT bench gates PASS
with fingerprints UNCHANGED, so no regeneration was needed. function-local-identity,
callable-id-lockstep, receiver-twin-list-drift and closure-binding-labels 71/71.
tsc --noEmit clean.
detect_changes {staged}: 9 changed symbols, 14 affected processes, risk HIGH —
expected, and the reason the sweep above is the gate rather than a targeted list.
Every affected process routes through `resolveDefGraphId`, the key chain Part A
measured at CRITICAL with 23 direct dependents.
Deliberately NOT done: the SCIP end state (opaque `local <id>` plus an explicit
enclosure EDGE instead of containment encoded in the id string). It is a design
direction, not a limitation of this work, and it is INCOMPATIBLE with A1 — A1
widens chain-encoded identity, that removes chain encoding entirely. Bundling
both would re-key every local twice. Written up in the research notes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0184RmD24KFJidYqpM7v3XjR
* test: update two assertions the #2699 changes correctly invalidated
Both failed on CI at
|
||
|---|---|---|
| .. | ||
| agents | ||
| analysis | ||
| bridge | ||
| configs | ||
| environments | ||
| prompts | ||
| tests | ||
| utils | ||
| workflow_bench | ||
| .env.example | ||
| .gitignore | ||
| __init__.py | ||
| constants.py | ||
| pyproject.toml | ||
| README.md | ||
| run_eval.py | ||
| tool_registry.py | ||
| uv.lock | ||
GitNexus SWE-bench Evaluation Harness
Evaluate whether GitNexus code intelligence improves AI agent performance on real software engineering tasks. Runs SWE-bench instances across multiple models and compares baseline (no graph) vs GitNexus-enhanced configurations.
What This Tests
Hypothesis: Giving AI agents structural code intelligence (call graphs, execution flows, blast radius analysis) improves their ability to resolve real GitHub issues — measured by resolve rate, cost, and efficiency.
Evaluation modes:
| Mode | What the agent gets |
|---|---|
baseline |
Standard bash tools (grep, find, cat, sed) — control group |
native |
Baseline + explicit GitNexus tools via eval-server (~100ms) |
native_augment |
Native tools + grep results automatically enriched with graph context (recommended) |
Recommended: Use
native_augmentmode. It mirrors the Claude Code model — the agent gets both explicit GitNexus tools (fast bash commands) AND automatic enrichment of grep results with callers, callees, and execution flows. The agent decides when to use explicit tools vs rely on enriched search output.
Models supported (see configs/models/ for the current list):
- Claude Haiku 4.5, Claude Sonnet 4, Claude Opus 4
- MiniMax M1 2.5, MiniMax M2.5
- GLM 4.7, GLM 5
- DeepSeek
- Any model supported by litellm (add a YAML config)
Prerequisites
- Python 3.11+
- Docker (for SWE-bench containers)
- Node.js 22+ (for GitNexus)
- API keys for your chosen models
Setup
cd eval
# Install dependencies
pip install -e .
# Set up API keys — copy the template and fill in your keys
cp .env.example .env
# Then edit .env and paste your key(s)
All models are routed through OpenRouter by default, so a single OPENROUTER_API_KEY is all you need. To use provider APIs directly (Anthropic, ZhipuAI, etc.), edit the model YAML in configs/models/ and set the corresponding key in .env.
# Pull SWE-bench Docker images (pulled on-demand, but you can pre-pull)
docker pull swebench/sweb.eval.x86_64.django_1776_django-16527:latest
Debug logging
Set GITNEXUS_EVAL_DEBUG=1 to include full Python tracebacks in run summaries and logs. By default, errors are sanitized to avoid leaking host paths or stack traces.
Quick Start
Debug a single instance
# Fastest way to verify everything works
python run_eval.py debug -m claude-haiku -i django__django-16527 --subset lite
Run a single configuration
# 5 instances, Claude Sonnet, native_augment mode (default)
python run_eval.py single -m claude-sonnet --subset lite --slice 0:5
# Baseline comparison (no GitNexus)
python run_eval.py single -m claude-sonnet --mode baseline --subset lite --slice 0:5
# Full Lite benchmark, 4 parallel workers
python run_eval.py single -m claude-sonnet --subset lite -w 4
Run the full matrix
# All models x all modes
python run_eval.py matrix --subset lite -w 4
# Key comparison: baseline vs native_augment
python run_eval.py matrix -m claude-sonnet -m claude-haiku --modes baseline --modes native_augment --subset lite --slice 0:50
Analyze results
# Summary table
python -m analysis.analyze_results results/
# Compare modes for a specific model
python -m analysis.analyze_results compare-modes results/ -m claude-sonnet
# GitNexus tool usage analysis
python -m analysis.analyze_results gitnexus-usage results/
# Export as CSV for further analysis
python -m analysis.analyze_results summary results/ --format csv > results.csv
# Run official SWE-bench test evaluation
python -m analysis.analyze_results summary results/ --swebench-eval
List available configurations
python run_eval.py list-configs
Architecture
eval/
run_eval.py # Main entry point (single, matrix, debug commands)
agents/
gitnexus_agent.py # GitNexusAgent: extends DefaultAgent with augmentation + metrics
environments/
gitnexus_docker.py # Docker env with GitNexus + eval-server + standalone tool scripts
bridge/
gitnexus_tools.sh # Bash wrappers (legacy — now standalone scripts are installed directly)
mcp_bridge.py # Legacy MCP bridge (kept for reference)
prompts/
system_baseline.jinja # System: persona + format rules
instance_baseline.jinja # Instance: task + workflow
system_native.jinja # System: + GitNexus tool reference
instance_native.jinja # Instance: + GitNexus debugging workflow
system_native_augment.jinja # System: + GitNexus tools + grep enrichment docs
instance_native_augment.jinja # Instance: + GitNexus workflow + risk assessment
configs/
models/ # Per-model YAML configs
modes/ # Per-mode YAML configs (baseline, native, native_augment)
analysis/
analyze_results.py # Post-run comparative analysis
results/ # Output directory (gitignored)
How It Works
Template structure
mini-swe-agent requires two Jinja templates:
- system_template → system message: persona, format rules, tool reference (static)
- instance_template → first user message: task, workflow, rules, examples (contains
{{task}})
Each mode has a system_{mode}.jinja + instance_{mode}.jinja pair. The agent loads both automatically based on the configured mode.
Per-instance flow
- Docker container starts with SWE-bench instance (repo at specific commit)
- GitNexus setup: Node.js + gitnexus installed,
gitnexus analyzeruns (or restores from cache) - Eval-server starts:
gitnexus eval-serverdaemon (persistent HTTP server, keeps LadybugDB warm) - Standalone tool scripts installed in
/usr/local/bin/— works withsubprocess.run(no.bashrcneeded) - Agent runs with the configured model + system prompt + GitNexus tools
- Agent's patch is extracted as a git diff
- Metrics collected: cost, tokens, tool calls, GitNexus usage, augmentation stats
Tool architecture
Agent → bash command → /usr/local/bin/gitnexus-query
→ curl http://127.0.0.1:4848/tool/query (fast path: eval-server, ~100ms)
→ npx gitnexus query (fallback: cold CLI, ~5-10s)
Each tool script in /usr/local/bin/ is standalone — no sourcing, no env inheritance needed. This is critical because mini-swe-agent runs every command via subprocess.run in a fresh subshell.
Eval-server
The eval-server is a lightweight HTTP daemon that:
- Keeps LadybugDB warm in memory (no cold start per tool call)
- Returns LLM-friendly text (not raw JSON — saves tokens)
- Includes next-step hints to guide tool chaining (query → context → impact → fix)
- Auto-shuts down after idle timeout
CLI flags:
| Flag | Default | Purpose |
|---|---|---|
--port <port> |
4848 |
Port to listen on |
--host <host> |
127.0.0.1 |
Bind address — use 0.0.0.0 for cross-container access |
--idle-timeout <seconds> |
0 (disabled) |
Auto-shutdown after N seconds of inactivity |
READY signal:
When the server is ready, it writes to stdout:
# IPv4
GITNEXUS_EVAL_SERVER_READY:127.0.0.1:4848
# IPv6 (bracketed to avoid colon ambiguity)
GITNEXUS_EVAL_SERVER_READY:[::1]:4848
Parse the port as the last colon-segment (split(':').pop()) — not split(':')[1], which breaks for IPv6 and for non-loopback IPv4 hosts added in this release.
Custom port and host
run_eval.py does not expose --port or --host as CLI flags. Configure them in your mode YAML under the environment: key:
# configs/modes/native_augment.yaml (or whichever mode you're running)
environment:
eval_server_port: 4849 # change if 4848 is already in use on the host
eval_server_host: "0.0.0.0" # bind all interfaces — needed for cross-container setups
Defaults are port: 4848 and host: 127.0.0.1 (loopback only). Use 0.0.0.0 only when the agent container needs to reach the eval-server from a separate network namespace. The health probe and tool scripts connect via the configured bind host (defaulting to 127.0.0.1), which is reachable for both loopback and all-interface binds.
"localhost" is also a valid eval_server_host value. The OS resolves it at bind time — typically 127.0.0.1 on dual-stack or IPv4-only systems, and ::1 on IPv6-only systems. The exact result depends on your /etc/hosts and gai.conf. The READY signal will reflect the actual bound address (e.g. GITNEXUS_EVAL_SERVER_READY:127.0.0.1:4848 or GITNEXUS_EVAL_SERVER_READY:[::1]:4848), not the literal string localhost. Use this when you want the server to bind to whichever loopback address the OS prefers rather than forcing IPv4.
Running eval-server directly in Docker / Docker Compose:
# Bind to all interfaces so sibling containers can reach it
gitnexus eval-server --host 0.0.0.0 --port 4848
# Then probe from a sibling container via its service hostname
curl http://eval-container:4848/health
If you need a non-default port (e.g. to avoid conflicts), pass --port <port> alongside --host. The READY signal will reflect both:
GITNEXUS_EVAL_SERVER_READY:0.0.0.0:5000
Parse the port as the last colon-segment (split(':').pop()) — safe for both IPv4 and bracketed IPv6 forms.
Index caching
SWE-bench repos repeat (Django has 200+ instances at different commits). The harness caches GitNexus indexes per (repo, commit) hash in ~/.gitnexus-eval-cache/ to avoid redundant re-indexing.
Grep augmentation (native_augment mode)
When the agent runs grep or rg, the observation is post-processed: the agent class calls gitnexus-augment on the search pattern and appends [GitNexus] annotations showing callers, callees, and execution flows for matched symbols. This mirrors the Claude Code / Cursor hook integration.
Adding Models
Create a YAML file in configs/models/:
# configs/models/my-model.yaml
model:
model_name: "openrouter/provider/model-name"
cost_tracking: "ignore_errors" # if not in litellm's cost DB
model_kwargs:
max_tokens: 8192
temperature: 0
The model name follows litellm conventions.
Metrics Collected
| Metric | Description |
|---|---|
| Patch Rate | % of instances where agent produced a patch |
| Resolve Rate | % of instances where patch passes tests (requires --swebench-eval) |
| Total Cost | API cost across all instances |
| Avg Cost/Instance | Cost efficiency |
| API Calls | Number of LLM calls |
| GN Tool Calls | How many GitNexus tools the agent used |
| Augment Hits | How many grep/find results got enriched |
| Augment Hit Rate | % of search commands that got useful enrichment |