mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-09-17 23:52:36 +00:00
18cbeb907c
* feat(eval): record provider-native usage at the gateway, not after translation
The benchmark reads token counts out of Claude Code's session output, which is
Anthropic-shaped whatever actually served the request. That holds until the
upstream is OpenAI, because the two providers do not merely name their fields
differently - they mean opposite things by them:
Anthropic: total_input = input_tokens + cache_creation + cache_read
(input_tokens is the UNCACHED remainder; cache fields ADD)
OpenAI: total_input = input_tokens
ordinary = input_tokens - cached - cache_write
(input_tokens is the WHOLE; cache fields are SUBSETS)
Adding OpenAI's three double-counts; subtracting Anthropic's under-counts. One
shared struct cannot be right for both, so the seam goes at the gateway, on the
far side of the translation: a LiteLLM callback appends each upstream request's
usage verbatim, along with the model that actually answered, the response id and
the cell it belongs to. Normalization is derived offline from that record, so the
derivation can be revisited without re-running a paid sweep.
Two rules the tests encode literally.
The native object is authoritative. The callback stores it unflattened,
unrenamed and unsummed. Reasoning tokens are kept as the decomposition of output
tokens they are, not added to them a second time.
A field nobody reported is unknown, never zero. A stored cache_read of 0 used to
mean either "the provider said zero" or "our adapter never looked" - the first
says caching is not working, the second says we cannot tell. NormalizedUsage
therefore uses None, and refuses to compute the ordinary portion when a term is
missing rather than subtracting an invented zero.
Mutation-checked three ways. Giving OpenAI Anthropic's arithmetic fails four
tests. Making unknown fall back to zero fails the unknown test. Dropping
input_tokens_details in the callback fails the end-to-end accounting test with
"assert None == 3000" - it goes unknown rather than passing with zeros, which
was the point of the exercise.
The actual model is recorded separately from the requested role because several
Claude role names map onto one upstream model here; pricing must follow what
answered. Cost is deliberately NOT stored: prices change, and tokens plus a
versioned pricing table can answer both what a past run cost and what the same
usage would cost today, without rewriting historical evidence.
The callback never raises. A cell that fails still spent money upstream, and
losing the accounting because a log write failed is the worse outcome. Failed
requests are recorded too.
No caching configuration, model, skill or promotion change: this installs the
thermometer without altering the experiment. 538 eval tests pass plus 27 gateway
tests; ruff clean. The two test_model_gateway.py failures are environmental -
litellm[proxy]'s console script is absent in this venv - and predate this branch.
* fix(eval): drop the accidentally committed .venv symlink
I symlinked eval/.venv at a sibling worktree's virtualenv to avoid rebuilding
it, and git add -A committed the symlink. .gitignore lists ".venv/" with a
trailing slash, which matches a directory and not a symlink, so nothing stopped
it.
That broke eval / containment (windows), where uv then refused to create the
environment: "failed to create directory eval\\.venv: Cannot create a file when
that file already exists". A machine-specific absolute path had no business in
the tree in the first place.
Removed, and .gitignore now also lists the bare name so the same slip cannot
repeat.
* Address PR review feedback (#3220)
Forward the usage environment into the proxy. This is the one that mattered:
the callback returns immediately when GITNEXUS_BENCH_PROVIDER_USAGE is absent,
the proxy runs as its own process, and Popen(env=...) REPLACES the parent
environment rather than extending it. The gateway's allowlist carried the
OpenAI and master keys and nothing else, so the callback loaded, found no
destination, and silently recorded nothing on every request. The accounting
looked configured and measured nothing at all.
My tests could not see it. They set the variable in-process and called the
logger directly, so none of them ever crossed the subprocess boundary the
feature actually runs behind. The new test drives OpenAIGateway.__enter__ with
Popen captured and asserts each variable reaches the child - and that the
result is still an allowlist rather than the inherited parent environment,
since forwarding by name is what keeps the credential boundary explicit.
Resolve the provider label into an adapter key. The callback recorded
LiteLLM's custom_llm_provider, which is "openai", while the adapter table is
keyed "openai-responses" - so nothing the logger wrote could have been
normalized. The end-to-end test hid this by passing OPENAI_RESPONSES by hand
instead of using the provider the log recorded; it now uses the logged value,
which is what makes the mismatch visible.
The label alone cannot pick an adapter: LiteLLM reports "openai" for Chat
Completions as well, and the two report usage differently. canonical_provider
combines the label with the call type and returns None when it cannot resolve
one, so normalize_usage refuses rather than guessing token semantics. Both are
stored - provider_label is what LiteLLM said, provider is the adapter key.
The shared env-var names moved into provider_usage.py so model_gateway can
import them without importing litellm, which only the in-proxy callback needs.
Mutation-checked. Removing the forwarding loop fails the gateway test; using
the raw label as the adapter key fails two.
656 eval tests pass, ruff clean. The two test_model_gateway.py failures are the
environmental ones - litellm[proxy]'s console script is absent here, which is
also why the new test patches the argv builder to reach Popen at all.
* fix(eval): stop recording a cell id the proxy cannot know
Setting out to build the correlation this PR was missing - cell usage as the
sum of its upstream requests - turned up that the field it would have been
built on cannot hold what its name claims.
attach_openai_gateway wraps the whole sweep (runner.py:2122), so ONE proxy
serves every cell, and its environment is fixed for that process's lifetime.
Cells run concurrently under --workers and interleave requests through it. A
cell id forwarded at launch is therefore the same constant on every event the
callback ever writes - not an attribution, just a label that looks like one.
Worse than absent, because a reader would trust it.
So GITNEXUS_BENCH_CELL_ID is gone rather than left to be wired up later. What
remains is honest about its scope: sweep_id is genuinely sweep-wide, and
session_id is the per-request half - the only thing that can attribute a
request to a cell, since anything read from the environment is shared by all of
them. It is recorded even when the provider supplies nothing, because knowing
attribution is unavailable is itself a fact about the run.
Pinned by a test asserting the forwarded set contains no per-cell variable, so
a later change does not reintroduce one and quietly stamp a single value across
concurrent cells.
What this leaves open, stated plainly: per-cell attribution is NOT built, and
cannot be until a per-request identifier is available. Whether Claude Code
propagates a session identifier through the proxy is unverified - determining
it needs a real session against the gateway, which is a paid run. Sweep-level
totals and per-request cache ratios do not need it, and those are what the
caching question actually turns on.
658 eval tests pass, ruff clean; the two test_model_gateway.py failures remain
environmental.
* fix(eval): keep the usage callback importable the way LiteLLM loads it
CI caught a regression I introduced: "ImportError: Could not import handler
from provider_usage_callback", and the proxy exited before becoming ready.
Moving the shared constants into provider_usage.py, I imported them from the
callback with "from .provider_usage import ...". But LiteLLM resolves a dotted
callback through spec_from_file_location against the config directory, so the
copied file runs as a top-level module with no parent package and no sys.path
entry - the relative import raises and the gateway never starts. The module's
own docstring says it is deliberately self-contained for exactly this reason,
and I broke that invariant while tidying.
The in-package tests could not see it. They import
workflow_bench.litellm_usage_callback, where the relative import resolves
fine; the failure only exists on the path where the file is copied and loaded
standalone.
The callback carries its own literals again. Two tests keep that honest: one
loads the copied file the way LiteLLM does - by path, as a top-level module -
so an import that only works in-package fails there, and one asserts the
copied constants and the provider resolver still agree with the canonical
copies in provider_usage.py, so the deliberate duplication cannot drift
silently.
Mutation-checked: restoring the relative import reproduces CI's exact error.
660 eval tests pass locally; the two remaining test_model_gateway.py failures
are the environmental ones (litellm[proxy]'s console script is absent here,
which is also why this never reproduced locally).
* test(eval): import the installed callback instead of grepping it
Two review findings on the same weakness, both correct.
The install test asserted "class ProviderUsageLogger" appeared in the copied
file's text. That passes whenever the string is present, including when the
module cannot load at all - which is precisely how a package-relative import
got through review here and took the proxy down. It now loads the copy the way
LiteLLM does, by path as a top-level module, and checks the handler instance
the config actually names.
The gateway-forwarding test built its work directory with tempfile.mkdtemp(),
which nothing removed, so every run left the generated config and the copied
callback behind in the system temp directory. It uses the pytest-managed
tmp_path fixture like its neighbours.
660 eval tests pass; the two test_model_gateway.py failures are the
environmental ones.
* fix(eval): record failures on the synchronous callback path too
ProviderUsageLogger overrode both async hooks and the sync SUCCESS hook, but
not the sync failure hook. On that path failures fell through to CustomLogger's
base implementation and were never appended - so a sweep recorded its
successes and quietly understated what it spent, since a failed request is
billed all the same. That contradicts the module's own stated reason for
handling failures at all.
The failure test could not have caught it: it called _append directly, which
exercises neither public hook. Both failure tests now drive the hooks LiteLLM
actually calls, and a new one walks all four - sync and async, success and
failure - asserting each records in order. Removing the sync failure hook fails
both.
661 eval tests pass; the two test_model_gateway.py failures remain
environmental.
---------
Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
|
||
|---|---|---|
| .. | ||
| agents | ||
| analysis | ||
| bridge | ||
| configs | ||
| environments | ||
| prompts | ||
| tests | ||
| utils | ||
| workflow_bench | ||
| .env.example | ||
| .gitignore | ||
| __init__.py | ||
| constants.py | ||
| pyproject.toml | ||
| README.md | ||
| run_eval.py | ||
| tool_registry.py | ||
| uv.lock | ||
GitNexus SWE-bench Evaluation Harness
Evaluate whether GitNexus code intelligence improves AI agent performance on real software engineering tasks. Runs SWE-bench instances across multiple models and compares baseline (no graph) vs GitNexus-enhanced configurations.
What This Tests
Hypothesis: Giving AI agents structural code intelligence (call graphs, execution flows, blast radius analysis) improves their ability to resolve real GitHub issues — measured by resolve rate, cost, and efficiency.
Evaluation modes:
| Mode | What the agent gets |
|---|---|
baseline |
Standard bash tools (grep, find, cat, sed) — control group |
native |
Baseline + explicit GitNexus tools via eval-server (~100ms) |
native_augment |
Native tools + grep results automatically enriched with graph context (recommended) |
Recommended: Use
native_augmentmode. It mirrors the Claude Code model — the agent gets both explicit GitNexus tools (fast bash commands) AND automatic enrichment of grep results with callers, callees, and execution flows. The agent decides when to use explicit tools vs rely on enriched search output.
Models supported (see configs/models/ for the current list):
- Claude Haiku 4.5, Claude Sonnet 4, Claude Opus 4
- MiniMax M1 2.5, MiniMax M2.5
- GLM 4.7, GLM 5
- DeepSeek
- Any model supported by litellm (add a YAML config)
Prerequisites
- Python 3.11+
- Docker (for SWE-bench containers)
- Node.js 22+ (for GitNexus)
- API keys for your chosen models
Setup
cd eval
# Install dependencies
pip install -e .
# Set up API keys — copy the template and fill in your keys
cp .env.example .env
# Then edit .env and paste your key(s)
All models are routed through OpenRouter by default, so a single OPENROUTER_API_KEY is all you need. To use provider APIs directly (Anthropic, ZhipuAI, etc.), edit the model YAML in configs/models/ and set the corresponding key in .env.
# Pull SWE-bench Docker images (pulled on-demand, but you can pre-pull)
docker pull swebench/sweb.eval.x86_64.django_1776_django-16527:latest
Debug logging
Set GITNEXUS_EVAL_DEBUG=1 to include full Python tracebacks in run summaries and logs. By default, errors are sanitized to avoid leaking host paths or stack traces.
Quick Start
Debug a single instance
# Fastest way to verify everything works
python run_eval.py debug -m claude-haiku -i django__django-16527 --subset lite
Run a single configuration
# 5 instances, Claude Sonnet, native_augment mode (default)
python run_eval.py single -m claude-sonnet --subset lite --slice 0:5
# Baseline comparison (no GitNexus)
python run_eval.py single -m claude-sonnet --mode baseline --subset lite --slice 0:5
# Full Lite benchmark, 4 parallel workers
python run_eval.py single -m claude-sonnet --subset lite -w 4
Run the full matrix
# All models x all modes
python run_eval.py matrix --subset lite -w 4
# Key comparison: baseline vs native_augment
python run_eval.py matrix -m claude-sonnet -m claude-haiku --modes baseline --modes native_augment --subset lite --slice 0:50
Analyze results
# Summary table
python -m analysis.analyze_results results/
# Compare modes for a specific model
python -m analysis.analyze_results compare-modes results/ -m claude-sonnet
# GitNexus tool usage analysis
python -m analysis.analyze_results gitnexus-usage results/
# Export as CSV for further analysis
python -m analysis.analyze_results summary results/ --format csv > results.csv
# Run official SWE-bench test evaluation
python -m analysis.analyze_results summary results/ --swebench-eval
List available configurations
python run_eval.py list-configs
Architecture
eval/
run_eval.py # Main entry point (single, matrix, debug commands)
agents/
gitnexus_agent.py # GitNexusAgent: extends DefaultAgent with augmentation + metrics
environments/
gitnexus_docker.py # Docker env with GitNexus + eval-server + standalone tool scripts
bridge/
gitnexus_tools.sh # Bash wrappers (legacy — now standalone scripts are installed directly)
mcp_bridge.py # Legacy MCP bridge (kept for reference)
prompts/
system_baseline.jinja # System: persona + format rules
instance_baseline.jinja # Instance: task + workflow
system_native.jinja # System: + GitNexus tool reference
instance_native.jinja # Instance: + GitNexus debugging workflow
system_native_augment.jinja # System: + GitNexus tools + grep enrichment docs
instance_native_augment.jinja # Instance: + GitNexus workflow + risk assessment
configs/
models/ # Per-model YAML configs
modes/ # Per-mode YAML configs (baseline, native, native_augment)
analysis/
analyze_results.py # Post-run comparative analysis
results/ # Output directory (gitignored)
How It Works
Template structure
mini-swe-agent requires two Jinja templates:
- system_template → system message: persona, format rules, tool reference (static)
- instance_template → first user message: task, workflow, rules, examples (contains
{{task}})
Each mode has a system_{mode}.jinja + instance_{mode}.jinja pair. The agent loads both automatically based on the configured mode.
Per-instance flow
- Docker container starts with SWE-bench instance (repo at specific commit)
- GitNexus setup: Node.js + gitnexus installed,
gitnexus analyzeruns (or restores from cache) - Eval-server starts:
gitnexus eval-serverdaemon (persistent HTTP server, keeps LadybugDB warm) - Standalone tool scripts installed in
/usr/local/bin/— works withsubprocess.run(no.bashrcneeded) - Agent runs with the configured model + system prompt + GitNexus tools
- Agent's patch is extracted as a git diff
- Metrics collected: cost, tokens, tool calls, GitNexus usage, augmentation stats
Tool architecture
Agent → bash command → /usr/local/bin/gitnexus-query
→ curl http://127.0.0.1:4848/tool/query (fast path: eval-server, ~100ms)
→ npx gitnexus query (fallback: cold CLI, ~5-10s)
Each tool script in /usr/local/bin/ is standalone — no sourcing, no env inheritance needed. This is critical because mini-swe-agent runs every command via subprocess.run in a fresh subshell.
Eval-server
The eval-server is a lightweight HTTP daemon that:
- Keeps LadybugDB warm in memory (no cold start per tool call)
- Returns LLM-friendly text (not raw JSON — saves tokens)
- Includes next-step hints to guide tool chaining (query → context → impact → fix)
- Auto-shuts down after idle timeout
CLI flags:
| Flag | Default | Purpose |
|---|---|---|
--port <port> |
4848 |
Port to listen on |
--host <host> |
127.0.0.1 |
Bind address — use 0.0.0.0 for cross-container access |
--idle-timeout <seconds> |
0 (disabled) |
Auto-shutdown after N seconds of inactivity |
READY signal:
When the server is ready, it writes to stdout:
# IPv4
GITNEXUS_EVAL_SERVER_READY:127.0.0.1:4848
# IPv6 (bracketed to avoid colon ambiguity)
GITNEXUS_EVAL_SERVER_READY:[::1]:4848
Parse the port as the last colon-segment (split(':').pop()) — not split(':')[1], which breaks for IPv6 and for non-loopback IPv4 hosts added in this release.
Custom port and host
run_eval.py does not expose --port or --host as CLI flags. Configure them in your mode YAML under the environment: key:
# configs/modes/native_augment.yaml (or whichever mode you're running)
environment:
eval_server_port: 4849 # change if 4848 is already in use on the host
eval_server_host: "0.0.0.0" # bind all interfaces — needed for cross-container setups
Defaults are port: 4848 and host: 127.0.0.1 (loopback only). Use 0.0.0.0 only when the agent container needs to reach the eval-server from a separate network namespace. The health probe and tool scripts connect via the configured bind host (defaulting to 127.0.0.1), which is reachable for both loopback and all-interface binds.
"localhost" is also a valid eval_server_host value. The OS resolves it at bind time — typically 127.0.0.1 on dual-stack or IPv4-only systems, and ::1 on IPv6-only systems. The exact result depends on your /etc/hosts and gai.conf. The READY signal will reflect the actual bound address (e.g. GITNEXUS_EVAL_SERVER_READY:127.0.0.1:4848 or GITNEXUS_EVAL_SERVER_READY:[::1]:4848), not the literal string localhost. Use this when you want the server to bind to whichever loopback address the OS prefers rather than forcing IPv4.
Running eval-server directly in Docker / Docker Compose:
# Bind to all interfaces so sibling containers can reach it
gitnexus eval-server --host 0.0.0.0 --port 4848
# Then probe from a sibling container via its service hostname
curl http://eval-container:4848/health
If you need a non-default port (e.g. to avoid conflicts), pass --port <port> alongside --host. The READY signal will reflect both:
GITNEXUS_EVAL_SERVER_READY:0.0.0.0:5000
Parse the port as the last colon-segment (split(':').pop()) — safe for both IPv4 and bracketed IPv6 forms.
Index caching
SWE-bench repos repeat (Django has 200+ instances at different commits). The harness caches GitNexus indexes per (repo, commit) hash in ~/.gitnexus-eval-cache/ to avoid redundant re-indexing.
Grep augmentation (native_augment mode)
When the agent runs grep or rg, the observation is post-processed: the agent class calls gitnexus-augment on the search pattern and appends [GitNexus] annotations showing callers, callees, and execution flows for matched symbols. This mirrors the Claude Code / Cursor hook integration.
Adding Models
Create a YAML file in configs/models/:
# configs/models/my-model.yaml
model:
model_name: "openrouter/provider/model-name"
cost_tracking: "ignore_errors" # if not in litellm's cost DB
model_kwargs:
max_tokens: 8192
temperature: 0
The model name follows litellm conventions.
Metrics Collected
| Metric | Description |
|---|---|
| Patch Rate | % of instances where agent produced a patch |
| Resolve Rate | % of instances where patch passes tests (requires --swebench-eval) |
| Total Cost | API cost across all instances |
| Avg Cost/Instance | Cost efficiency |
| API Calls | Number of LLM calls |
| GN Tool Calls | How many GitNexus tools the agent used |
| Augment Hits | How many grep/find results got enriched |
| Augment Hit Rate | % of search commands that got useful enrichment |