GitNexus/gitnexus/bench/mcp-tools-list/baselines.json
svjack ccf6b4743d
Some checks are pending
CodeQL / Analyze (javascript-typescript) (push) Waiting to run
CodeQL / Analyze (python) (push) Waiting to run
Gitleaks / gitleaks (push) Waiting to run
Publish / Publish to npm (push) Blocked by required conditions
Publish / Build & Push RC Docker images (push) Blocked by required conditions
Publish / Classify release event (push) Waiting to run
Publish / RC guard (marker + release-PR skip) (push) Blocked by required conditions
Publish / ci (push) Blocked by required conditions
Scorecard / Scorecard analysis (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-cli) (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-web) (push) Waiting to run
feat(mcp): add read_file + grep tools (REST parity for /api/file slice + /api/grep) (#3377)
* feat(mcp): add read_file + grep tools (REST parity for /api/file slice + /api/grep)

* chore(autofix): apply prettier + eslint fixes via /autofix command

* Address PR review feedback (#3377)

- Fail read_file and grep when full source is unavailable, matching the HTTP 410 contract instead of an empty grep or a not-found on a missing checkout.
- Reject branch on those tools so a pinned index is not labeled onto checkout bytes, and stop advertising branch in their schemas.
- Point the grep hint at a 0-based read_file window, pass caseSensitive and literal through, and test the handlers.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback (#3377)

- Keep read_file and grep in the multi-repo schema requirement without advertising branch.
- Reject negative maxLines and return integer slice bounds for fractional line positions.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback (#3377)

- Reject a negative read_file endLine before slicing so JavaScript does not treat it as an offset from the end of the file.
- Drop the fractional startLine/endLine claim so the integer schema is the advertised contract.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address PR review feedback (#3377)

- Skip indexed grep paths whose realpath leaves the checkout so a symlink cannot return lines from outside the repo.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(bench): record the 19-tool MCP roster

read_file and grep are real tools, so tools/list and GITNEXUS_TOOLS both
moved from 17 to 19. The timing ratios were already inside budget.

Co-authored-by: Cursor <cursoragent@cursor.com>

* refactor(mcp): share read_file and grep contracts with existing helpers

Boolean grep flags go through isFlagTrue, the whole-file cap is one constant, and checkout tools stay on the per-repo schema without advertising branch.

---------

Co-authored-by: svjack <svjack@example.com>
Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-27 13:16:11 +00:00

35 lines
3.4 KiB
JSON
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

{
"_what": "Baselines for bench/mcp-tools-list/measure.mjs --check. Guards unrestricted MCP tools/list after #3259: validated registry cardinality (countRepos) instead of listRepos() staleness git. Same approach as bench/parse-dispatch-rounds and bench/python-workspace-import-scan — exact floors first; the only timing arms are ratios. Never a millisecond ceiling.",
"_triage": "READ THIS BEFORE RE-RUNNING. n_repos, count_repos, list_repos, tools_listed, schema_read_only_requires_repo and schema_mutating_requires_repo are DETERMINISTIC: a re-run never changes them, and none may be re-baselined to make CI green. count_vs_listRepos_ratio and listTools_vs_listRepos_ratio are UPPER timing arms. Runner contention dominates both, so re-run on an idle machine before investigating and read the reported `reps` first. If exactly one arm fails and it is a timing arm, suspect the machine. If listRepos() itself stops paying git (unrelated change), both ratios move toward 1 and need an explained re-baseline — do not just raise the budget.",
"n_repos": 200,
"count_repos": 200,
"list_repos": 200,
"_shape_note": "THE FLOOR. Without these three, every ratio below is a ceiling over nothing. listTools_vs_listRepos_ratio only asserts something while the corpus still pays N parallel rev-list processes. Shrink it to three happy-path rows and both arms are cheap; the ratio still passes, asserting a property the corpus no longer has.",
"tools_listed": 19,
"_tools_note": "Exact GITNEXUS_TOOLS roster size returned by client.listTools(). 19 after read_file and grep were added as the MCP twins of GET /api/file and GET /api/grep (was 17). GITNEXUS_TOOLS.length and listTools must move together; a schema path that throws, filters, or returns [] still looks fast on the ratio arm.",
"schema_read_only_requires_repo": true,
"schema_mutating_requires_repo": true,
"_schema_note": "This unrestricted 200-repo fixture has no cwd default, so both flags must stay true. Skipping the cwd probe and advertising a single-repo schema would pass every timing arm.",
"count_vs_listRepos_budget": 0.15,
"_count_ratio_note": "countRepos_ms / listRepos_ms. A RATIO rather than a millisecond ceiling, deliberately: wall-clock is runner-speed-dependent, and this repo has already been bitten by a fixed ms budget. Putting staleness git back on countRepos collapses this toward 1. Budget is 0.15 — more than 10x above the measured ~0.01, same fail-closed presence check as import-target. min-of-7 estimator.",
"listTools_vs_listRepos_budget": 0.75,
"_listTools_ratio_note": "client.listTools()_ms / listRepos_ms. Restoring listRepos() on toolSchemaRepoRequirements collapses this toward 1+. Budget is 0.75 — listRepos is git-bound and can get relatively faster on GitHub-hosted runners than the Node-bound listTools arm (measured ~0.22–0.28 here). The collapse-to-1 regression still fails. min-of-7 estimator.",
"_measured": {
"count_vs_listRepos_ratio": 0.026,
"count_vs_listRepos_ratio_samples": [0.025, 0.025, 0.026, 0.026, 0.025],
"listTools_vs_listRepos_ratio": 0.283,
"listTools_vs_listRepos_ratio_samples": [0.242, 0.242, 0.223, 0.267, 0.234, 0.283],
"count_ms": 8.99,
"listRepos_ms": 349.63,
"listTools_ms": 99.04,
"reps": 7
},
"_measured_note": "Maxima (and sample lists) over consecutive local runs using the min-of-7 estimator, including one --check pass. Milliseconds are diagnostic context only — nothing gates on them."
}