mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-10-03 02:21:44 +00:00
3 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Yahoo
|
0edf9ce0ff
|
fix(ruby): guard gem requires with dependency metadata (#3096)
* docs(plans): add ruby gem require boundary plan * fix(ruby): guard gem requires with dependency metadata * fix(ruby): scope gem sources by manifest * test(ruby): model resolved lockfile specs * fix(ruby): stop local gem suffix fallthrough * docs: remove Ruby resolution plan * test(ruby): gate gem resolution correctness and scaling * test(ruby): align gem benchmark baseline with ratio gates * Address PR review feedback (#3096) - Strip a trailing .rb so local and external gem prefix matching follows Ruby's optional-suffix require. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Gergő Magyar <gergomagyar@icloud.com> Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Gergő Magyar
|
95f87fc12a
|
perf(ingestion): Linux-kernel-scale analysis — worker-pool parse + finalize O(n²) + scope-resolution memory wall (#1983) (#2038)
* fix(ingestion): reduce parse-phase memory for huge repos (#1983)
Stop retaining full parse-cache chunks in RAM alongside the merged graph,
slim on-disk shards, defer worker ParsedFile emission for scope-resolver
languages, and add GITNEXUS_DEBUG_HEAP probes for OOM diagnosis.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(ingestion): address #2038 tri-review findings (parse-phase memory)
Resolves the confirmed review findings on PR #2038:
- P1: thread exportedTypeMap through the sequential parse path
(processParsingSequential) so a no-worker run over a partially-warm
cache no longer silently drops the sequential-miss files' exported
types. Cache hits made exportedTypeMap.size > 0, suppressing the
end-of-loop buildExportedTypeMapFromGraph rebuild, but the sequential
path never populated the map. Regression test added (fails on the
pre-fix tree, passes after) plus a fully-sequential differential oracle.
- P2: saveParseCache builds its on-disk index from hashes actually
written/copied (writtenKeys), never a usedKeys hash whose shard write
or copy was skipped — no more phantom index entries.
- P2: add a unit test asserting SCOPE_RESOLUTION_LANGUAGES stays in sync
with SCOPE_RESOLVERS (asymmetric drift would lose a language's ParsedFile).
- Backfill cache coverage: loadParseCacheChunk missing/corrupt -> undefined,
pruneCache onDiskKeys branch, slim preserves nodes, saveParseCache
copy-evicted-shard round-trip.
- Cleanups: single-source heap-probe gating via isDebugHeapEnabled();
hoist the per-chunk mkdir in persistParseCacheChunk behind a
process-scoped Set; gate COBOL's unused worker-side ParsedFile
extraction (graph nodes still come from cobolPhase) while keeping
fileCount/progress unconditional.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ingestion): remove dead worker-side ParsedFile extraction
After #2038 gated worker `ParsedFile` emission behind `!isScopeResolutionLanguage(language)`, and with all 16 SupportedLanguages registered in SCOPE_RESOLVERS, that gate was structurally always true — the worker already produced no ParsedFiles and scope-resolution re-extracts each file from source on the main thread (run.ts). Remove the now-dead machinery:
- Drop both worker `extractParsedFile` call-sites (tree-sitter processFileGroup + the standalone-provider branch) and the `result.parsedFiles.push`. The standalone branch keeps fileCount/onFileProcessed per file. `result.parsedFiles` stays declared but empty (field removal deferred).
- Remove the now-orphaned `scopeSourceKind` var + `ScopeCaptureSourceKind`/`extractParsedFile`/`isScopeResolutionLanguage` imports.
- Delete the consumerless `migrated-languages.ts` (isScopeResolutionLanguage + SCOPE_RESOLUTION_LANGUAGES) and its drift-guard test — parse-worker was their only importer. Also improves AGENTS.md "shared ingestion code must not name languages" compliance.
`extractParsedFile` and the scope-extractor-bridge stay (scope-resolution/run.ts + Vue resolver use them). Behavior-preserving: worker-sequential-parity passes before and after; tsc/eslint clean; no baseline/golden drift.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ingestion): worker-pool-only parsing; remove sequential parser (#1983)
Completes the #1983 huge-repo parse-OOM effort by making the worker pool
GitNexus's sole parse path.
Parallel serialization (the perf core): workers serialize their ParsedFiles to
a disk store in parallel and stream them back to scope-resolution, so the main
thread no longer re-parses every file (the tree-sitter native-memory leak that
caused the OOM). Adds chunk merge-pipelining + work-proportional chunk sizing so
the pool stays saturated.
Remove the sequential parser: `--workers 0`, `GITNEXUS_WORKER_POOL_SIZE=0`, and
`skipWorkers` now hard-error (no silent degrade — #1741); the small-repo
threshold no longer selects an in-process path; pool creation stays lazy /
cache-miss-gated so warm all-hit runs never spawn workers.
Worker-path parity fixes — removing sequential surfaced two pre-existing gaps
that tiny-fixture tests had masked by running below the worker threshold, both
fixed by carrying per-file metadata as DATA across the worker boundary (never
re-parsing on the main thread, preserving the OOM fix):
- C++: templateConstraints wired into worker node identity (SFINAE overload
disambiguation) + ADL / inline-namespace capture side-channel serialized
onto the ParsedFile.
- Kotlin: companion-scope side-channel serialized the same way (companion /
static dispatch).
Validation: tsc + build clean; full suite green (10,190 pass — the only
deterministic failures were the now-fixed C++/Kotlin worker-path gaps; the 2
remaining full-run failures are pre-existing load flakiness, green in
isolation); cpp-pipeline benchmark stays linear on a 1-worker pool.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ingestion): wire C static-linkage side-channel + ADL O(1) collect + tri-review cleanups (#1983)
Follow-up to the worker-pool-only refactor, from a tri-review of the parse path.
- C static-linkage side-channel (P1): cProvider had no collect/applyCaptureSideChannel,
so on the now-sole worker path C `static` file-local marks were lost across the worker
boundary -> false cross-file CALLS edges + over-broad #include wildcard visibility on
every C analysis (the Linux kernel is C). Mirror the C++/Kotlin wiring: serialize
`staticNames` per file onto ParsedFile.captureSideChannel and restore it on the main
thread (no re-parse). + a worker-path regression test (the existing c-static-isolation
fixture passed vacuously — its collision resolves via #include before the global
free-call fallback ever consults static-linkage).
- captureSideChannel `kind` discriminant: add `kind:'cpp'`/`kind:'c'` tags + guards
(Kotlin already had one) now that C/C++/Kotlin share the single generic field.
- Perf: collectCppAdlSideChannel scanned the whole argInfoBySite/noAdlSites maps per file
(O(F^2) per sub-batch, ~100M parseSiteKey calls at kernel scale). Add per-filePath
lockstep indexes -> O(1) collect; serialized snapshot byte-identical.
- Cleanups: inline the one-line processParsingWithWorkers wrapper into processParsing;
drop the always-empty WorkerExtractedData.calls/assignments/constructorBindings fields;
remove the voided astCache param from processParsing; refresh stale "sequential
fallback" JSDoc.
Validation: tsc + build clean; cpp 297/297, c 8/8 (incl. the new worker-path
static-linkage guard), typescript + parsedfile-store green; cpp ADL benchmark stays linear.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(scope-resolution): index C/C++ #include resolution in finalize (O(n²)→O(n))
Kernel-scale C/C++ analysis ground in finalizeScopeModel because three
per-#include operations each did a full O(F) scan with no index — the
finalize O(n²) that surfaced once the #1983 parse-phase OOM was fixed:
- expand{C,Cpp}WildcardNames: parsedFiles.find() per wildcard edge → O(R·F)
- resolveImportTarget: new Set(allFilePaths) rebuilt per #include
- resolveCImportTarget: suffix-match scanned all workspace paths
Each is replaced with a WeakMap-per-pass index keyed on the stable
parsedFiles/allFilePaths references that scope-resolution run.ts passes
once per pass:
- Map<ScopeId,ParsedFile> for wildcard expansion (c/static-linkage.ts +
cpp/file-local-linkage.ts)
- memoized augmented header set (c/scope-resolver.ts + cpp/scope-resolver.ts)
- basename-bucketed suffix index in resolveCImportTarget (c/import-target.ts),
shared by C and C++ since resolveCppImportTarget delegates to it
Collapses the C/C++ finalize from O(R·F) to O(R+F). Pure-perf, byte-identical
edge output: 962 targeted tests green (490 C + 472 C/C++ scope-resolution);
the basename index preserves the exact endsWith('/'+target) match and the
fewest-path-components-then-lexicographic tie-break.
The kernel's ~25-30k .h headers are classified C++, so both providers must
be fixed. Proven on the Linux kernel: the C finalize completed
(sr-post-finalize lang=c → sr-end lang=c), which the pre-fix run never
reached in 16+ min of grinding.
Build-independent follow-ups (separate from this finalize fix), documented
for later: emitFreeCallFallback same-name buckets (emit phase),
buildGraphNodeLookup + precount global setup, the ParsedFile store-load,
the dart/go/ruby expand-wildcards .find siblings, and the ~26GB
scope-resolution memory floor (full kernel completion needs >~40GB RAM).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(bench): regenerate C scope-capture baseline for the #1983 c-static-linkage-worker fixture
bench/scope-capture/measure.mjs fingerprints emitCScopeCaptures over the
lang-resolution/c-* fixture corpus. The #1983 PR added the
c-static-linkage-worker fixture (caller.c/lib.c/lib.h/local.c — the
worker-path static-linkage side-channel test) but did not regenerate the C
baseline, so `--check` has been red on this branch (main, lacking the
fixture, still matches 0de009b).
Pure fixture-corpus drift — no c/captures.ts or query change branch-vs-main,
existing fixtures' captures byte-identical (c-captures.test.ts 45/45),
scaling stays linear (~0.97). Regenerated: 0de009b -> 39f3a83. Bench now
PASS (14 languages). Unrelated to the finalize O(n²) fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(scope-resolution): lower kernel-scale resident memory floor + setup cost
Reduce the scope-resolution resident-memory floor and setup throughput on
huge repos (Linux kernel), the wall that remains after #1983 (parse OOM) and
the finalize O(n^2) fix (
|
||
|
Gergő Magyar
|
05d269ec28
|
feat(ruby): migrate Ruby to scope-based resolution (RFC #909 Ring 3) (#1831)
* feat(ruby): migrate Ruby to scope-based resolution (RFC #909 Ring 3) Implement the full scope-resolution pipeline for Ruby following the PR #1639 (Rust migration) standard, targeting registration in MIGRATED_LANGUAGES with 100% scope parity. Scope resolver hooks (languages/ruby/): - query.ts: RUBY_SCOPE_QUERY covering scopes, declarations, imports, type-bindings (constructor inference via .new), and references - captures.ts: emitRubyScopeCaptures orchestrator with import decomposition, receiver-binding synthesis, method reclassification, and arity metadata for both declarations and calls - receiver-binding.ts: self type-binding synthesis for instance methods, singleton methods, and class << self blocks - interpret.ts: interpretRubyImport (wildcard semantics) and interpretRubyTypeBinding (YARD, constructor, alias sources) - import-target.ts: resolveRubyImportTarget adapting the existing suffix resolver for require/require_relative/load - merge-bindings.ts: tier-based shadowing (local > namespace > import) - arity.ts: Ruby arity check with *args/**kwargs/&block support - scope-resolver.ts: rubyScopeResolver with custom buildRubyMro (kind-aware IMPLEMENTS partitioning: prepend > direct > include; extend excluded from instance MRO per legacy semantics) - simple-hooks.ts: bindingScopeFor, importOwningScope, receiverBinding Wiring: - ruby.ts provider gains 7 scope-resolution hooks - Registered in SCOPE_RESOLVERS map and MIGRATED_LANGUAGES - 127 legacy tests wired with createResolverParityIt('ruby') - 27 new scope-specific tests in ruby-scope.test.ts Parity: 89/127 legacy tests pass under registry-primary; 38 are heritage/property/YARD gaps expected in V1. All 127 pass under legacy. Closes #931 * feat(ruby): add emitHeritageEdges hook, YARD parsing, bare calls, property emission Extend the scope-resolution pipeline with a new optional `emitHeritageEdges` hook (ScopeResolver contract + run.ts wiring) that runs between `preEmitInheritanceEdges` and `buildMro`. This lets languages whose heritage declarations are syntactic method calls (Ruby include/extend/prepend) emit IMPLEMENTS edges from the scope-resolver without touching the legacy pipeline. Ruby scope-resolution improvements: - Heritage: intercept include/extend/prepend in captures.ts, encode as special imports, emit IMPLEMENTS edges via emitHeritageEdges hook - Properties: intercept attr_accessor/attr_reader/attr_writer, emit Property nodes + HAS_PROPERTY edges via the same hook - Bare calls: add (body_statement (identifier)) capture to scope query, matching the legacy query pattern for zero-arity method calls - YARD parsing: second-pass comment scanner for @param/@return/@type annotations with findFollowingMethod that handles body_statement nesting - Query fixes: @declaration.trait for modules (was @declaration.module which normalizeNodeLabel didn't recognize), constant constructor bindings (SERVICE = UserService.new), call-return inference Parity: 114/127 legacy tests pass under registry-primary (up from 89). Remaining 13 are advanced type-inference chain resolution (compound receiver, cross-file return-type propagation, for-in element types). * feat(ruby): achieve 100% scope-resolution parity (127/127) Fix all 13 remaining type-inference failures: - Add expandsWildcardTo hook (expandRubyWildcardNames) so finalize can materialize individual bindings from require/require_relative wildcard imports, unblocking cross-file return-type propagation - Add member-call-return type binding synthesis in captures.ts for assignments like `x = obj.method()` — enables compound receiver chaining through member call return types - Add YARD @return support for attr_accessor/attr_reader/attr_writer calls, creating field-type bindings for chain resolution - Add @declaration.property captures alongside __property__ imports so properties register in localDefs → model.fields → write-access - Add constructor-return inference for methods ending with Foo.new() - Add for-loop variable type aliasing in scope query - Rebuild nodeLookup after emitHeritageEdges in run.ts so Property nodes created by the heritage hook are visible to downstream passes - Extend compound-receiver resolver to handle compound member-call rawNames with () and increase max depth from 4 to 8 - Extend receiver-bound-calls Case 3b for compound rawNames All 127 legacy Ruby tests pass under both REGISTRY_PRIMARY_RUBY=0 (legacy) and =1 (registry-primary). Ruby is now fully registered in MIGRATED_LANGUAGES with 100% scope parity. * test(ruby): add pipeline benchmark exercising heritage emission Synthetic Ruby codebases at 100/250/500 files with include + extend + prepend mixins, diamond mixin patterns (shared BaseMixin modules), attr_accessor properties, YARD annotations, and cross-file imports. Strict equality assertions verify exact IMPLEMENTS and HAS_PROPERTY edge counts: 4 IMPLEMENTS per class (include x2, extend, prepend) plus 1 per non-base mixin module, 3 HAS_PROPERTY per class. Dedup in emitRubyMixinEdges prevents double-counting when the worker path (repos >= 15 files) already created Property/IMPLEMENTS edges before scope-resolution runs. Scaling: 0.76x and 1.40x (both linear, well under 3x threshold). * ci: retrigger build * fix(ci): resolve format, registry-primary-flag, and sequential-mixin test failures - Run prettier on all changed files (captures.ts, run.ts, ruby-scope.test.ts, ruby.test.ts, ruby-pipeline-benchmark.test.ts) - Update registry-primary-flag.test.ts: use Swift (not in MIGRATED_LANGUAGES) instead of Ruby for the isolation and env-var mutation tests - Pin ruby-sequential-mixin.test.ts to REGISTRY_PRIMARY_RUBY=0 (legacy mode) since it tests inferImplicitReceiver + selectDispatch hooks that live in the legacy call-processor (gated off under registry-primary) --------- Co-authored-by: Test <test@example.com> |