mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-08-28 05:25:25 +00:00
2 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
3f5fbb05e0
|
feat(group+ingestion): resolve Java constant-based route paths (@PostMapping(ApiPathConstants.X)) (#2980)
* feat(group): resolve Java constant-based route paths via repo constant map - prepareRepo builds repo-wide Java constant map (constant-definition files only, cheap regex gate; per-file try/catch so one bad file degrades not forfeits) - bind parser language in prepareRepo (orchestrator hands over a bare Parser) - scan() lazily overlays the importing file's own import table (extracted from the tree already in hand, zero extra parses) before folding operands - foldJavaOperands resolves qualified refs (Class.CONST) + static imports + string concatenation against the merged view; unresolved refs are skipped, never guessed Real-repo validation (winning-winex-opt, 23k Java files): providers 2 -> 1701 (1700 source_scan_resolved), cross-links 0 -> 589 exact Unit: 14/14 (java-route-const-resolver.test.ts) * fix(review): address bot review findings on PR #2980 - P2-1 (real): spring.ts route loop dropped every @value_expr match — the '!valueNode' guard ran before the operand branch, so ingestion emitted zero constant-referencing routes. Guard now accepts @value_expr when @value is absent; two downstream valueNode dereferences made conditional. Added 2 extractor-level regression tests (16 total). - P2-2 (real): collectSpringTypes copied rawPath:'' for constant routes into the shared Spring inheritance view — now skipped there (fold happens in scan(); empty-path noise would leak into inheritance-based providers). - P1-1 (false positive): Java 'static final' allows exactly one initializer (duplicate declarations are compile errors), so the Python-style rebinding shadowing cleanup does not apply — documented at the site. - P1-2 (false positive): constant-resolver.ts and prepareDurableParsedFileChunk both exist on upstream main (#2391 / parsedfile-store.ts:562); the bot's 'repository lookup' appears to have compared against a stale index. - P3: removed dead FQN_CONTROLLER fixture. Real-repo regression: 589 cross-links / 2423 contracts (was 2424 — the dropped contract is the empty-path inheritance artifact fixed above). * docs(cache): note Java constant-route capture set in the SCHEMA_BUMP ledger The Java constant-route harvest (route-extractors/java-const-resolver.ts + the spring.ts operand branch + the parse-worker Java constant harvest) changes the worker capture set: a warm pre-feature cache replays moduleConstants=0 captures verbatim and silently drops every constant-based Spring route on unchanged files. After rebasing onto current main the ledger already sits at 70, whose capture set post-dates and includes this harvest, so v70 invalidates those caches — no additional bump is needed. * fix(feign): guard @RequestLine against the constant-valued shape A constant-valued `@RequestLine(SOME_CONST)` is captured as @value_expr, not @value, so `valueNode` is undefined in that shape and the literal dereference crashed the scan. Skip instead — folding verb+path literals through the constant map is out of scope for this PR. Found in maintainer review of #2980. * fix(resolver): bound qualified-ref recursion depth for self/mutual import cycles Maintainer review point: the qualified branch of resolveJavaConstant recurses through resolveJavaImport without a guard — a self-import (X = SelfConsts.X + ...) or a pair of mutually-importing constants would recurse without bound before reaching the shared fold's visited-stack, which only guards the bare-name path. Bound the Java-qualified walk with a depth cap (32) and thread it through every recursive call. Two regression tests use real repo shapes (repoOf fixtures): self-import and mutual-import cycles both terminate with null (skip floor), as before, but promptly. Also drops the stray machine-local .gitignore entry that rode along from the fork's dev branch. * fix(routes): address round-2 review — provider hooks, FQN fold, interface nesting F1 (High): production harvest silently dropped routes when the constants class is not named *Constants (e.g. ApiPaths). The content gate is now SYNTAX-driven (static-final String field or any class import) and lives in the provider (moduleConstantHeuristic), not a shared-layer regex. F2: shared ingestion layers no longer branch on language. The harvest and the qualified-ref fold run through new provider hooks (extractModuleConstants / foldRoutePathOperands); parse-impl resolves the provider by filePath (getProviderForFile). Python wires the same hooks for architecture parity. F3: multi-segment FQN chains (com.example.ApiPaths.USERS) now flatten recursively; verified via tree-sitter that the existing query already captures the whole nested field_access — the gap was resolver-side only. F4: implicit-final interface semantics no longer leak into nested classes at type boundaries (JLS 9.5). F5: nested same-name shadowing now drops the stale entry (rebind-drop, matching Python #2391 semantics) instead of keeping the first binding. Tests: 9 new unit tests (27/27) + real-pipeline e2e over a reviewer-shaped fixture (non-*Constants class, cold run + warm parse-cache replay) — the exact production gap unit tests missed. * style: prettier --write on the two touched test files (CI format gate) * fix(routes): address the open review findings on Java constant route folding Answers every reproduced finding still open on #2980, plus the defects an adversarial pass found in the first round of those fixes. The wrong-path group each turned a *missing* fact into a *wrong* one, which is what this module's skip-or-correct contract exists to prevent. Wrong-path fixes * Escapes were deleted from constant values. tree-sitter-java splits a `string_literal` around its `escape_sequence` children, so joining `string_fragment`s alone folded `"/user/{id:\\d+}"` — the standard Spring path-variable constraint — to `/user/{id:d+}`, and a pure-escape literal to the empty string. Worse, the LITERAL path keeps escapes verbatim, so one Java route had two irreconcilable spellings. `stringLiteralValue` now reuses `unquoteSpringLiteral`, the helper that literal path already uses. Java text blocks are excluded: that helper's `"""` arm would hand back the raw block, newline and incidental indentation included, so they keep the old skip. * A constant-valued class prefix produced a truncated route. The new `@value_expr` query branches were `method_declaration`-only, so `@RequestMapping(ApiPaths.BASE)` left the prefix empty and the method route was emitted unprefixed — a path the application does not serve, where the base emitted nothing at all. Both subsystems now detect such a class and suppress its method routes, the rule `classesWithArrayPrefix` already encodes for the array form. The suppression covers ingestion's separate no-argument-mapping loop too, without which a bare `@GetMapping` under a constant prefix still shipped an empty-path Route while the group emitted nothing. * A shadowed static import survived a non-foldable rebind. The rebind-drop deleted `literals`/`exprs` but not `imports`, so a name both static-imported and locally redeclared resolved through the stale import to the imported value instead of skipping (#2393's Python defect, reproduced for Java). * `resolveJavaImport` guessed where its own docstring promised null. The nearest-shared-directory tie-break is gone: javac resolves duplicate FQNs by classpath order, so proximity can return a src/test fixture copy. Parity and coverage fixes * One constant-file gate, exported as `isJavaConstantFile` and used by both the ingestion provider and the group `prepareRepo` pre-pass. The two spellings disagreed on a constant INTERFACE — implicitly `public static final`, so it carries neither keyword — which the group admitted and ingestion rejected, so the group published a contract while the graph got no Route node. It is also modifier-order agnostic now, and its interface arm requires a String assignment so a javadoc mentioning "interface" no longer costs a parse. * Import ambiguity is measured over constant-DEFINING files on both sides. Ingestion's harvest gate also admits import-only files, so handing `resolveJavaImport` every repo key let a duplicate FQN that defines nothing make ingestion alone floor to skip — reopening the same parity break in the same losing direction. * Python's constant harvest is unconditional again. The gate added here required NAME immediately followed by `=`, so it dropped `API: str = "/api"`, `API: Final[str] = "/api"` and every composed constant whose RHS starts with an identifier — routes that already resolve on main. The worker now treats a missing heuristic as "harvest" rather than "skip". * Enum and record declarations were traversed but never collected, so a `static final String` declared in one was absent from the map. The walk still descends the whole body, so a type nested in an enum-constant body is kept. * Constants composed across files through a qualified ref never resolved: operands found inside an initializer went to the agnostic core, which only knows bare names, so `X = BConsts.Y + "/tail"` floored to null even acyclically. The Java binding now folds its own expressions — and carries the core's guards with them: a `visited` stack popped on unwind, a memo of successes, and `MAX_FOLD_LENGTH`. Without the memo a shared-descendant DAG re-folds each child per reference; because a chain of empty strings never accumulates output, the length cap could not stop it, and one route over a 31-line constants file took 11 s at 28 levels on the main thread. * Dropped the dead `com.java.lang.` type normalization. Cache * `SCHEMA_BUMP` 70 -> 72. Leaving it at 70 was justified by "the ledger already sits at 70, whose capture set post-dates and includes this harvest" — it does not: 70 was cut by |
||
|
|
5f4964b4e6
|
fix: resolve imported/composed FastAPI route path constants (#2391) (#2393)
Some checks are pending
CodeQL / Analyze (javascript-typescript) (push) Waiting to run
CodeQL / Analyze (python) (push) Waiting to run
Gitleaks / gitleaks (push) Waiting to run
Publish / Classify release event (push) Waiting to run
Publish / RC guard (marker + release-PR skip) (push) Blocked by required conditions
Publish / ci (push) Blocked by required conditions
Publish / Publish to npm (push) Blocked by required conditions
Publish / Build & Push RC Docker images (push) Blocked by required conditions
Scorecard / Scorecard analysis (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-cli) (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-web) (push) Waiting to run
* feat(routes): add pure Python string-constant resolver (#2391 U1) * feat(routes): extract Python module constants from tree (#2391 U2) * feat(routes): capture non-literal FastAPI decorator args + per-file constants, bump parse-cache schema (#2391 U3) * feat(routes): resolve composed decorator route constants in parse-impl + skip floor (#2391 U4) * feat(routes): resolve composed FastAPI route constants in group HTTP-contract layer (#2391 U5) * test(routes): multi-hop, ingestion↔group parity, and warm-cache regression locks (#2391 U6) * docs(routes): mark the language-agnostic seam for cross-language const resolution (#2391) * refactor(routes): extract language-agnostic constant-fold core; Python becomes a binding (#2391) The fold, cycle guard, and depth cap now live in constant-resolver.ts and take a pluggable ImportResolver. python-const-resolver.ts supplies the Python import semantics + tree extractor and re-exports the same surface, so no call site changes. A Spring/Kotlin/C# binding can now reuse the core with its own resolver (proven by constant-resolver.test.ts driving it with a Java-style resolver). * fix(routes): treat the constant-fold cycle guard as a recursion stack (#2391) The `visited` set in `foldName` was added-to but never removed on unwind, so a constant referenced more than once in a single fold — `A + A`, a reused separator (`SLASH + PATH + SLASH`), or a diamond `X = P + Q` where P and Q share a base — tripped the cycle guard on its second occurrence and the whole route was silently dropped by the skip floor. Pop the guard in `finally` so it tracks the ACTIVE resolution stack, not every name ever seen: a true cycle (a name still on the stack) is still caught, but a name that already resolved and popped folds again. Re-computation stays bounded by MAX_RESOLVE_DEPTH, so no blowup is reintroduced. Locked in constant-resolver.test.ts (A+A, reused separator, shared-base diamond); the pre-existing real-cycle and depth-cap cases still return null. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(routes): make module-constant binding writes mutually exclusive (#2391) `extractPythonModuleConstants` kept `literals`, `exprs`, and `imports` as three independent maps: `setName` cleared literals+exprs but never `imports`, and an import never cleared a prior literal/expr. Since `foldName` checks literals > exprs > imports regardless of source order, a name that was both imported and locally (re)assigned kept both bindings and the wrong one won — `from .c import ROUTE; ROUTE = os.getenv(...)` resolved the STALE import instead of dropping, a confidently wrong route path (the exact skip-floor invariant this feature is meant to uphold). Treat the three maps as one logical namespace: any write to one clears the other two for that name (via `imports.delete` in `setName` and a `bindImport` helper), so last-binding-in-source-order wins, matching Python. An import both imported and dynamically rebound now drops. Folding `+=`/`+` onto an imported base remains deferred (it drops safely, never a stale value). Locked in python-const-resolver.test.ts: dynamic-rebind drops, literal-shadows- import, import-shadows-literal, and `+=`-on-import drops. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(routes): widen the group cost-gate to catch literal-leading concats (#2391) `NONLITERAL_ROUTE_DECORATOR_RE` required the first decorator argument to START with an identifier, so a string-literal-leading concat like `@router.get("/api" + SUFFIX)` never tripped `hasComposedRoute`. When such a route was the ONLY composed shape in a repo, the group layer left `constantsByFile` empty and dropped the route, while the ingestion side (which has no gate) resolved `/api/users` and emitted a Route node — an R4 provider/graph parity break. Widen the gate to also fire on a string-literal-leading `+`-concat, detected by a `+` before the closing paren on the decorator line. Gating on the `+` (not merely a leading quote) keeps a plain literal route `@router.get("/x")` OFF the gate, so a literal-only repo still pays no parse pass. Locked in fastapi-composed-provider.test.ts: a sole literal-leading concat now resolves (parseCalls>0 + provider emitted), plus previously-uncovered `@app.<verb>(CONST)` EXPR-branch resolution; the literal-only no-parse gate case still passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(routes): correct package-init and over-deep relative import resolution (#2391) Two edges in `resolvePythonImport`: - `from . import X` (empty module after the dots) resolved to a sibling `<dir>.py` instead of the package `<dir>/__init__.py`. Resolve the bare-package case to `__init__.py`. - An over-deep relative import (more extra dots than the importing file has directory levels) silently clamped `dirOf('')` to `''` and could match an unrelated root-level `<name>.py` — a wrong file. Guard with `walk > depth → null` so an import that escapes above the repo root drops (skip floor). Both preserve the exact-match / ambiguity→null behavior for ordinary relative and absolute imports. Locked in python-const-resolver.test.ts: `from . import` → `__init__.py` (and null when absent), and an over-deep import returns null even when the clamped target file exists. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(routes): bound parseConstOperands recursion depth (#2391) `parseConstOperands` recursed on `binary_operator` children with no depth bound. A stack overflow is not currently reachable (tree-sitter caps expression nesting below the JS stack limit, so it throws on a deep `+`-chain before this runs), but add a depth guard (cap 64, mirroring the fold engine's MAX_RESOLVE_DEPTH) as defense-in-depth: a pathological chain now floors to null (skip) rather than relying on tree-sitter's limit. The `depth` parameter defaults to 0, so all existing callers are unaffected. Locked in python-const-resolver.test.ts: a 100-term `+` chain yields no binding (null) instead of throwing; ordinary short chains still fold. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(routes): read each .py once in buildPythonRepoContext (#2391) The group repo-context builder read every `.py` file from disk twice: once in the `include_router` cross-file pre-pass and again in the #2391 constant cost-gate loop — an unconditional 2x read on every Python repo, on every group extraction. Hoist a single read pass that populates one `pyContents` map (and computes the composed-route cost gate); both the include_router pre-pass and the constant-map pass now consume the cached content. Behavior-preserving — a literal-only repo still does one read and zero parses. Covered by the existing group unit + integration suites (R4 parity and include_router prefix joins unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(routes): tidy constant-resolver docs and declaration order (#2391) Three no-behavior nits from the PR #2393 review: - Name `conditional_expression` (`x if c else y`) in the `parseConstOperands` jsdoc list of shapes that deferred to null. - Move `NONLITERAL_ROUTE_DECORATOR_RE` above `buildPythonRepoContext`, which references it — it read as a forward reference before (runtime-safe, but confusing). - Correct the integration-test comment that called `/v2/api/v1/widgets/get` "ingestion-only garnish": the group side emits it too (asserted separately); the four paths in that block are the shared-parity set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(routes): fold `X += "…"` onto an imported base constant (#2391) Previously `from .c import BASE; BASE += "/v1"` dropped (the extractor could not represent "the imported prior value" as an operand without self-referencing X and tripping the cycle guard). Preserve the imported prior under a synthetic `$imp$N` key — `$` can never appear in a Python identifier, so it cannot collide with a real name — and reference it, so the augmented assignment folds to `<imported BASE>/v1`. Extractor-only: no change to the `Operand` type, the fold core, or the cache shape, so no SCHEMA_BUMP. An imported base that is itself unresolvable still drops (skip floor preserved — never a wrong path). Locked in python-const-resolver.test.ts: single and chained `+=` fold onto an imported base; an unresolvable base still drops. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(routes): resolve bare decorator constants via the by-name entry (#2391) The group `resolveExprArg` hand-built `[{ kind: 'ref', name }]` and called `resolveOperands` for a bare-constant decorator argument — exactly what the language-agnostic core's `resolveConstant(file, name, repo)` seam does. Call it directly for the identifier case. This gives the previously test-only by-name entry point a real production caller (it is the documented reuse seam for future JVM/other bindings), drops the synthetic operand construction, and lets the now- unused `Operand` type import go. Behavior-identical — the `+`-concat path still parses to an operand list and folds via `resolveOperands`. Guarded by the existing group provider suite (bare-constant and concat cases). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(routes): parse each .py once in buildPythonRepoContext (#2391) The repo-context builder ran two parse loops — the include_router prefix pre-pass and the #2391 constant-map pass — so an include_router file in a composed repo was tree-sitter-parsed twice. Merge them into a single pass that parses each `.py` at most once and feeds both extractions from the same tree; a file that needs neither pass is still not parsed at all (cost gates unchanged). Complements the earlier single-read-pass change (this is the single-parse counterpart). Behavior-preserving (prefixes, R4 parity, and cost gates verified by the group + integration suites). Locked with a parseCalls assertion: a file needing both passes is parsed once, not twice. Note: a cross-run (cross-process) constant-map cache — the other deferred perf idea — remains out of scope; it needs disk persistence + invalidation and would add hashing/IO cost on the common path, so it fails the minimal-change bar this single-parse dedup meets. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(routes): bound constant-fold work and output to prevent OOM (#2391) The `finally`-popped cycle guard (recursion-stack semantics) correctly folds diamonds/repeated refs, but popping the guard removed the accidental work cap the old seen-ever set provided: a wide shared-descendant DAG re-folds each child once per reference, and a self-multiplying concat (`X = A + A; A = B + B; …`) builds a genuinely exponential string. Reviewers reproduced ~16.8M folds escalating to `RangeError: Invalid string length` and heap OOM — and neither fold call site is wrapped in try/catch, so it crashed the whole phase rather than dropping the route. Two complementary bounds, both flooring to null (skip), never a wrong value: - a never-popped `memo` in `foldName` caps recomputation at O(nodes) (successes only — a null may be transient on a cyclic branch); - a `MAX_FOLD_LENGTH` (8192) cap in `foldExpr` drops a fold whose output grows past any real route path, bounding the string size the depth cap does not. Corrects the prior "≤ 2^8 folds" comment (output grows multiplicatively, not additively). Locked with a 64^4-fanout construction that now drops in ~ms instead of OOMing; diamonds/cycles/depth-cap behavior unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(routes): snapshot assignment RHS refs at the assignment line (#2391) `ROUTE = BASE` was stored as a lazy `ref(BASE)`, resolved against BASE's FINAL binding. So `ROUTE = BASE; BASE += "/v1"` (or `ROUTE = API; API = "/other"`) resolved ROUTE to the MUTATED value — a confidently wrong path, since Python assigns by value at the `ROUTE =` line. This was latent for local constants at the base of this feature and the `+=`-on-import work extended it to imports. Snapshot each assignment/`+=` RHS reference to a bound name into that name's current frozen value at the assignment line (`freeze`/`snapshot`): a literal value, a copy of the current expr (whose refs are already frozen), or an import preserved under a `$imp$N` alias. Unbound refs (forward references) stay lazy. A later rebind of the aliased name can no longer change the earlier binding. `freeze` also unifies the previous `currentOps` + inline import-alias logic. Locked in python-const-resolver.test.ts: aliased-import-then-`+=`, aliased-local-then-`+=`, aliased-local-then-rebind all resolve to the pre-mutation value; normal reference chains still fold. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(routes): fold group identifier args via resolveOperands for parity (#2391) Resolving a bare-constant decorator arg through `resolveConstant` entered `foldName` at depth 0, whereas the ingestion side folds `routePathOperands` through `resolveOperands([{ref}])`, entering at depth 1. At the MAX_RESOLVE_DEPTH boundary the group tolerated one more hop than ingestion, so a deep alias/re-export chain resolved in the group provider set but dropped from the graph Route nodes — an R4 parity break. Restore the operand-list path in the group so both subsystems share identical fold-entry depth. (`resolveConstant` reverts to the documented agnostic-core seam.) Locked in constant-resolver.test.ts: a 4-hop chain that `resolveOperands([ref])` drops but `resolveConstant` resolves, documenting why the group must use the operand-list entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(routes): match multiline literal-leading concats in the cost gate (#2391) `NONLITERAL_ROUTE_DECORATOR_RE` used `[^)\n]*` so it only saw a literal-leading `+`-concat when the `+` was on the same line as the opening quote. A Black-formatted `@router.get(\n "/api"\n + SUFFIX\n)` therefore failed the gate, and when it was the only composed route in a repo the group dropped it while ingestion (which parses the tree, not the raw line) resolved it — an R4 parity break. Drop the `\n` exclusion: `[^)]*` spans the wrapped argument but stays bounded by the decorator's own closing paren, so a plain literal route still never trips the gate. Locked in fastapi-composed-provider.test.ts with a multiline concat fixture. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(routes): bump SCHEMA_BUMP for changed extractor output + E2E snapshot lock (#2391) `extractPythonModuleConstants` now emits DIFFERENT `moduleConstants` for the same source (binding mutual-exclusivity clears stale imports; RHS refs are snapshotted; `$imp$N` aliases). That output is cached verbatim in the parse cache, so a warm shard built at the pre-fix version would replay stale — in one case actively wrong — folded values, and the correctness fixes would silently no-op on upgrade. Bump SCHEMA_BUMP 11→12 to force re-extraction (same warm-cache-replay class the original 10→11 bump addressed for the field addition). Also adds the first end-to-end coverage for the new behavior through the real ingestion pipeline: app/snapshot.py aliases a constant (`SNAP = API_V1`) then mutates the source (`API_V1 += "/mutated"`), and the test asserts the Route node is `/api/v1`, never `/api/v1/mutated` — a case the pure-function unit tests covered but the pipeline did not. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |