GitNexus/gitnexus/test/integration/fastapi-composed-route-constants.test.ts
Gergő Magyar 5f4964b4e6
Some checks are pending
CodeQL / Analyze (javascript-typescript) (push) Waiting to run
CodeQL / Analyze (python) (push) Waiting to run
Gitleaks / gitleaks (push) Waiting to run
Publish / Classify release event (push) Waiting to run
Publish / RC guard (marker + release-PR skip) (push) Blocked by required conditions
Publish / ci (push) Blocked by required conditions
Publish / Publish to npm (push) Blocked by required conditions
Publish / Build & Push RC Docker images (push) Blocked by required conditions
Scorecard / Scorecard analysis (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-cli) (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-web) (push) Waiting to run
fix: resolve imported/composed FastAPI route path constants (#2391) (#2393)
* feat(routes): add pure Python string-constant resolver (#2391 U1)

* feat(routes): extract Python module constants from tree (#2391 U2)

* feat(routes): capture non-literal FastAPI decorator args + per-file constants, bump parse-cache schema (#2391 U3)

* feat(routes): resolve composed decorator route constants in parse-impl + skip floor (#2391 U4)

* feat(routes): resolve composed FastAPI route constants in group HTTP-contract layer (#2391 U5)

* test(routes): multi-hop, ingestion↔group parity, and warm-cache regression locks (#2391 U6)

* docs(routes): mark the language-agnostic seam for cross-language const resolution (#2391)

* refactor(routes): extract language-agnostic constant-fold core; Python becomes a binding (#2391)

The fold, cycle guard, and depth cap now live in constant-resolver.ts and take a
pluggable ImportResolver. python-const-resolver.ts supplies the Python import
semantics + tree extractor and re-exports the same surface, so no call site
changes. A Spring/Kotlin/C# binding can now reuse the core with its own resolver
(proven by constant-resolver.test.ts driving it with a Java-style resolver).

* fix(routes): treat the constant-fold cycle guard as a recursion stack (#2391)

The `visited` set in `foldName` was added-to but never removed on unwind, so a
constant referenced more than once in a single fold — `A + A`, a reused
separator (`SLASH + PATH + SLASH`), or a diamond `X = P + Q` where P and Q share
a base — tripped the cycle guard on its second occurrence and the whole route
was silently dropped by the skip floor. Pop the guard in `finally` so it tracks
the ACTIVE resolution stack, not every name ever seen: a true cycle (a name
still on the stack) is still caught, but a name that already resolved and popped
folds again. Re-computation stays bounded by MAX_RESOLVE_DEPTH, so no blowup is
reintroduced.

Locked in constant-resolver.test.ts (A+A, reused separator, shared-base
diamond); the pre-existing real-cycle and depth-cap cases still return null.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(routes): make module-constant binding writes mutually exclusive (#2391)

`extractPythonModuleConstants` kept `literals`, `exprs`, and `imports` as three
independent maps: `setName` cleared literals+exprs but never `imports`, and an
import never cleared a prior literal/expr. Since `foldName` checks
literals > exprs > imports regardless of source order, a name that was both
imported and locally (re)assigned kept both bindings and the wrong one won —
`from .c import ROUTE; ROUTE = os.getenv(...)` resolved the STALE import instead
of dropping, a confidently wrong route path (the exact skip-floor invariant
this feature is meant to uphold).

Treat the three maps as one logical namespace: any write to one clears the
other two for that name (via `imports.delete` in `setName` and a `bindImport`
helper), so last-binding-in-source-order wins, matching Python. An import both
imported and dynamically rebound now drops. Folding `+=`/`+` onto an imported
base remains deferred (it drops safely, never a stale value).

Locked in python-const-resolver.test.ts: dynamic-rebind drops, literal-shadows-
import, import-shadows-literal, and `+=`-on-import drops.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(routes): widen the group cost-gate to catch literal-leading concats (#2391)

`NONLITERAL_ROUTE_DECORATOR_RE` required the first decorator argument to START
with an identifier, so a string-literal-leading concat like
`@router.get("/api" + SUFFIX)` never tripped `hasComposedRoute`. When such a
route was the ONLY composed shape in a repo, the group layer left `constantsByFile`
empty and dropped the route, while the ingestion side (which has no gate)
resolved `/api/users` and emitted a Route node — an R4 provider/graph parity break.

Widen the gate to also fire on a string-literal-leading `+`-concat, detected by a
`+` before the closing paren on the decorator line. Gating on the `+` (not merely
a leading quote) keeps a plain literal route `@router.get("/x")` OFF the gate, so a
literal-only repo still pays no parse pass.

Locked in fastapi-composed-provider.test.ts: a sole literal-leading concat now
resolves (parseCalls>0 + provider emitted), plus previously-uncovered
`@app.<verb>(CONST)` EXPR-branch resolution; the literal-only no-parse gate case
still passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(routes): correct package-init and over-deep relative import resolution (#2391)

Two edges in `resolvePythonImport`:

- `from . import X` (empty module after the dots) resolved to a sibling
  `<dir>.py` instead of the package `<dir>/__init__.py`. Resolve the bare-package
  case to `__init__.py`.
- An over-deep relative import (more extra dots than the importing file has
  directory levels) silently clamped `dirOf('')` to `''` and could match an
  unrelated root-level `<name>.py` — a wrong file. Guard with `walk > depth →
  null` so an import that escapes above the repo root drops (skip floor).

Both preserve the exact-match / ambiguity→null behavior for ordinary relative and
absolute imports.

Locked in python-const-resolver.test.ts: `from . import` → `__init__.py` (and
null when absent), and an over-deep import returns null even when the clamped
target file exists.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(routes): bound parseConstOperands recursion depth (#2391)

`parseConstOperands` recursed on `binary_operator` children with no depth bound.
A stack overflow is not currently reachable (tree-sitter caps expression nesting
below the JS stack limit, so it throws on a deep `+`-chain before this runs), but
add a depth guard (cap 64, mirroring the fold engine's MAX_RESOLVE_DEPTH) as
defense-in-depth: a pathological chain now floors to null (skip) rather than
relying on tree-sitter's limit. The `depth` parameter defaults to 0, so all
existing callers are unaffected.

Locked in python-const-resolver.test.ts: a 100-term `+` chain yields no binding
(null) instead of throwing; ordinary short chains still fold.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(routes): read each .py once in buildPythonRepoContext (#2391)

The group repo-context builder read every `.py` file from disk twice: once in
the `include_router` cross-file pre-pass and again in the #2391 constant
cost-gate loop — an unconditional 2x read on every Python repo, on every group
extraction. Hoist a single read pass that populates one `pyContents` map (and
computes the composed-route cost gate); both the include_router pre-pass and the
constant-map pass now consume the cached content. Behavior-preserving — a
literal-only repo still does one read and zero parses.

Covered by the existing group unit + integration suites (R4 parity and
include_router prefix joins unchanged).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(routes): tidy constant-resolver docs and declaration order (#2391)

Three no-behavior nits from the PR #2393 review:
- Name `conditional_expression` (`x if c else y`) in the `parseConstOperands`
  jsdoc list of shapes that deferred to null.
- Move `NONLITERAL_ROUTE_DECORATOR_RE` above `buildPythonRepoContext`, which
  references it — it read as a forward reference before (runtime-safe, but
  confusing).
- Correct the integration-test comment that called `/v2/api/v1/widgets/get`
  "ingestion-only garnish": the group side emits it too (asserted separately);
  the four paths in that block are the shared-parity set.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(routes): fold `X += "…"` onto an imported base constant (#2391)

Previously `from .c import BASE; BASE += "/v1"` dropped (the extractor could not
represent "the imported prior value" as an operand without self-referencing X and
tripping the cycle guard). Preserve the imported prior under a synthetic `$imp$N`
key — `$` can never appear in a Python identifier, so it cannot collide with a
real name — and reference it, so the augmented assignment folds to
`<imported BASE>/v1`. Extractor-only: no change to the `Operand` type, the fold
core, or the cache shape, so no SCHEMA_BUMP. An imported base that is itself
unresolvable still drops (skip floor preserved — never a wrong path).

Locked in python-const-resolver.test.ts: single and chained `+=` fold onto an
imported base; an unresolvable base still drops.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(routes): resolve bare decorator constants via the by-name entry (#2391)

The group `resolveExprArg` hand-built `[{ kind: 'ref', name }]` and called
`resolveOperands` for a bare-constant decorator argument — exactly what the
language-agnostic core's `resolveConstant(file, name, repo)` seam does. Call it
directly for the identifier case. This gives the previously test-only by-name
entry point a real production caller (it is the documented reuse seam for future
JVM/other bindings), drops the synthetic operand construction, and lets the now-
unused `Operand` type import go. Behavior-identical — the `+`-concat path still
parses to an operand list and folds via `resolveOperands`.

Guarded by the existing group provider suite (bare-constant and concat cases).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* perf(routes): parse each .py once in buildPythonRepoContext (#2391)

The repo-context builder ran two parse loops — the include_router prefix pre-pass
and the #2391 constant-map pass — so an include_router file in a composed repo was
tree-sitter-parsed twice. Merge them into a single pass that parses each `.py` at
most once and feeds both extractions from the same tree; a file that needs neither
pass is still not parsed at all (cost gates unchanged). Complements the earlier
single-read-pass change (this is the single-parse counterpart).

Behavior-preserving (prefixes, R4 parity, and cost gates verified by the group +
integration suites). Locked with a parseCalls assertion: a file needing both
passes is parsed once, not twice.

Note: a cross-run (cross-process) constant-map cache — the other deferred perf
idea — remains out of scope; it needs disk persistence + invalidation and would
add hashing/IO cost on the common path, so it fails the minimal-change bar this
single-parse dedup meets.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(routes): bound constant-fold work and output to prevent OOM (#2391)

The `finally`-popped cycle guard (recursion-stack semantics) correctly folds
diamonds/repeated refs, but popping the guard removed the accidental work cap the
old seen-ever set provided: a wide shared-descendant DAG re-folds each child once
per reference, and a self-multiplying concat (`X = A + A; A = B + B; …`) builds a
genuinely exponential string. Reviewers reproduced ~16.8M folds escalating to
`RangeError: Invalid string length` and heap OOM — and neither fold call site is
wrapped in try/catch, so it crashed the whole phase rather than dropping the route.

Two complementary bounds, both flooring to null (skip), never a wrong value:
- a never-popped `memo` in `foldName` caps recomputation at O(nodes) (successes
  only — a null may be transient on a cyclic branch);
- a `MAX_FOLD_LENGTH` (8192) cap in `foldExpr` drops a fold whose output grows
  past any real route path, bounding the string size the depth cap does not.

Corrects the prior "≤ 2^8 folds" comment (output grows multiplicatively, not
additively). Locked with a 64^4-fanout construction that now drops in ~ms instead
of OOMing; diamonds/cycles/depth-cap behavior unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(routes): snapshot assignment RHS refs at the assignment line (#2391)

`ROUTE = BASE` was stored as a lazy `ref(BASE)`, resolved against BASE's FINAL
binding. So `ROUTE = BASE; BASE += "/v1"` (or `ROUTE = API; API = "/other"`)
resolved ROUTE to the MUTATED value — a confidently wrong path, since Python
assigns by value at the `ROUTE =` line. This was latent for local constants at
the base of this feature and the `+=`-on-import work extended it to imports.

Snapshot each assignment/`+=` RHS reference to a bound name into that name's
current frozen value at the assignment line (`freeze`/`snapshot`): a literal
value, a copy of the current expr (whose refs are already frozen), or an import
preserved under a `$imp$N` alias. Unbound refs (forward references) stay lazy.
A later rebind of the aliased name can no longer change the earlier binding.
`freeze` also unifies the previous `currentOps` + inline import-alias logic.

Locked in python-const-resolver.test.ts: aliased-import-then-`+=`,
aliased-local-then-`+=`, aliased-local-then-rebind all resolve to the pre-mutation
value; normal reference chains still fold.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(routes): fold group identifier args via resolveOperands for parity (#2391)

Resolving a bare-constant decorator arg through `resolveConstant` entered
`foldName` at depth 0, whereas the ingestion side folds `routePathOperands`
through `resolveOperands([{ref}])`, entering at depth 1. At the MAX_RESOLVE_DEPTH
boundary the group tolerated one more hop than ingestion, so a deep alias/re-export
chain resolved in the group provider set but dropped from the graph Route nodes —
an R4 parity break. Restore the operand-list path in the group so both subsystems
share identical fold-entry depth. (`resolveConstant` reverts to the documented
agnostic-core seam.)

Locked in constant-resolver.test.ts: a 4-hop chain that `resolveOperands([ref])`
drops but `resolveConstant` resolves, documenting why the group must use the
operand-list entry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(routes): match multiline literal-leading concats in the cost gate (#2391)

`NONLITERAL_ROUTE_DECORATOR_RE` used `[^)\n]*` so it only saw a literal-leading
`+`-concat when the `+` was on the same line as the opening quote. A
Black-formatted `@router.get(\n "/api"\n + SUFFIX\n)` therefore failed the gate,
and when it was the only composed route in a repo the group dropped it while
ingestion (which parses the tree, not the raw line) resolved it — an R4 parity
break. Drop the `\n` exclusion: `[^)]*` spans the wrapped argument but stays
bounded by the decorator's own closing paren, so a plain literal route still
never trips the gate.

Locked in fastapi-composed-provider.test.ts with a multiline concat fixture.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(routes): bump SCHEMA_BUMP for changed extractor output + E2E snapshot lock (#2391)

`extractPythonModuleConstants` now emits DIFFERENT `moduleConstants` for the same
source (binding mutual-exclusivity clears stale imports; RHS refs are snapshotted;
`$imp$N` aliases). That output is cached verbatim in the parse cache, so a warm
shard built at the pre-fix version would replay stale — in one case actively
wrong — folded values, and the correctness fixes would silently no-op on upgrade.
Bump SCHEMA_BUMP 11→12 to force re-extraction (same warm-cache-replay class the
original 10→11 bump addressed for the field addition).

Also adds the first end-to-end coverage for the new behavior through the real
ingestion pipeline: app/snapshot.py aliases a constant (`SNAP = API_V1`) then
mutates the source (`API_V1 += "/mutated"`), and the test asserts the Route node
is `/api/v1`, never `/api/v1/mutated` — a case the pure-function unit tests
covered but the pipeline did not.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 13:23:05 +01:00

200 lines
8.4 KiB
TypeScript

/**
* End-to-end coverage of imported/composed FastAPI route path constants (#2391).
*
* `@router.post(API_V1_WIDGETS_GET)` — where the path is an imported constant
* built by `+`-concatenation in another module — must index as
* `POST /api/v1/widgets/get` in the ingestion `Route` graph nodes (which drive
* `route_map` / `api_impact`), NOT as `POST /`. An argument that cannot be folded
* to a literal is skipped entirely (KTD5 floor), never recorded as `/`.
*
* The group HTTP-contract parity, multi-hop chains, the module-collision floor,
* and the warm-cache guard are added by U5/U6 (see the sibling describe blocks
* and `http-route-extractor.test.ts`).
*
* Fixture: `test/fixtures/fastapi-composed-app/`.
*/
import { describe, it, expect, beforeAll } from 'vitest';
import path from 'node:path';
import * as fs from 'node:fs';
import * as os from 'node:os';
import Parser from 'tree-sitter';
import Python from 'tree-sitter-python';
import { runPipelineFromRepo } from '../../src/core/ingestion/pipeline.js';
import type { PipelineResult } from '../../types/pipeline.js';
import { PYTHON_HTTP_PLUGIN } from '../../src/core/group/extractors/http-patterns/python.js';
import {
loadParseCache,
saveParseCache,
PARSE_CACHE_VERSION,
} from '../../src/storage/parse-cache.js';
const FIXTURE = path.resolve(__dirname, '..', 'fixtures', 'fastapi-composed-app');
describe('FastAPI composed route constants — ingestion pipeline (#2391)', () => {
let result: PipelineResult;
beforeAll(async () => {
result = await runPipelineFromRepo(FIXTURE, () => {}, {});
}, 60_000);
function routes(): { method: string | undefined; url: string }[] {
const out: { method: string | undefined; url: string }[] = [];
result.graph.forEachNode((n) => {
if (n.label !== 'Route') return;
const method = n.properties.method;
out.push({
method: method === undefined ? undefined : String(method),
url: String(n.properties.name),
});
});
return out;
}
const urls = (): string[] =>
routes()
.map((r) => r.url)
.sort();
it('resolves the imported composed constant to its full path', () => {
expect(routes()).toContainEqual({ method: 'POST', url: '/api/v1/widgets/get' });
});
it('never records a phantom `/` for a non-literal path', () => {
expect(urls()).not.toContain('/');
});
it('leaves an ordinary string-literal sibling route unchanged', () => {
expect(routes()).toContainEqual({ method: 'GET', url: '/literal/health' });
});
it('skips an unresolvable constant argument (no Route node, not `/`)', () => {
// `@router.delete(UNKNOWN_ROUTE_CONST)` — the constant is defined nowhere, so
// it folds to null and the route is dropped rather than indexed as `DELETE /`.
expect(routes().some((r) => r.method === 'DELETE')).toBe(false);
});
it('joins an APIRouter(prefix=…) with a resolved composed path', () => {
// prefixed.py: `router = APIRouter(prefix="/v2")` + `@router.post(COMPOSED)`.
expect(routes()).toContainEqual({ method: 'POST', url: '/v2/api/v1/widgets/get' });
});
it('keeps two composed routes at distinct paths as distinct nodes', () => {
const composed = routes().filter((r) => r.url.endsWith('/api/v1/widgets/get'));
expect(composed.map((r) => r.url).sort()).toEqual([
'/api/v1/widgets/get',
'/v2/api/v1/widgets/get',
]);
});
it('resolves a multi-hop import chain (leaf → mid → base) with an inline concat', () => {
// deep/base.py ROOT=/root → deep/mid.py MID=ROOT+"/mid" → deep/leaf.py
// @router.get(MID + "/leaf").
expect(routes()).toContainEqual({ method: 'GET', url: '/root/mid/leaf' });
});
it('resolves same-named constants in different packages against their OWN package', () => {
// pkg_a/constants.py SHARED="/a-shared" and pkg_b/constants.py SHARED="/b-shared",
// each imported via `from .constants import SHARED`. Never crossed (KTD4).
expect(routes()).toContainEqual({ method: 'GET', url: '/a-shared' });
expect(routes()).toContainEqual({ method: 'GET', url: '/b-shared' });
expect(urls().filter((u) => u.endsWith('-shared'))).toEqual(['/a-shared', '/b-shared']);
});
it('snapshots an aliased constant before a later mutation, end-to-end (#2393)', () => {
// app/snapshot.py: `SNAP = API_V1` (captures "/api/v1") then `API_V1 += "/mutated"`.
// SNAP's route must be the pre-mutation value, never the mutated one.
expect(routes()).toContainEqual({ method: 'GET', url: '/api/v1' });
expect(urls()).not.toContain('/api/v1/mutated');
});
});
// ─── R4 parity: the group HTTP-contract layer resolves the same paths ─────────
describe('FastAPI composed route constants — ingestion↔group parity (#2391 R4)', () => {
it('group provider paths match the ingestion Route-node paths for composed routes', async () => {
const ingestion = await runPipelineFromRepo(FIXTURE, () => {}, {});
const ingestionUrls = new Set<string>();
ingestion.graph.forEachNode((n) => {
if (n.label === 'Route') ingestionUrls.add(String(n.properties.name));
});
// Run the group plugin over the same fixture files.
const files: Record<string, string> = {};
const walk = (dir: string, rel: string): void => {
for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
const abs = path.join(dir, entry.name);
const r = rel ? `${rel}/${entry.name}` : entry.name;
if (entry.isDirectory()) walk(abs, r);
else if (entry.name.endsWith('.py')) files[r] = fs.readFileSync(abs, 'utf8');
}
};
walk(FIXTURE, '');
const parser = new Parser();
const parseSource = (p: Parser, src: string): Parser.Tree => {
p.setLanguage(Python);
return p.parse(src);
};
const ctx = PYTHON_HTTP_PLUGIN.prepareRepo?.({
files: Object.keys(files),
parser,
readFile: (r) => files[r] ?? null,
parseSource,
});
const groupPaths = new Set<string>();
for (const [rel, src] of Object.entries(files)) {
for (const d of PYTHON_HTTP_PLUGIN.scan(parseSource(parser, src), ctx, rel)) {
if (d.role === 'provider') groupPaths.add(d.path);
}
}
// Every composed route the ingestion side resolved is also a group provider
// path, and vice versa — the two subsystems agree (R4), including the
// multi-hop and per-package-collision cases. (The `/v2` APIRouter(prefix)
// route is emitted by BOTH sides as well — asserted separately above; the
// four paths below are this block's shared-parity set.)
for (const composed of ['/api/v1/widgets/get', '/root/mid/leaf', '/a-shared', '/b-shared']) {
expect(ingestionUrls.has(composed)).toBe(true);
expect(groupPaths.has(composed)).toBe(true);
}
// Neither side invents a phantom `/` for the unresolvable DELETE route.
expect(ingestionUrls.has('/')).toBe(false);
expect(groupPaths.has('/')).toBe(false);
}, 60_000);
});
// ─── Warm parse-cache: composed routes survive the cache serialization ────────
describe('FastAPI composed route constants — warm parse-cache (#2391 SCHEMA_BUMP)', () => {
it('re-resolves the composed route on an all-hit warm run after a save/load round-trip', async () => {
const storageDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gitnexus-composed-warm-'));
try {
// Run #1 populates the parse cache.
const cold = {
version: PARSE_CACHE_VERSION,
entries: new Map(),
usedKeys: new Set<string>(),
};
await runPipelineFromRepo(FIXTURE, () => {}, { parseCache: cold });
// Force the JSON round-trip (mapReplacer/mapReviver) the real warm path uses
// — this is where the new `moduleConstants` Maps and `routePathExpr` fields
// must survive, or a warm re-analyze silently drops the composed route.
await saveParseCache(storageDir, cold);
const warm = await loadParseCache(storageDir);
expect(warm).not.toBeNull();
const result = await runPipelineFromRepo(FIXTURE, () => {}, {
parseCache: warm ?? undefined,
});
const urls = new Set<string>();
result.graph.forEachNode((n) => {
if (n.label === 'Route') urls.add(String(n.properties.name));
});
expect(urls.has('/api/v1/widgets/get')).toBe(true);
expect(urls.has('/root/mid/leaf')).toBe(true);
expect(urls.has('/')).toBe(false);
} finally {
fs.rmSync(storageDir, { recursive: true, force: true });
}
}, 120_000);
});