docs+fix(group): address remaining Claude review items + add pipeline flow chart

## Fixes

**Remaining 🔴 — HTTP contract id wildcard format.** Documented the
`http::*::<path>` format as an intentional wildcard for manifest links
that omit the HTTP method, alongside the explicit-method form
(`GET::/path` → `http::GET::/path`). The docblock on `buildContractId`
now states both forms, notes that wildcard-aware matching is the
responsibility of the sync / cross-impact layer (#793), and
recommends the explicit-method form whenever the author knows the
method (it round-trips through exact equality without needing
wildcard logic downstream). Tests unchanged — the wildcard format is
what they've always asserted.

**Minor 1 — stale comment at `manifest-extractor.ts:124-126`.** The
comment claimed "creates a contract with an empty symbolUid/ref" but
the code switched to `manifestSymbolUid(repo, contractId)` a few
commits back. Updated to describe the actual synthetic-uid fallback
semantics and the cross-impact path that relies on both sides of the
join deriving the same uid.

**Minor 2 — exhaustiveness guard on `buildContractId`.** The
`switch(type)` covered all five current `ContractType` variants but
silently returned `undefined` if a new variant was added. Added a
`default: const _exhaustive: never = type; throw new Error(...)`
clause so the build fails loudly on an unhandled variant.

**Minor 3 — `tree.rootNode.text` in `grpc-patterns/node.ts`.** Already
fixed in `2f28bfc` via a dedicated structural query
(`LOAD_PACKAGE_DEFINITION_SPEC`). No action needed.

## New: pipeline flow chart (per @magyargergo's request)

Added `src/core/group/PIPELINE.md` with four Mermaid diagrams:
1. **High-level overview** — `group.yaml` → extractors + manifest →
   contract matching → `bridge.lbug` → `runGroupImpact`.
2. **Per-repo extractor two-strategy shape** — graph-assisted
   Strategy A vs. source-scan Strategy B.
3. **Plugin architecture** — orchestrator → registry →
   per-language `*-patterns/<lang>.ts` → `tree-sitter-scanner.ts` →
   `ExtractedContract`.
4. **Manifest extraction** — label-scoped `resolveSymbol` with the
   synthetic-uid fallback.
5. **Cross-impact query (#606)** — local impact → bridge join →
   cross-repo fan-out.

Each diagram is annotated with which PRs own which stage (this PR:
extractors + manifest; #795: bridge storage; #606: cross-impact
runtime) and points at the concrete files/functions involved.

## Tests

- 99/99 extractor tests pass
- `npx tsc -p tsconfig.json --noEmit` clean

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
ivkond 2026-04-13 07:36:15 +00:00
parent 7b22776560
commit 7016b32af2
No known key found for this signature in database
2 changed files with 172 additions and 3 deletions

View file

@ -0,0 +1,139 @@
# Group Analysis Pipeline
Flow chart of the cross-repo contract extraction + matching pipeline.
This covers what runs **inside this PR** (extractors + manifest) and
the downstream handoff to the bridge storage (PR #795) and
cross-impact query (PR #606).
## High-level overview
```mermaid
flowchart TD
A[group.yaml] --> B[GroupConfig parser]
B --> C{For each repo<br/>in group}
C --> D[Per-repo LadybugDB<br/>indexed by main pipeline]
D --> E1[TopicExtractor]
D --> E2[HttpRouteExtractor]
D --> E3[GrpcExtractor]
E1 --> F[ExtractedContract array<br/>per repo]
E2 --> F
E3 --> F
B --> M[ManifestExtractor]
M --> G[Manifest contracts<br/>+ cross-links]
F --> H[Contract matching<br/>exact + wildcard]
G --> H
H --> I[(bridge.lbug<br/>#795)]
I --> J[runGroupImpact<br/>#606]
J --> K[CrossRepoImpact]
```
## Per-repo extractor pipeline
Each extractor under `src/core/group/extractors/` follows the same
two-strategy shape:
```mermaid
flowchart TD
R[RepoHandle + CypherExecutor<br/>for this repo] --> S{Graph-assisted<br/>Strategy A<br/>available?}
S -->|yes| A1[Cypher query against<br/>per-repo LadybugDB]
A1 --> A2{non-empty<br/>result?}
A2 -->|yes| OUT[ExtractedContract array]
A2 -->|no| B1
S -->|no| B1[Source-scan Strategy B]
B1 --> B2[glob repo source files]
B2 --> B3{ext in registry?}
B3 -->|yes| B4[Per-language plugin<br/>scan parsed tree]
B3 -->|no| SKIP[skip file]
B4 --> OUT
SKIP --> B2
```
**Strategy A** (graph-assisted) uses Cypher over edges already produced
by the main ingestion pipeline:
- HTTP: `HANDLES_ROUTE` / `FETCHES` edges from `(File)-[]->(Route)`
- topic: none (pipeline doesn't yet produce topic nodes — Strategy B only)
- gRPC: none (Strategy B + proto map only)
**Strategy B** (source-scan) is 100% tree-sitter based after this PR.
Each `*-patterns/<lang>.ts` plugin owns its grammar + S-expression
queries; the top-level orchestrator imports neither.
## Plugin architecture
```mermaid
flowchart LR
O[Orchestrator<br/>topic|http|grpc-extractor.ts] --> REG[REGISTRY<br/>*-patterns/index.ts]
REG --> P1[java.ts<br/>tree-sitter-java]
REG --> P2[go.ts<br/>tree-sitter-go]
REG --> P3[python.ts<br/>tree-sitter-python]
REG --> P4[node.ts<br/>JS + TS + TSX]
REG --> P5[php.ts<br/>tree-sitter-php<br/>HTTP only]
REG --> P6[proto.ts<br/>tree-sitter-proto<br/>gRPC only, optional]
P1 --> SCAN[tree-sitter-scanner.ts<br/>compilePatterns + runCompiledPatterns]
P2 --> SCAN
P3 --> SCAN
P4 --> SCAN
P5 --> SCAN
P6 --> SCAN
SCAN --> DET[Detection objects<br/>TopicMeta / HttpDetection / GrpcDetection]
DET --> O
O --> CT[ExtractedContract array]
```
The orchestrator never imports a grammar. Adding a new language /
framework = drop one file in `*-patterns/`, register it in
`index.ts`. No orchestrator edits required.
## Manifest extraction
```mermaid
flowchart TD
Y[group.yaml links] --> ME[ManifestExtractor]
ME --> LOOP{for each link}
LOOP --> RES[resolveSymbol<br/>label-scoped Cypher]
RES --> OK{found?}
OK -->|yes| REF[real symbol uid + ref]
OK -->|no| SYN[synthetic uid<br/>manifest::repo::cid]
REF --> EMIT[emit provider + consumer<br/>Contract objects<br/>+ CrossLink]
SYN --> EMIT
EMIT --> BRIDGE[(bridge.lbug<br/>#795)]
```
Label-scoped queries in `resolveSymbol` keep accidental cross-matches
out:
- `topic``(n:Function|Method|Class|Interface)`
- `grpc` method → `(n:Function|Method)`, service → `(n:Class|Interface)`
- `lib``(n:Package|Module)`
## Cross-impact query (PR #606)
```mermaid
flowchart TD
U[User changes symbol S<br/>in repo R] --> LI[Local impact engine<br/>per-repo uid expansion]
LI --> IDS[Affected uid set]
IDS --> BR[Bridge query<br/>MATCH Contract WHERE uid IN ids]
BR --> CL[CrossLink traversal]
CL --> OTHER[Matching contract in<br/>other repo]
OTHER --> FE[Fan-out impact<br/>to consuming repo]
FE --> OUT[CrossRepoImpact<br/>per affected repo]
```
The bridge stores every extracted contract keyed by `symbolUid`.
Manifest-sourced contracts use the synthetic uid form so both sides
of the `(local impact) ↔ (bridge query)` join derive the same uid
without coordinating through any shared state.

View file

@ -121,9 +121,12 @@ export class ManifestExtractor {
// match "/suborders", and a gRPC manifest entry in a repo with any
// .proto file would attach to a random proto symbol.
//
// If resolveSymbol returns null, the extractor creates a contract with
// an empty symbolUid/ref — cross-impact still works via name-based
// matching through the `hint` path in runGroupImpact.
// If resolveSymbol returns null, the extractor falls back to a
// deterministic synthetic uid via `manifestSymbolUid(repo, contractId)`
// (see the function's docstring for why synthetic rather than empty).
// Cross-impact still works: the bridge query joins on the synthetic
// uid, and the local impact engine derives the same uid for the
// unresolved symbol — name-based hints are the additional safety net.
try {
let rows: Record<string, unknown>[];
if (link.type === 'http') {
@ -219,6 +222,29 @@ export class ManifestExtractor {
return null;
}
/**
* Build a canonical contract id for a manifest link.
*
* HTTP is the only type with two valid forms:
* - Explicit method: `"GET::/api/orders"` `"http::GET::/api/orders"`
* (matches exactly against `HttpRouteExtractor` provider/consumer
* contracts, which are also keyed by `http::<METHOD>::<path>`).
* - Method-agnostic: `"/api/orders"` `"http::*::/api/orders"`
* the `*` is a wildcard and is intended to match any concrete
* HTTP method on that path. Wildcard-aware matching is the
* responsibility of the sync / cross-impact layer (see #793);
* downstream code should treat `http::*::<path>` as matching
* every `http::<METHOD>::<path>` for the same path.
*
* Recommend the explicit-method form in group.yaml whenever the
* manifest author knows the method it round-trips through exact
* equality matching without requiring wildcard logic downstream.
*
* NOTE on exhaustiveness: the switch covers every current
* `ContractType` variant and falls through to a `never` assertion so
* TypeScript fails the build if a new variant is added without a
* corresponding case.
*/
private buildContractId(type: ContractType, contract: string): string {
switch (type) {
case 'http': {
@ -233,6 +259,10 @@ export class ManifestExtractor {
return `lib::${contract}`;
case 'custom':
return `custom::${contract}`;
default: {
const _exhaustive: never = type;
throw new Error(`Unhandled ContractType: ${String(_exhaustive)}`);
}
}
}
}