scriptoriumadmin/GitNexus
1
0
Fork 0
mirror of https://github.com/abhigyanpatwari/GitNexus.git synced 2026-09-14 23:22:54 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 7

feat(group): extractor expansion + manifest extractor (2/4 of #606 split) (#796)

Browse source 
* feat(group): extractor expansion + manifest extractor

Part 2 of 4 in the split of #606 (ticket: #792). Follows #795
(bridge.lbug storage foundation, already merged), but this PR has no
code-level dependency on #795 — it only imports types and the
ContractExtractor interface that existed on upstream main before
either PR. It could have been reviewed in parallel with #795.

## What changed

Expands the 3 existing contract extractors with substantially more
language/framework coverage, and adds a new `manifest-extractor`
that resolves `group.yaml`-declared cross-links against the per-repo
graph via exact-name lookups.

### New file (228 LOC)

- `gitnexus/src/core/group/extractors/manifest-extractor.ts` —
  exact graph lookup for `group.yaml`-declared cross-links. HTTP
  paths are canonicalized before Route.name matching; gRPC is
  resolved by service/method name (NO `.proto`-filename fallback);
  topic and lib use exact-name match. Falls back to a synthetic
  `manifest::<repo>::<contractId>` uid when the graph has no
  matching symbol, so cross-impact traversal still has a stable
  anchor for the contract.

### Modified extractors (+958 LOC prod)

- `extractors/grpc-extractor.ts` (+522) — `.proto` parser with
  comment and string-literal sanitization (braces inside strings no
  longer truncate service bodies); package/service/method canonical
  IDs; server/client detection across Go (`grpc.NewServer`,
  `RegisterXxxServer`, `XxxGrpc.XxxImplBase`), Java (`@GrpcService`,
  `BlockingStub`), Python (`servicer_to_server`, `XxxStub`), and
  TypeScript/Node (`@GrpcMethod`, `ClientGrpc`, `loadPackageDefinition`).
- `extractors/http-route-extractor.ts` (+174) — Go gin/echo/stdlib
  `HandleFunc`, NestJS `@Controller`+`@Get`/etc, Python FastAPI
  decorators, Java Spring `@RequestMapping`/`@GetMapping`,
  restTemplate / WebClient / OkHttp consumers.
- `extractors/topic-extractor.ts` (+98) — sarama `ProducerMessage{}`
  struct literal detection (replaces a constructor-anchored regex
  that missed topics inside producer loops), kafka-go Writer/Reader,
  Python NATS (`await nc.subscribe`/`await nc.publish`), JetStream
  helpers.

### Modified and new tests (+1264 LOC)

- `grpc-extractor.test.ts` (+539) — full coverage of the new proto
  parser (strings-with-braces regression, comments-with-braces
  regression), per-language server/client detection
- `http-route-extractor.test.ts` (+240) — per-framework route
  extraction + normalization edge cases
- `topic-extractor.test.ts` (+177) — the sarama in-loop regression,
  JetStream, Python NATS, kafka-go Writer/Reader
- `manifest-extractor.test.ts` (+308 NEW) — HTTP path normalization,
  gRPC exact lookup with proto-fallback regression, lib and topic
  exact matching, synthetic-uid fallback behavior

### Self-review fixes folded in

Carried forward from the #606 self-review (commit `d15b8cb`):

- **HIGH #1** — `manifest-extractor.resolveSymbol` was too fuzzy.
  Previously used `CONTAINS` on route/name fields plus an
  unconditional `filePath ENDS WITH '.proto'` fallback for gRPC.
  Consequences: `/orders` matched `/suborders`, and any repo with
  any `.proto` file returned a random proto symbol for a gRPC
  manifest entry. Replaced with exact equality + deterministic
  `ORDER BY` + synthetic-uid fallback for unresolved manifests.
  Regression tests included.
- **MED #3** — gRPC proto parser brace-depth counting now sanitizes
  strings and comments first (`stripProtoCommentsAndStrings`). A
  valid proto with `option deprecated_reason = "use NewService {
  instead"` used to have its service body closed early by the `"{"`
  inside the literal, silently dropping methods after the offending
  string. Regression tests for both string-with-brace and
  comment-with-brace cases.
- **MED #4** — sarama Kafka regex changed from
  `sarama.NewSyncProducer[\s\S]{0,300}?Topic:` (anchored on
  constructor, caught only first topic in a loop) to
  `sarama.ProducerMessage{...Topic:}` (matches every struct literal
  directly). Regression test with a for-loop that constructs
  multiple `ProducerMessage`s.
- **MED #7** — `manifest-extractor.resolveSymbol` no longer has a
  silent `catch { /* fall through */ }`. Errors from the graph
  executor are logged via `console.warn` with link type, contract
  name, repo key, and error message before falling through to the
  synthetic-uid path.

## Why

Reviewer focus here is pure regex / parser correctness — no
storage, no Cypher queries, no algorithmic changes to the cross-link
algorithm. Separating this from the bridge foundation PR (#795)
meant reviewers could stay in a single mental mode (parsing logic)
instead of context-switching between DDL, Cypher, and regex.

## How to verify

- `cd gitnexus && npx tsc --noEmit`
- `cd gitnexus && npx vitest run test/unit/group/grpc-extractor.test.ts --pool=forks`
- `cd gitnexus && npx vitest run test/unit/group/http-route-extractor.test.ts --pool=forks`
- `cd gitnexus && npx vitest run test/unit/group/topic-extractor.test.ts --pool=forks`
- `cd gitnexus && npx vitest run test/unit/group/manifest-extractor.test.ts --pool=forks`

Local pre-push: typecheck clean, all 99 extractor unit tests pass
(grpc 43, http 18, topic 30, manifest 8).

## Risk / rollback

**Low.** Extractors have no user-facing surface in this PR — they
produce `ExtractedContract[]` that is consumed by `sync.ts` in the
next split (#793). No existing behavior changes for users who don't
run a `group sync`. Rollback = `git revert` of the merge commit;
the modifications to `grpc-extractor.ts` / `http-route-extractor.ts`
/ `topic-extractor.ts` revert to the pre-PR versions that still
work (they're subsets of the new functionality).

## Scope discipline (per GUARDRAILS.md)

- Only the 8 files above are touched; no drive-by refactors
- No CI/release/security config changes
- No secrets or machine-specific paths
- Content lifted from #606 (CI 11/11 green on `d15b8cb`)

## Dependencies

- **Base:** `main` (upstream already includes #795 as `1ff324c`)
- **Blocks:** sync pipeline (#793) and the cross-impact feature (#794)
- **Tracker issue:** #792
- **Parent PR:** #606

Co-authored-by: Claude <noreply@anthropic.com>

* refactor(group): migrate topic-extractor from regex to tree-sitter queries

Addresses @magyargergo's feedback on #796 that regex-based lookups
should use tree-sitter nodes instead, and that the top-level
extractors must NOT carry language dependencies. This is phase 1 of
a multi-step migration — topic-extractor first because its patterns
are the most uniform (16 "call/annotation with first-arg string
literal" variants), which makes it a clean proof of the approach
before grpc-extractor and http-route-extractor get the same treatment.

## Architecture: language-agnostic orchestrator + per-language plugins

The top-level extractor is a thin orchestrator that never imports a
tree-sitter grammar or a query string. Per-language knowledge lives
in a new `topic-patterns/` folder with one file per language plus a
registry that maps file extensions to compiled plugins:

```
src/core/group/extractors/
├── tree-sitter-scanner.ts         # shared, language-agnostic scanning utilities
├── topic-extractor.ts              # thin orchestrator (no grammar imports)
└── topic-patterns/
    ├── types.ts                    # TopicMeta, Broker
    ├── index.ts                    # registry: extension → compiled provider
    ├── java.ts                     # tree-sitter-java + JAVA_TOPIC_PROVIDER
    ├── go.ts                       # tree-sitter-go + GO_TOPIC_PROVIDER
    ├── python.ts                   # tree-sitter-python + PYTHON_TOPIC_PROVIDER
    └── node.ts                     # tree-sitter-javascript + tree-sitter-typescript
                                    # → JAVASCRIPT_/TYPESCRIPT_/TSX_TOPIC_PROVIDER
```

**Shared scanner (`tree-sitter-scanner.ts`)** — defines
`PatternSpec<TMeta>`, `LanguagePatterns<TMeta>`, `CompiledPatterns<TMeta>`
and the `scanFile(parser, plugin, content)` helper. Plugins compile their
queries eagerly at module load via `compilePatterns()`, so a broken
pattern fails loudly at import time instead of silently at scan time.
`unquoteLiteral()` handles single/double/template quotes, Python
triple-quoted strings, and Go raw backtick strings.

**Per-language plugins** own:
- the tree-sitter grammar import (this is the ONLY place in
  `src/core/group/` where tree-sitter grammars are imported),
- the query S-expressions,
- the `TopicMeta` payload (role, broker, confidence, symbolName) that
  the orchestrator receives back on every match.

Each plugin uses a `@value` capture name to bind the topic literal node.
The JavaScript and TypeScript grammars share AST node names for every
construct we query, so `node.ts` defines the pattern sources once and
compiles them against `JavaScript`, `TypeScript.typescript`, and
`TypeScript.tsx` — exporting three providers because `Parser.Query`
objects are NOT portable across grammar instances.

**Registry (`topic-patterns/index.ts`)** — maps `.java` → Java provider,
`.go` → Go, `.py` → Python, `.js`/`.jsx` → JS, `.ts` → TS, `.tsx` → TSX.
Also exports `TOPIC_SCAN_GLOB` so adding a new language is a single
file-level edit (drop `topic-patterns/<lang>.ts`, import + register it
here — zero edits required in `topic-extractor.ts`).

**Orchestrator (`topic-extractor.ts`)** — ~110 lines, no grammar or
query imports. Per file: `getProviderForFile(rel)` → `scanFile(parser,
provider, content)` → `unquoteLiteral(valueText)` → `makeContract(...)`.
Reuses one `Parser` instance across files; the scanner calls
`setLanguage` per plugin.

## Why this is better than regex

1. **Comments and strings are respected for free.** The old regex
   would match `// kafkaTemplate.send("fake.topic")` as a real
   producer; tree-sitter never visits comments or string literals as
   code nodes, so false positives from commented-out code are
   eliminated.
2. **Struct/object literal patterns are structural, not textual.**
   `sarama.ProducerMessage{Topic: "..."}` no longer needs a 300-char
   lookahead (which was a known cross-match bug partly mitigated by a
   loop regression test in the self-review). The new query matches a
   specific `composite_literal` with a specific `qualified_type` and
   `keyed_element` — exactly one struct literal per match.
3. **No order-of-operations fragility.** Regex for
   `channel.publish` vs `channel.consume` was independent and
   file-wide; the AST scopes matches to the specific `call_expression`.
4. **Language-agnostic extension.** Adding Ruby, Rust, or C# topic
   detection later means dropping one file in `topic-patterns/` — no
   changes to shared scanner or orchestrator, and no tree-sitter
   imports leak into top-level code.

## Per-file fault tolerance

- Malformed files that tree-sitter can't parse are silently skipped
  (`parser.parse` is wrapped by `scanFile`). The ingestion pipeline
  already logs unparseable files at index time.
- A syntactically invalid query is caught at `compilePatterns` time,
  not scan time — broken plugins fail loudly at import.
- Per-pattern `matches()` failures are swallowed so one broken query
  in a plugin doesn't block the rest.

## Tests

All 30 existing `topic-extractor.test.ts` tests pass **without any
changes to the test file** — they were written as input/output contract
tests (given this source file, expect these `ExtractedContract` objects)
and that contract is unchanged. Regression coverage includes:

- Kafka: Java `@KafkaListener` + `kafkaTemplate.send`; Node
  `producer.send` + `consumer.subscribe`; Go sarama producer/consumer
  (sync and async); kafka-go Writer/Reader; Python `KafkaConsumer` +
  `producer.send/produce`
- RabbitMQ: Java `@RabbitListener` + `rabbitTemplate.convertAndSend`;
  Node `channel.consume/publish/sendToQueue`; Python `basic_consume/
  basic_publish` with keyword args
- NATS: Go and Node `nc.Subscribe/Publish`; Go and Node JetStream
  `js.Subscribe/Publish`; Python `await nc.subscribe/publish`

Including the regression test for the sarama `ProducerMessage`
in-loop case — the AST-based query captures every literal in the
file independently, not just the first one after `NewSyncProducer`.

## Neighbor regression check

- `topic-extractor.test.ts` — 30/30 pass (rewritten extractor)
- `http-route-extractor.test.ts` — 18/18 pass (untouched)
- `grpc-extractor.test.ts` — 43/43 pass (untouched)
- `manifest-extractor.test.ts` — 8/8 pass (untouched)
- Full `npx tsc --noEmit` clean

## Scope discipline (per GUARDRAILS.md)

- Only files under `src/core/group/extractors/` are touched; no
  changes to other extractors, tests, MCP surface, or pipeline.ts.
- No CI/release/security config changes, no secrets.
- New tree-sitter imports all reference grammars that are already
  installed as dependencies (`tree-sitter`, `tree-sitter-javascript`,
  `tree-sitter-typescript`, `tree-sitter-python`, `tree-sitter-java`,
  `tree-sitter-go` — all in `package.json` for the existing pipeline).

## Phase 2 / phase 3 plan

- **Phase 2 (next commit):** rewrite `http-route-extractor.ts`
  Strategy B (regex fallback) on the same plugin pattern. Graph-assisted
  Strategy A stays as-is (already uses pipeline-built tree-sitter data
  via `HANDLES_ROUTE` Cypher queries).
- **Phase 3 (commit after):** rewrite `grpc-extractor.ts` for Java /
  Go / Python / TypeScript detection. `.proto` files are the one
  outstanding question — there is no `tree-sitter-proto` grammar
  installed; the in-tree string-sanitizing parser stays as a pragmatic
  exception with a comment, alternative being to add
  `tree-sitter-proto` as a dep (open for the maintainer).

Co-authored-by: Claude <noreply@anthropic.com>

* refactor(group): migrate http-route-extractor Strategy B to tree-sitter plugins

Phase 2 of the extractor refactor requested by @magyargergo on #796.
Same architecture as the phase 1 topic-extractor rewrite: a thin,
language-agnostic orchestrator plus per-language plugins that own
tree-sitter grammars and query sources. The top-level extractor file
no longer imports any tree-sitter grammar or query string.

## Architecture

```
src/core/group/extractors/
├── tree-sitter-scanner.ts          # shared, language-agnostic primitives
├── http-route-extractor.ts         # thin orchestrator (no grammar imports)
└── http-patterns/
    ├── types.ts                    # HttpDetection, HttpLanguagePlugin, HttpRole
    ├── index.ts                    # registry: ext → plugin + HTTP_SCAN_GLOB
    ├── java.ts                     # tree-sitter-java: Spring + RestTemplate/WebClient/OkHttp
    ├── go.ts                       # tree-sitter-go: gin/echo/HandleFunc + http/resty consumers
    ├── python.ts                   # tree-sitter-python: FastAPI + requests
    ├── php.ts                      # tree-sitter-php: Laravel Route::get/...
    └── node.ts                     # tree-sitter-javascript + tree-sitter-typescript:
                                    #   NestJS controllers, Express, fetch, axios
```

**Shared scanner (`tree-sitter-scanner.ts`)** — generalised from phase 1:
- `ScanMatch<TMeta>.captures` is now a full `CaptureMap` (every named
  capture the query binds, not just a single `@value`). Topic extractor
  updated to read `match.captures.value` accordingly.
- New `runCompiledPatterns(plugin, tree)` helper lets plugins run
  multiple query bundles against the same pre-parsed tree. This is
  needed for HTTP plugins that combine a class-prefix query with a
  method-route query (Spring, NestJS).
- `scanFile` becomes a thin wrapper over `parser.parse + runCompiledPatterns`.

**HTTP plugin shape** — unlike topic plugins, HTTP plugins expose a
`scan(tree)` function rather than a flat pattern list. This reflects
HTTP's more complex extraction: each detection needs method + path +
handler name, and framework patterns like Spring `@RequestMapping` /
NestJS `@Controller` require cross-referencing a class-level prefix
with method-level annotations. Plugins internally use
`compilePatterns` + `runCompiledPatterns` and walk the AST to resolve
the class/method relationships.

**Per-framework coverage:**

- **Java (`java.ts`)**
  - Spring: `@RequestMapping("/api/v2")` class prefix + `@(Get|Post|Put|
    Delete|Patch)Mapping("/sub")` method routes, joined via the
    enclosing `class_declaration` node id.
  - `RestTemplate.getForObject/postForEntity/put/delete/patchForObject` →
    method derived from API name.
  - `WebClient.method(HttpMethod.X, "/path")` → method from
    `HttpMethod.X` capture.
  - `new Request.Builder().url("/path")` → OkHttp consumer.

- **Go (`go.ts`)**
  - gin / echo / chi frameworks: `\w+.GET("/path", handler)` captures
    upper-case verb + handler identifier.
  - `net/http.HandleFunc("/path", handler)` → provider (default GET).
  - `http.Get/Post/Head` consumer, `http.NewRequest("METHOD", ...)`,
    resty `client.R().Get/Post/...`.

- **Python (`python.ts`)**
  - `@app.get("/path")` FastAPI decorators.
  - `requests.get/post/...` and `requests.request("METHOD", "url")`.

- **PHP (`php.ts`)**
  - Laravel `Route::get/post/.../patch('/path', ...)` via
    `scoped_call_expression`. Uses `PHP.php_only` to match the
    existing ingestion pipeline's grammar selection.

- **Node (`node.ts`) — JS + TS + TSX**
  - Pattern sources defined once, compiled against three grammar
    variants (`JavaScript`, `TypeScript.typescript`, `TypeScript.tsx`)
    because `Parser.Query` objects are not portable across grammars.
    Exports three plugins sharing the same `scan` logic.
  - NestJS: `@Controller('prefix')` decorators are siblings of the
    class in `export_statement` / `program`; `@Get(':id')` decorators
    are siblings of the method in `class_body`. The plugin walks
    decorator → next named sibling to find the decorated class /
    method, then combines the class prefix with the method path.
    Only emits NestJS detections when the enclosing class has a real
    `@Controller` decorator — prevents false positives from generic
    classes that happen to use `@Get` from another library.
  - Express: `(router|app).<verb>('/path', ...)`.
  - `fetch(url)` (default GET) + `fetch(url, { method: 'X' })`
    (uses two queries + a SyntaxNode-id dedupe set so URL literals
    aren't double-emitted by the options variant).
  - `axios.get/post/...`.

## Orchestrator changes

`http-route-extractor.ts` drops every `scanXxxProviders` / `scanXxxConsumers`
regex method and replaces them with a single source-scan loop that
delegates to `getPluginForFile(rel).scan(tree)`. The orchestrator
still owns:

- **Path normalization** (`normalizeHttpPath`, `normalizeConsumerPath`)
  — language-agnostic string processing shared by both strategies.
- **Graph-assisted Strategy A** (`HANDLES_ROUTE` / `FETCHES` / `CONTAINS`
  Cypher queries) — unchanged in spirit. The only regex helpers it
  used (`inferMethodFromFileScan`, `pickJavaHandlerName`) are now
  replaced by a lookup against the plugin's detections for the same
  file: for each route row, find the detection whose normalized path
  matches, and pull the HTTP method + handler name from it.
- **Per-file parse cache** — the orchestrator parses each relevant
  file at most once per `extract()` call. Both the graph-assisted
  enrichment loop and the source-scan fallback share the same
  `cachedDetections` map, so we never run the plugin twice for the
  same file.

## Why this is better than the regex version

1. **Comments and strings for free.** The old regex would match
   `// router.get('/fake')` as a real Express route; tree-sitter
   never visits string/comment nodes.
2. **Structural controller-prefix.** Spring and NestJS class-prefix
   joining is now scoped to the enclosing class via `class_declaration`
   node ids, eliminating file-wide state that broke when a file had
   multiple controllers.
3. **Precise NestJS disambiguation.** The plugin only emits a NestJS
   detection when the enclosing class has a real `@Controller`
   decorator — the old regex would fire on any `@Get(...)` in the
   file regardless of surrounding context.
4. **Language-agnostic extension.** Adding Ruby / Rust / Kotlin HTTP
   detection later means dropping one file in `http-patterns/` — no
   changes to the shared scanner, the orchestrator, or the Strategy A
   Cypher queries.

## Tests

- `http-route-extractor.test.ts` — **18/18 pass** (tests unchanged;
  they're contract-style input/output tests and the contract shape is
  unchanged). Covers Spring class prefix, Express, gin/echo, stdlib
  HandleFunc, NestJS, Laravel, FastAPI for providers and
  fetch/axios/python-requests/rest-template/webClient/okhttp/go-stdlib/
  resty for consumers, plus graph-first Strategy A for both.
- `topic-extractor.test.ts` — **30/30 pass** after the `captures.value`
  API migration.
- `grpc-extractor.test.ts` — 43/43 pass (untouched; phase 3).
- `manifest-extractor.test.ts` — 8/8 pass (untouched).
- `service.test.ts`, `sync.test.ts`, `storage.test.ts` — 41/41 pass.
- `npx tsc -p tsconfig.json --noEmit` clean.

## Scope discipline (per GUARDRAILS.md)

- Only files under `src/core/group/extractors/` are touched.
- No changes to pipeline.ts, MCP surface, ingestion, or tests.
- No CI / release / security / secrets changes.
- Tree-sitter grammars imported by plugins (`tree-sitter-java`,
  `tree-sitter-go`, `tree-sitter-python`, `tree-sitter-php`,
  `tree-sitter-javascript`, `tree-sitter-typescript`) are all already
  in `package.json` for the existing ingestion pipeline.

## Phase 3 plan

- **grpc-extractor** gets the same treatment: plugin-per-language under
  `grpc-patterns/` for Java / Go / Python / TS detection. `.proto`
  files remain an open question — no `tree-sitter-proto` grammar is
  installed, so the in-tree string-sanitizing parser from PR #796's
  self-review stays as a pragmatic exception unless the maintainer
  wants us to add `tree-sitter-proto` as a new dep.

Co-authored-by: Claude <noreply@anthropic.com>

* refactor(group): migrate grpc-extractor source scans to tree-sitter plugins

Phase 3 (final) of the extractor refactor requested by @magyargergo on
#796. Same architecture as phase 1 (topic) and phase 2 (http): thin
language-agnostic orchestrator + per-language plugins that own
tree-sitter grammars and query sources. With this commit the top-level
extractors under `src/core/group/extractors/` import ZERO tree-sitter
grammars and ZERO query strings — every grammar import lives in a
`*-patterns/<lang>.ts` plugin file, and the orchestrators go through
the registry indirection.

## Architecture

```
src/core/group/extractors/
├── tree-sitter-scanner.ts         # shared primitives (unchanged)
├── grpc-extractor.ts               # orchestrator (only `.proto` parser left)
└── grpc-patterns/
    ├── types.ts                    # GrpcDetection, GrpcLanguagePlugin, GrpcRole
    ├── index.ts                    # registry: ext → plugin + GRPC_SCAN_GLOB
    ├── go.ts                       # tree-sitter-go: RegisterXxxServer, Unimplemented, NewXxxClient
    ├── java.ts                     # tree-sitter-java: @GrpcService + XxxImplBase + newBlockingStub
    ├── python.ts                   # tree-sitter-python: add_XxxServicer_to_server + XxxStub
    └── node.ts                     # tree-sitter-javascript + tree-sitter-typescript:
                                    #   @GrpcMethod, @GrpcClient field type,
                                    #   .getService<X>('Svc'), new XxxServiceClient,
                                    #   loadPackageDefinition dynamic constructors
```

## Per-language coverage

**Go (`go.ts`)**
- Provider: `\w+.RegisterXxxServer(...)` via `call_expression →
  selector_expression → field_identifier` + JS regex filter
  `^Register(\w+)Server$`.
- Provider: `pb.UnimplementedXxxServer` embedded in a struct via
  `struct_type → field_declaration_list → field_declaration →
  qualified_type → type_identifier` + JS filter.
- Consumer: `\w+.NewXxxClient(...)` via the same call_expression
  query + JS filter `^New(\w+)Client$`.

**Java (`java.ts`)**
- Provider: `class X extends YyyGrpc.YyyImplBase` — two queries
  handle the scoped and plain forms. `scoped_type_identifier`'s
  children are positional (no `scope:`/`name:` fields), so the
  query matches the two `type_identifier` children by position.
- `#match? @inner "ImplBase$"` restricts matches at query time.
- Whether the class has `@GrpcService` or not controls only the
  `source` metadata label — the plugin walks the class_declaration's
  `modifiers` child in JS to detect the marker_annotation.
- Consumer: `YyyGrpc.newStub(ch)` / `newBlockingStub(ch)` via a
  `method_invocation` query with `#match? @method
  "^new(Blocking)?Stub$"`, service name extracted via
  `^(\w+)Grpc$` on the object identifier.

**Python (`python.ts`)**
- Single call-expression query covers both bare identifier and
  `obj.method` attribute forms:
  `(call function: [(identifier) @fn (attribute attribute: (identifier) @fn)])`.
- Plugin filters `@fn.text` against two JS regexes:
  `^add_(\w+)Servicer_to_server$` (provider) and `^(\w+)Stub$`
  (consumer), with a reserved-names ignore list for the Stub case
  (Mock / Test / Fake / Stub).

**Node — JavaScript + TypeScript + TSX (`node.ts`)**
- Pattern sources defined once, compiled three times (one per grammar)
  because `Parser.Query` objects are not portable across grammars.
  Exports three `GrpcLanguagePlugin`s sharing the same `scan`.
- `@GrpcMethod('Service', 'Method')`: decorator query captures the
  two string literals. Confidence is hard-coded 0.8 regardless of
  proto map resolution (matches the original regex version's
  behaviour).
- `@GrpcClient(...) field: XxxServiceClient`: decorator query
  captures the decorator node, plugin walks up to find the enclosing
  `public_field_definition` (decorators on fields are CHILDREN of
  the field definition in tree-sitter-typescript, not siblings) and
  reads its first `type_annotation → type_identifier`, then runs the
  `^(\w+Service)Client$` JS filter.
- `client.getService<X>('AuthService')`: call-expression query on
  `member_expression.property = "getService"` + string literal arg.
- `new XxxServiceClient(...)`: `new_expression` with a bare
  identifier constructor, filtered by `^(\w+Service)Client$` so
  generic `new AuthClient(...)` (missing the `Service` infix) does
  NOT falsely register as a consumer. Preserves the regression test
  `test_extract_ts_non_service_client_constructor_is_ignored`.
- `loadPackageDefinition` dynamic loader: gated on
  `tree.rootNode.text.includes('loadPackageDefinition')`. When set,
  `new foo.bar.Xxx(...)` qualified constructors with a capitalised
  property name register as consumers.

## Orchestrator changes

`grpc-extractor.ts` loses every `scanGoProviders` / `scanJavaProviders`
/ ... helper and replaces them with a single source-scan loop that:

1. Parses each file with the plugin's grammar (one shared `Parser`
   instance across all files, `setLanguage` called per plugin).
2. Calls `plugin.scan(tree)` to get `GrpcDetection[]`.
3. Converts each detection to an `ExtractedContract` via the private
   `detectionToContract` helper, which:
   - Looks the short service name up in the proto map (filled by
     the `.proto` parser).
   - Picks confidence = `confidenceWithProto` if resolved, else
     `confidenceWithoutProto`.
   - Builds a method-level contract id (`grpc::pkg.Svc/Method`) when
     the detection carries a `methodName` (TS `@GrpcMethod` only),
     otherwise a service-level id (`grpc::pkg.Svc/*`).

Everything else — the `.proto` parser, `buildProtoContext`,
`buildProtoMap`, `resolveProtoConflict`, `serviceContractId`,
`stripProtoCommentsAndStrings`, `extractServiceBlocks`, the dedupe
function — stays exactly as before. The `.proto` parser is kept as a
pragmatic exception to the "no regex in extractors" rule because no
`tree-sitter-proto` grammar is installed in the repo; a comment at the
top of the file explains this and flags the maintainer option of
adding `tree-sitter-proto` as a dependency.

## Why this is better than the regex version

1. **Comments and strings are respected for free.** Matched node types
   are only code constructs, never text inside comments or string
   literals.
2. **No false positives on partial names.** The old `(\w+?)Grpc`-style
   regexes would cross-match unrelated identifiers; structural queries
   restrict matches to the exact AST shape (`scoped_type_identifier →
   type_identifier` pairs, `method_invocation → identifier` etc.).
3. **NestJS `@GrpcClient` is structural, not regex-based.** The old
   regex required a specific textual layout
   (`@GrpcClient(...) private readonly foo!: XxxServiceClient`); the
   plugin now walks the AST, so modifier order / optional modifiers /
   multi-line formatting don't break it.
4. **Language-agnostic extension.** Adding Kotlin / Rust / C# gRPC
   detection later is a one-file edit in `grpc-patterns/index.ts` —
   no touches to the shared scanner, the orchestrator, or the proto
   parser.

## Tests

- `grpc-extractor.test.ts` — **43/43 pass** (tests unchanged; the
  contract shape is identical). Covers .proto parsing (including the
  brace-inside-string regression), Go provider/consumer,
  Java @GrpcService / plain ImplBase provider + newBlockingStub
  consumer, Python servicer + stub, TS @GrpcMethod + @GrpcClient +
  .getService + new XxxServiceClient + loadPackageDefinition + the
  `AuthClient` vs `AuthServiceClient` discrimination, dedupe across
  multiple patterns in one file, proto-aware confidence, and the
  inherited-package resolution for split proto definitions.
- `topic-extractor.test.ts` — 30/30 pass.
- `http-route-extractor.test.ts` — 18/18 pass.
- `manifest-extractor.test.ts` — 8/8 pass.
- `service.test.ts`, `sync.test.ts`, `storage.test.ts` — 41/41 pass.
- `npx tsc -p tsconfig.json --noEmit` clean.

## Scope discipline (per GUARDRAILS.md)

- Only files under `src/core/group/extractors/` are touched.
- No pipeline.ts, MCP surface, ingestion, CI / release / security, or
  test changes.
- New tree-sitter grammar imports (`tree-sitter-go`, `tree-sitter-java`,
  `tree-sitter-python`, `tree-sitter-javascript`, `tree-sitter-typescript`)
  are all already installed for the ingestion pipeline.

## End of phase series

This commit completes the three-phase extractor refactor:
  - **Phase 1** (`ea06d11`): topic-extractor → `topic-patterns/`
  - **Phase 2** (`b6015f6`): http-route-extractor → `http-patterns/`
  - **Phase 3** (this commit): grpc-extractor → `grpc-patterns/`

Every remaining regex-based extractor helper under the `src/core/group/
extractors/` directory is either (a) language-agnostic string
processing (path normalization, dedupe keys) or (b) the `.proto`
parser, which is documented as an explicit exception.

Co-authored-by: Claude <noreply@anthropic.com>

* feat(group): add tree-sitter-proto for .proto file parsing

Addresses @magyargergo's suggestion on #796 to replace the manual
string-sanitizing .proto parser with a tree-sitter grammar.

- **Vendored `tree-sitter-proto`** in `vendor/tree-sitter-proto/`.
  Grammar source from [coder3101/tree-sitter-proto](https://github.com/coder3101/tree-sitter-proto)
  (latest `grammar.js`), parser.c regenerated with `tree-sitter-cli
  0.24` to produce ABI version 14 — compatible with the project's
  `tree-sitter 0.25` runtime (which supports ABI ≤ 14). Added as
  `optionalDependency` with `file:./vendor/tree-sitter-proto`.

- **New `grpc-patterns/proto.ts` plugin** — uses the same
  `compilePatterns` + `runCompiledPatterns` infrastructure as every
  other plugin. Two queries:
  - `(package (full_ident) @pkg)` — package declaration
  - `(service (service_name) @service_name (rpc (rpc_name) @rpc_name))`
    — one match per (service, rpc) pair

- **Graceful fallback** — `tree-sitter-proto` is an optional
  dependency. If it fails to install (platform incompatibility) or
  fails the runtime smoke-test (`setLanguage` + `parse` on a trivial
  proto), `PROTO_GRPC_PLUGIN` stays `null` and the orchestrator
  uses the existing manual parser. The smoke-test catches the
  `SyntaxNode` TDZ error that occurs in vitest's fork-based test
  runner.

- **Orchestrator updated** — when `hasProtoPlugin` is true, `.proto`
  files are handled by the plugin loop (they're included in
  `GRPC_SCAN_GLOB`), and the manual `parseProtoFile` loop is
  skipped. `buildProtoContext` still runs to build the proto map
  for cross-referencing source-file detections.

1. **No manual comment/string stripping.** The old parser needed
   `stripProtoCommentsAndStrings` (110 lines) to avoid counting
   braces inside comments and string literals. tree-sitter handles
   this natively.
2. **No brace-depth tracking.** `extractServiceBlocks` used a manual
   depth counter to find service boundaries. tree-sitter's AST gives
   us `service` → `service_name` + `rpc` → `rpc_name` directly.
3. **Performance.** tree-sitter's C-based parser is faster than
   character-by-character JS scanning + regex on large proto files.

- `grpc-extractor.test.ts` — **43/43 pass** (unchanged)
- All other extractor tests — 99/99 pass
- `npx tsc -p tsconfig.json --noEmit` clean

Co-authored-by: Claude <noreply@anthropic.com>

* chore: add .gitignore for vendored tree-sitter-proto build artifacts

https://claude.ai/code/session_01SFUCxgKMMQ8EgRHYw91xPU

* fix: correct .gitignore paths for vendored tree-sitter-proto

Patterns should be relative to the .gitignore file's directory.

https://claude.ai/code/session_01SFUCxgKMMQ8EgRHYw91xPU

* refactor(group): address Copilot review feedback on #796

Six fixes suggested by the Copilot AI review:

1. **`normalizeHttpPath` root-path edge case** — stripping trailing
   slashes on the input `/` produced an empty string, yielding
   malformed contract ids like `http::GET::`. Now preserves `/` for
   the root handler/fetch case.

2. **Dedupe `scanFiles` call** — `extract()` was globbing the
   source-scan file list twice (once for the provider fallback, once
   for the consumer fallback). Moved to a single lazy call that
   memoizes the result for the rest of the method.

3. **HTTP `scanFiles` now ignores `**/vendor/**`** — every other
   extractor's glob already ignored vendored sources; the HTTP one
   didn't. Fixed for consistency.

4. **`loadPackageDefinition` check is now structural** — was calling
   `tree.rootNode.text.includes('loadPackageDefinition')` which forces
   materialization of the entire file text from the parse tree
   (expensive on large files). Replaced with a dedicated compiled
   query on `(call_expression function: [(identifier) | (member_expression)])`
   so the check stays in the AST domain.

5. **`grpc-extractor.ts` header docstring updated** — still claimed
   ".proto parsing is not tree-sitter-based because no grammar is
   installed". Now describes the actual behaviour: tree-sitter when
   `tree-sitter-proto` is available (optionalDependency), manual
   fallback otherwise.

6. **Eliminated the double proto file parse on the fallback path** —
   `buildProtoContext` already globs + parses every `.proto` file to
   build `servicesByName`. On the `!hasProtoPlugin` branch the
   extractor was globbing + parsing again via the now-removed
   `parseProtoFile` helper. The fallback branch now iterates the map
   that `buildProtoContext` already produced to emit provider
   contracts directly — single pass per proto file.

## Tests

- `topic-extractor.test.ts` — 30/30 pass
- `http-route-extractor.test.ts` — 18/18 pass
- `grpc-extractor.test.ts` — 43/43 pass
- `manifest-extractor.test.ts` — 8/8 pass
- `npx tsc -p tsconfig.json --noEmit` clean

Co-authored-by: Claude <noreply@anthropic.com>

* refactor(group): address Claude review feedback (bugs + dedup + hygiene) on #796

Follows up `2f28bfc` with the remaining items from the Claude AI review:

## Bugs

**Bug 2 — Label-unaware Cypher queries in `resolveSymbol`.**
The manifest-extractor's lookup queries were `MATCH (n) WHERE n.name = $x`
with no label filter, so a topic/service/package name could silently match
any node type (File, Variable, Import, Folder, …). Added label filters:
- `topic` → `(n:Function|Method|Class|Interface)` (topics are best-effort
  symbol-name matches against listener/publisher symbols)
- `grpc` method → `(n:Function|Method)`
- `grpc` service → `(n:Class|Interface)`
- `lib` → `(n:Package|Module)`

All 8 manifest-extractor tests still pass (mock executor is
label-agnostic, but the production LadybugDB graph now gets correctly
scoped queries).

**Bug 8 — Tautological `!handlerName` condition.**
`http-route-extractor.ts:extractProvidersGraph` had
`let handlerName = null; if (!method || !handlerName) { ... }` — the
`!handlerName` clause was always true since there was no intervening
assignment. Simplified to always run the plugin-scan lookup (we need
the handler name even when `methodFromRouteReason` already resolved
the method).

## Clean code / dedup

**Design 7 — `readSafe` was copy-pasted in all three orchestrators.**
Extracted to `extractors/fs-utils.ts` as the single source of truth
for the path-traversal guard. Dropped the three local copies and the
now-unused `fs`/`path` imports from topic-extractor.

**Style 10 — Language-specific `_test.go` skip in the topic orchestrator.**
Was `if (rel.endsWith('_test.go')) continue;` inside the language-
agnostic extraction loop. Pushed into the glob's ignore list
(`'**/*_test.go'`) alongside the existing `node_modules`, `vendor`,
`dist`, `build` entries, with a comment explaining that other
languages' test file conventions either live in separate directories
(Python `tests/`, Java `src/test/`) or are already covered by the
existing ignores.

## Already addressed in `2f28bfc` (mentioned again in Claude review)

- Bug 3: `normalizeHttpPath('/')` returns `''` — fixed
- Bug 4: double glob + double parse of `.proto` — fixed
- Bug 5: `scanFiles` called twice in HTTP — fixed
- Bug 6: missing `**/vendor/**` in HTTP glob — fixed
- Design 9 partially: `tree.rootNode.text.includes('loadPackageDefinition')`
  replaced with a dedicated structural query

## Deferred

- Bug 1 (`http::*::path` vs `http::GET::path` matching) — out of scope;
  sync.ts matching logic lands in #793, manifest extractor already
  emits correct synthetic uids for unresolved HTTP contracts.
- Design 9 full (change plugin `scan(tree)` → `scan(tree, source)`) —
  the only real use case (`loadPackageDefinition` gate) is already
  fixed via a structural query, so the interface change would be
  cosmetic churn without a concrete consumer.

## Tests

- `topic-extractor.test.ts` — 30/30 pass
- `http-route-extractor.test.ts` — 18/18 pass
- `grpc-extractor.test.ts` — 43/43 pass
- `manifest-extractor.test.ts` — 8/8 pass
- `npx tsc -p tsconfig.json --noEmit` clean

Co-authored-by: Claude <noreply@anthropic.com>

* docs+fix(group): address remaining Claude review items + add pipeline flow chart

## Fixes

**Remaining 🔴 — HTTP contract id wildcard format.** Documented the
`http::*::<path>` format as an intentional wildcard for manifest links
that omit the HTTP method, alongside the explicit-method form
(`GET::/path` → `http::GET::/path`). The docblock on `buildContractId`
now states both forms, notes that wildcard-aware matching is the
responsibility of the sync / cross-impact layer (#793), and
recommends the explicit-method form whenever the author knows the
method (it round-trips through exact equality without needing
wildcard logic downstream). Tests unchanged — the wildcard format is
what they've always asserted.

**Minor 1 — stale comment at `manifest-extractor.ts:124-126`.** The
comment claimed "creates a contract with an empty symbolUid/ref" but
the code switched to `manifestSymbolUid(repo, contractId)` a few
commits back. Updated to describe the actual synthetic-uid fallback
semantics and the cross-impact path that relies on both sides of the
join deriving the same uid.

**Minor 2 — exhaustiveness guard on `buildContractId`.** The
`switch(type)` covered all five current `ContractType` variants but
silently returned `undefined` if a new variant was added. Added a
`default: const _exhaustive: never = type; throw new Error(...)`
clause so the build fails loudly on an unhandled variant.

**Minor 3 — `tree.rootNode.text` in `grpc-patterns/node.ts`.** Already
fixed in `2f28bfc` via a dedicated structural query
(`LOAD_PACKAGE_DEFINITION_SPEC`). No action needed.

## New: pipeline flow chart (per @magyargergo's request)

Added `src/core/group/PIPELINE.md` with four Mermaid diagrams:
1. **High-level overview** — `group.yaml` → extractors + manifest →
   contract matching → `bridge.lbug` → `runGroupImpact`.
2. **Per-repo extractor two-strategy shape** — graph-assisted
   Strategy A vs. source-scan Strategy B.
3. **Plugin architecture** — orchestrator → registry →
   per-language `*-patterns/<lang>.ts` → `tree-sitter-scanner.ts` →
   `ExtractedContract`.
4. **Manifest extraction** — label-scoped `resolveSymbol` with the
   synthetic-uid fallback.
5. **Cross-impact query (#606)** — local impact → bridge join →
   cross-repo fan-out.

Each diagram is annotated with which PRs own which stage (this PR:
extractors + manifest; #795: bridge storage; #606: cross-impact
runtime) and points at the concrete files/functions involved.

## Tests

- 99/99 extractor tests pass
- `npx tsc -p tsconfig.json --noEmit` clean

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
ivkond 2026-04-13 10:49:30 +03:00 • committed by GitHub
parent b10d25bbca
commit 4d4756fe86
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
44 changed files with 17186 additions and 835 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

28
gitnexus/package-lock.json generated
View file

@ -65,6 +65,7 @@
"optionalDependencies": {
"tree-sitter-dart": "github:UserNobody14/tree-sitter-dart#80e23c07b64494f7e21090bb3450223ef0b192f4",
"tree-sitter-kotlin": "^0.3.8",
"tree-sitter-proto": "file:./vendor/tree-sitter-proto",
"tree-sitter-swift": "^0.6.0"
}
},
@ -5296,6 +5297,10 @@
"node": "^18 || ^20 || >= 21"
}
},
"node_modules/tree-sitter-proto": {
"resolved": "vendor/tree-sitter-proto",
"link": true
},
"node_modules/tree-sitter-python": {
"version": "0.23.4",
"resolved": "https://registry.npmjs.org/tree-sitter-python/-/tree-sitter-python-0.23.4.tgz",
@ -5877,6 +5882,29 @@
"peerDependencies": {
"zod": "^3.25.28 || ^4"
}
},
"vendor/tree-sitter-proto": {
"version": "0.4.1",
"hasInstallScript": true,
"license": "MIT",
"optional": true,
"dependencies": {
"node-addon-api": "^8.0.0",
"node-gyp-build": "^4.8.0"
},
"peerDependencies": {
"tree-sitter": ">=0.21.0"
}
},
"vendor/tree-sitter-proto/node_modules/node-addon-api": {
"version": "8.7.0",
"resolved": "https://registry.npmjs.org/node-addon-api/-/node-addon-api-8.7.0.tgz",
"integrity": "sha512-9MdFxmkKaOYVTV+XVRG8ArDwwQ77XIgIPyKASB1k3JPq3M8fGQQQE3YpMOrKm6g//Ktx8ivZr8xo1Qmtqub+GA==",
"license": "MIT",
"optional": true,
"engines": {
"node": "^18 || ^20 || >= 21"
}
}
}
}

1
gitnexus/package.json
View file

@ -87,6 +87,7 @@
"optionalDependencies": {
"tree-sitter-dart": "github:UserNobody14/tree-sitter-dart#80e23c07b64494f7e21090bb3450223ef0b192f4",
"tree-sitter-kotlin": "^0.3.8",
"tree-sitter-proto": "file:./vendor/tree-sitter-proto",
"tree-sitter-swift": "^0.6.0"
},
"devDependencies": {

139
gitnexus/src/core/group/PIPELINE.md Normal file
View file

@ -0,0 +1,139 @@
# Group Analysis Pipeline
Flow chart of the cross-repo contract extraction + matching pipeline.
This covers what runs **inside this PR** (extractors + manifest) and
the downstream handoff to the bridge storage (PR #795) and
cross-impact query (PR #606).
## High-level overview
```mermaid
flowchart TD
A[group.yaml] --> B[GroupConfig parser]
B --> C{For each repo<br/>in group}
C --> D[Per-repo LadybugDB<br/>indexed by main pipeline]
D --> E1[TopicExtractor]
D --> E2[HttpRouteExtractor]
D --> E3[GrpcExtractor]
E1 --> F[ExtractedContract array<br/>per repo]
E2 --> F
E3 --> F
B --> M[ManifestExtractor]
M --> G[Manifest contracts<br/>+ cross-links]
F --> H[Contract matching<br/>exact + wildcard]
G --> H
H --> I[(bridge.lbug<br/>#795)]
I --> J[runGroupImpact<br/>#606]
J --> K[CrossRepoImpact]
```
## Per-repo extractor pipeline
Each extractor under `src/core/group/extractors/` follows the same
two-strategy shape:
```mermaid
flowchart TD
R[RepoHandle + CypherExecutor<br/>for this repo] --> S{Graph-assisted<br/>Strategy A<br/>available?}
S -->|yes| A1[Cypher query against<br/>per-repo LadybugDB]
A1 --> A2{non-empty<br/>result?}
A2 -->|yes| OUT[ExtractedContract array]
A2 -->|no| B1
S -->|no| B1[Source-scan Strategy B]
B1 --> B2[glob repo source files]
B2 --> B3{ext in registry?}
B3 -->|yes| B4[Per-language plugin<br/>scan parsed tree]
B3 -->|no| SKIP[skip file]
B4 --> OUT
SKIP --> B2
```
**Strategy A** (graph-assisted) uses Cypher over edges already produced
by the main ingestion pipeline:
- HTTP: `HANDLES_ROUTE` / `FETCHES` edges from `(File)-[]->(Route)`
- topic: none (pipeline doesn't yet produce topic nodes — Strategy B only)
- gRPC: none (Strategy B + proto map only)
**Strategy B** (source-scan) is 100% tree-sitter based after this PR.
Each `*-patterns/<lang>.ts` plugin owns its grammar + S-expression
queries; the top-level orchestrator imports neither.
## Plugin architecture
```mermaid
flowchart LR
O[Orchestrator<br/>topic|http|grpc-extractor.ts] --> REG[REGISTRY<br/>*-patterns/index.ts]
REG --> P1[java.ts<br/>tree-sitter-java]
REG --> P2[go.ts<br/>tree-sitter-go]
REG --> P3[python.ts<br/>tree-sitter-python]
REG --> P4[node.ts<br/>JS + TS + TSX]
REG --> P5[php.ts<br/>tree-sitter-php<br/>HTTP only]
REG --> P6[proto.ts<br/>tree-sitter-proto<br/>gRPC only, optional]
P1 --> SCAN[tree-sitter-scanner.ts<br/>compilePatterns + runCompiledPatterns]
P2 --> SCAN
P3 --> SCAN
P4 --> SCAN
P5 --> SCAN
P6 --> SCAN
SCAN --> DET[Detection objects<br/>TopicMeta / HttpDetection / GrpcDetection]
DET --> O
O --> CT[ExtractedContract array]
```
The orchestrator never imports a grammar. Adding a new language /
framework = drop one file in `*-patterns/`, register it in
`index.ts`. No orchestrator edits required.
## Manifest extraction
```mermaid
flowchart TD
Y[group.yaml links] --> ME[ManifestExtractor]
ME --> LOOP{for each link}
LOOP --> RES[resolveSymbol<br/>label-scoped Cypher]
RES --> OK{found?}
OK -->|yes| REF[real symbol uid + ref]
OK -->|no| SYN[synthetic uid<br/>manifest::repo::cid]
REF --> EMIT[emit provider + consumer<br/>Contract objects<br/>+ CrossLink]
SYN --> EMIT
EMIT --> BRIDGE[(bridge.lbug<br/>#795)]
```
Label-scoped queries in `resolveSymbol` keep accidental cross-matches
out:
- `topic` → `(n:Function|Method|Class|Interface)`
- `grpc` method → `(n:Function|Method)`, service → `(n:Class|Interface)`
- `lib` → `(n:Package|Module)`
## Cross-impact query (PR #606)
```mermaid
flowchart TD
U[User changes symbol S<br/>in repo R] --> LI[Local impact engine<br/>per-repo uid expansion]
LI --> IDS[Affected uid set]
IDS --> BR[Bridge query<br/>MATCH Contract WHERE uid IN ids]
BR --> CL[CrossLink traversal]
CL --> OTHER[Matching contract in<br/>other repo]
OTHER --> FE[Fan-out impact<br/>to consuming repo]
FE --> OUT[CrossRepoImpact<br/>per affected repo]
```
The bridge stores every extracted contract keyed by `symbolUid`.
Manifest-sourced contracts use the synthetic uid form so both sides
of the `(local impact) ↔ (bridge query)` join derive the same uid
without coordinating through any shared state.

23
gitnexus/src/core/group/extractors/fs-utils.ts Normal file
View file

@ -0,0 +1,23 @@
import * as fs from 'node:fs';
import * as path from 'node:path';
/**
* Safely read a file inside a repo, rejecting any path that escapes
* `repoPath` via `..` traversal or absolute segments. Returns `null` if
* the path is outside the repo or the file can't be read.
*
* Used by every source-scan extractor under this directory. Kept as a
* single shared implementation so the path-traversal guard (security-
* sensitive) lives in exactly one place.
*/
export function readSafe(repoPath: string, rel: string): string | null {
const abs = path.resolve(repoPath, rel);
const base = path.resolve(repoPath);
const relToBase = path.relative(base, abs);
if (relToBase.startsWith('..') || path.isAbsolute(relToBase)) return null;
try {
return fs.readFileSync(abs, 'utf-8');
} catch {
return null;
}
}

626
gitnexus/src/core/group/extractors/grpc-extractor.ts
View file

@ -1,20 +1,38 @@
import * as fs from 'node:fs';
import * as path from 'node:path';
import { glob } from 'glob';
import Parser from 'tree-sitter';
import type { ContractExtractor, CypherExecutor } from '../contract-extractor.js';
import type { ExtractedContract, RepoHandle } from '../types.js';
import { readSafe } from './fs-utils.js';
import {
GRPC_SCAN_GLOB,
getPluginForFile,
hasProtoPlugin,
type GrpcDetection,
} from './grpc-patterns/index.js';
function readSafe(repoPath: string, rel: string): string | null {
const abs = path.resolve(repoPath, rel);
const base = path.resolve(repoPath);
const relToBase = path.relative(base, abs);
if (relToBase.startsWith('..') || path.isAbsolute(relToBase)) return null;
try {
return fs.readFileSync(abs, 'utf-8');
} catch {
return null;
}
}
/**
* Language-agnostic orchestrator for gRPC (provider + consumer) contract
* extraction.
*
* Two parts:
*
* 1. **`.proto` parsing** — tree-sitter when `tree-sitter-proto` is
* installed (optionalDependency vendored in `vendor/tree-sitter-proto/`),
* via the `.proto` entry in `grpc-patterns/` and `hasProtoPlugin`.
* When the grammar isn't available (platform incompatibility, native
* build failure) the orchestrator falls back to the in-process
* string-sanitizing parser defined below (`stripProtoCommentsAndStrings`
* + `extractServiceBlocks`). The fallback preserves offsets so any
* downstream regex scans run against a sanitized copy without
* affecting line numbers of the original.
*
* 2. **Source-scan providers / consumers** — delegated to per-language
* plugins in `./grpc-patterns/`. The orchestrator imports NO
* tree-sitter grammars or query strings — each plugin owns its own.
*/
// ─── .proto fallback parser (used only when tree-sitter-proto is absent) ───
function contractId(pkg: string, service: string, method: string): string {
const prefix = pkg ? `${pkg}.${service}` : service;
@ -25,20 +43,110 @@ function serviceOnlyContractId(serviceName: string): string {
return `grpc::${serviceName}/*`;
}
/**
* Replace all .proto comments and string literals with spaces, preserving the
* original length and character offsets of the input. This lets downstream
* regex / brace-depth parsers run on a "sanitized" copy without having to
* understand proto syntax, while any RegExp.exec/index-based lookups that
* were already positional against `content` continue to work against the
* original string.
*
* Supported comment forms: `// line comment`, `/* block comment * /`.
* Supported strings: double-quoted ("…") and single-quoted ('…') with `\`
* escape handling. Raw/unterminated strings are not supported — we stop
* on a line break for line-style comments and on EOF for unterminated
* strings/blocks, which matches how most real proto files parse.
*/
function stripProtoCommentsAndStrings(content: string): string {
const out = new Array<string>(content.length);
let i = 0;
while (i < content.length) {
const ch = content[i];
const next = content[i + 1];
// Line comment: // ... \n
if (ch === '/' && next === '/') {
out[i] = ' ';
out[i + 1] = ' ';
i += 2;
while (i < content.length && content[i] !== '\n') {
out[i] = content[i] === '\r' ? '\r' : ' ';
i++;
}
continue;
}
// Block comment: /* ... */
if (ch === '/' && next === '*') {
out[i] = ' ';
out[i + 1] = ' ';
i += 2;
while (i < content.length) {
if (content[i] === '*' && content[i + 1] === '/') {
out[i] = ' ';
out[i + 1] = ' ';
i += 2;
break;
}
// Preserve newlines so line numbers stay stable for downstream code.
out[i] = content[i] === '\n' || content[i] === '\r' ? content[i] : ' ';
i++;
}
continue;
}
// String literal: "..." or '...'
if (ch === '"' || ch === "'") {
const quote = ch;
out[i] = ' '; // replace opening quote
i++;
while (i < content.length) {
const c = content[i];
if (c === '\\' && i + 1 < content.length) {
// Skip escaped pair (e.g. \" \n \\)
out[i] = ' ';
out[i + 1] = ' ';
i += 2;
continue;
}
if (c === quote) {
out[i] = ' ';
i++;
break;
}
// Preserve newlines; proto technically disallows unescaped newlines
// inside strings, but real files occasionally have them.
out[i] = c === '\n' || c === '\r' ? c : ' ';
i++;
}
continue;
}
out[i] = ch;
i++;
}
return out.join('');
}
function extractServiceBlocks(content: string): Array<{ name: string; body: string }> {
const results: Array<{ name: string; body: string }> = [];
// v1: brace-depth only — braces inside comments or string literals are not filtered (see spec Fix 2)
// Sanitize comments and string literals so braces inside them don't
// throw off the depth counter. The sanitized copy has the same length
// and offsets as the original, so we use it ONLY to scan for service
// headers and braces; the service body we return is sliced from the
// ORIGINAL content to preserve exact source text for downstream use.
const sanitized = stripProtoCommentsAndStrings(content);
const headerRe = /service\s+(\w+)\s*\{/g;
let headerMatch: RegExpExecArray | null;
while ((headerMatch = headerRe.exec(content)) !== null) {
while ((headerMatch = headerRe.exec(sanitized)) !== null) {
const serviceName = headerMatch[1];
const bodyStart = headerMatch.index + headerMatch[0].length;
let depth = 1;
let pos = bodyStart;
while (pos < content.length && depth > 0) {
const ch = content[pos];
while (pos < sanitized.length && depth > 0) {
const ch = sanitized[pos];
if (ch === '{') depth++;
else if (ch === '}') depth--;
pos++;
@ -75,6 +183,165 @@ function makeContract(
};
}
export interface ProtoServiceInfo {
package: string;
serviceName: string;
methods: string[];
protoPath: string;
}
function normalizeProtoPath(rel: string): string {
return rel.replace(/\\/g, '/');
}
function extractProtoImports(content: string): string[] {
const imports: string[] = [];
const re = /^\s*import\s+"([^"]+)"\s*;/gm;
let match: RegExpExecArray | null;
while ((match = re.exec(content)) !== null) {
imports.push(match[1]);
}
return imports;
}
function longestSharedSegmentRun(aPath: string, bPath: string): number {
const a = aPath.split('/').filter(Boolean);
const b = bPath.split('/').filter(Boolean);
let best = 0;
for (let i = 0; i < a.length; i++) {
for (let j = 0; j < b.length; j++) {
let run = 0;
while (a[i + run] && b[j + run] && a[i + run] === b[j + run]) {
run++;
}
if (run > best) best = run;
}
}
return best;
}
async function buildProtoContext(repoPath: string): Promise<{
packagesByProto: Map<string, string>;
servicesByName: Map<string, ProtoServiceInfo[]>;
}> {
const servicesByName = new Map<string, ProtoServiceInfo[]>();
const protoFiles = await glob('**/*.proto', {
cwd: repoPath,
absolute: false,
nodir: true,
ignore: ['**/node_modules/**', '**/.git/**', '**/vendor/**'],
});
const contents = new Map<string, string>();
for (const rel of protoFiles) {
const content = readSafe(repoPath, rel);
if (!content) continue;
contents.set(normalizeProtoPath(rel), content);
}
const packagesByProto = new Map<string, string>();
const resolvePackage = (protoPath: string, seen = new Set<string>()): string => {
if (packagesByProto.has(protoPath)) return packagesByProto.get(protoPath) ?? '';
if (seen.has(protoPath)) return '';
const content = contents.get(protoPath);
if (!content) return '';
seen.add(protoPath);
const pkgMatch = content.match(/^\s*package\s+([\w.]+)\s*;/m);
if (pkgMatch?.[1]) {
packagesByProto.set(protoPath, pkgMatch[1]);
return pkgMatch[1];
}
for (const importPath of extractProtoImports(content)) {
const normalizedImport = normalizeProtoPath(importPath);
const candidates = [
normalizeProtoPath(
path.posix.normalize(path.posix.join(path.posix.dirname(protoPath), normalizedImport)),
),
normalizedImport,
];
for (const candidate of candidates) {
if (!contents.has(candidate)) continue;
const inheritedPackage = resolvePackage(candidate, seen);
if (inheritedPackage) {
packagesByProto.set(protoPath, inheritedPackage);
return inheritedPackage;
}
}
}
packagesByProto.set(protoPath, '');
return '';
};
for (const rel of protoFiles) {
const normalizedRel = normalizeProtoPath(rel);
const content = contents.get(normalizedRel);
if (!content) continue;
const pkg = resolvePackage(normalizedRel);
const serviceBlocks = extractServiceBlocks(content);
for (const block of serviceBlocks) {
const rpcRe = /rpc\s+(\w+)\s*\(/g;
const methods: string[] = [];
let m: RegExpExecArray | null;
while ((m = rpcRe.exec(block.body)) !== null) {
methods.push(m[1]);
}
const info: ProtoServiceInfo = {
package: pkg,
serviceName: block.name,
methods,
protoPath: normalizedRel,
};
const existing = servicesByName.get(block.name) ?? [];
existing.push(info);
servicesByName.set(block.name, existing);
}
}
return { packagesByProto, servicesByName };
}
export async function buildProtoMap(repoPath: string): Promise<Map<string, ProtoServiceInfo[]>> {
const { servicesByName } = await buildProtoContext(repoPath);
return servicesByName;
}
export function resolveProtoConflict(
_serviceName: string,
sourceFilePath: string,
candidates: ProtoServiceInfo[],
): ProtoServiceInfo | null {
if (candidates.length === 0) return null;
if (candidates.length === 1) return candidates[0];
const sourceDir = normalizeProtoPath(path.dirname(sourceFilePath));
let best = candidates[0];
let bestScore = -1;
for (const c of candidates) {
const protoDir = normalizeProtoPath(path.dirname(c.protoPath));
const sharedRun = longestSharedSegmentRun(sourceDir, protoDir);
if (sharedRun > bestScore) {
bestScore = sharedRun;
best = c;
}
}
return best;
}
export function serviceContractId(pkg: string, serviceName: string): string {
const prefix = pkg ? `${pkg}.${serviceName}` : serviceName;
return `grpc::${prefix}/*`;
}
// ─── Orchestrator ────────────────────────────────────────────────────
export class GrpcExtractor implements ContractExtractor {
type = 'grpc' as const;
@ -88,270 +355,111 @@ export class GrpcExtractor implements ContractExtractor {
_repo: RepoHandle,
): Promise<ExtractedContract[]> {
const out: ExtractedContract[] = [];
const protoContext = await buildProtoContext(repoPath);
const protoMap = protoContext.servicesByName;
// Proto files — definitive provider source
const protoFiles = await glob('**/*.proto', {
cwd: repoPath,
ignore: ['**/node_modules/**', '**/.git/**', '**/vendor/**'],
nodir: true,
});
for (const rel of protoFiles) {
const content = readSafe(repoPath, rel);
if (content) out.push(...this.parseProtoFile(content, rel));
// ─── Proto files — definitive provider source ─────────────────
// When tree-sitter-proto is available, .proto files are handled by
// the plugin loop below (they're in GRPC_SCAN_GLOB). Otherwise
// emit provider contracts directly from the proto map that
// `buildProtoContext` already built — no second glob / parse pass.
if (!hasProtoPlugin) {
for (const infos of protoMap.values()) {
for (const info of infos) {
for (const methodName of info.methods) {
const cid = contractId(info.package, info.serviceName, methodName);
out.push(
makeContract(
cid,
'provider',
info.protoPath,
`${info.serviceName}.${methodName}`,
0.85,
{
package: info.package,
service: info.serviceName,
method: methodName,
source: 'proto',
},
),
);
}
}
}
}
// Source files — server/client detection
const sourceFiles = await glob('**/*.{go,java,py,ts,tsx,js,jsx}', {
// ─── Source files (+ .proto when plugin available) ────────────
const sourceFiles = await glob(GRPC_SCAN_GLOB, {
cwd: repoPath,
ignore: ['**/node_modules/**', '**/.git/**', '**/vendor/**', '**/dist/**', '**/build/**'],
nodir: true,
});
const parser = new Parser();
for (const rel of sourceFiles) {
const plugin = getPluginForFile(rel);
if (!plugin) continue;
const content = readSafe(repoPath, rel);
if (!content) continue;
const ext = path.extname(rel).toLowerCase();
if (ext === '.go') {
out.push(...this.scanGoProviders(content, rel));
out.push(...this.scanGoConsumers(content, rel));
} else if (ext === '.java') {
out.push(...this.scanJavaProviders(content, rel));
out.push(...this.scanJavaConsumers(content, rel));
} else if (ext === '.py') {
out.push(...this.scanPythonProviders(content, rel));
out.push(...this.scanPythonConsumers(content, rel));
} else if (['.ts', '.tsx', '.js', '.jsx'].includes(ext)) {
out.push(...this.scanTsProviders(content, rel));
let detections: GrpcDetection[] = [];
try {
parser.setLanguage(plugin.language);
const tree = parser.parse(content);
detections = plugin.scan(tree);
} catch {
continue;
}
for (const d of detections) {
out.push(this.detectionToContract(d, rel, protoMap));
}
}
return this.dedupe(out);
}
private parseProtoFile(content: string, filePath: string): ExtractedContract[] {
const out: ExtractedContract[] = [];
const pkgMatch = content.match(/^package\s+([\w.]+)\s*;/m);
const pkg = pkgMatch ? pkgMatch[1] : '';
for (const { name: serviceName, body } of extractServiceBlocks(content)) {
const rpcRe = /rpc\s+(\w+)\s*\(/g;
let rpcMatch: RegExpExecArray | null;
while ((rpcMatch = rpcRe.exec(body)) !== null) {
const methodName = rpcMatch[1];
const cid = contractId(pkg, serviceName, methodName);
out.push(
makeContract(cid, 'provider', filePath, `${serviceName}.${methodName}`, 0.85, {
package: pkg,
service: serviceName,
method: methodName,
source: 'proto',
}),
);
}
}
return out;
}
private scanGoProviders(content: string, filePath: string): ExtractedContract[] {
const out: ExtractedContract[] = [];
// pb.RegisterXxxServer(
const registerRe = /\w+\.Register(\w+)Server\s*\(/g;
let m: RegExpExecArray | null;
while ((m = registerRe.exec(content)) !== null) {
const serviceName = m[1];
out.push(
makeContract(
serviceOnlyContractId(serviceName),
'provider',
filePath,
`Register${serviceName}Server`,
0.8,
{ service: serviceName, source: 'go_register' },
),
);
}
// pb.UnimplementedXxxServer
const unimplRe = /\w+\.Unimplemented(\w+)Server\b/g;
while ((m = unimplRe.exec(content)) !== null) {
const serviceName = m[1];
out.push(
makeContract(
serviceOnlyContractId(serviceName),
'provider',
filePath,
`Unimplemented${serviceName}Server`,
0.8,
{ service: serviceName, source: 'go_unimplemented' },
),
);
}
return out;
}
private scanGoConsumers(content: string, filePath: string): ExtractedContract[] {
const out: ExtractedContract[] = [];
const re = /\w+\.New(\w+)Client\s*\(/g;
let m: RegExpExecArray | null;
while ((m = re.exec(content)) !== null) {
const serviceName = m[1];
out.push(
makeContract(
serviceOnlyContractId(serviceName),
'consumer',
filePath,
`New${serviceName}Client`,
0.7,
{ service: serviceName, source: 'go_client' },
),
);
}
return out;
}
private scanJavaProviders(content: string, filePath: string): ExtractedContract[] {
const out: ExtractedContract[] = [];
// @GrpcService
if (content.includes('@GrpcService')) {
const implBaseRe = /extends\s+(\w+)Grpc\.(\w+)ImplBase/;
const m = content.match(implBaseRe);
if (m) {
out.push(
makeContract(serviceOnlyContractId(m[1]), 'provider', filePath, m[2], 0.8, {
service: m[1],
source: 'java_grpc_service',
}),
);
} else {
// Try extracting service name from class name
const classRe =
/class\s+(\w*?)(?:Grpc)?(?:Service)?\s+extends\s+(\w+)(?:Grpc\.(\w+))?ImplBase/;
const cm = content.match(classRe);
if (cm) {
const svcName = cm[2].replace(/Grpc$/, '');
out.push(
makeContract(serviceOnlyContractId(svcName), 'provider', filePath, cm[1], 0.8, {
service: svcName,
source: 'java_grpc_service',
}),
);
}
}
}
// extends XxxImplBase (without @GrpcService)
if (!content.includes('@GrpcService')) {
const implRe = /extends\s+(\w+?)(?:Grpc\.(\w+))?ImplBase/;
const m = content.match(implRe);
if (m) {
const svcName = m[2] || m[1].replace(/Grpc$/, '');
out.push(
makeContract(serviceOnlyContractId(svcName), 'provider', filePath, svcName, 0.8, {
service: svcName,
source: 'java_impl_base',
}),
);
}
}
return out;
}
private scanJavaConsumers(content: string, filePath: string): ExtractedContract[] {
const out: ExtractedContract[] = [];
// XxxGrpc.newBlockingStub( or XxxGrpc.newStub(
const re = /(\w+)Grpc\.new(?:Blocking)?Stub\s*\(/g;
let m: RegExpExecArray | null;
while ((m = re.exec(content)) !== null) {
const serviceName = m[1];
out.push(
makeContract(
serviceOnlyContractId(serviceName),
'consumer',
filePath,
`${serviceName}Stub`,
0.7,
{ service: serviceName, source: 'java_stub' },
),
);
}
return out;
}
private scanPythonProviders(content: string, filePath: string): ExtractedContract[] {
const out: ExtractedContract[] = [];
// add_XxxServicer_to_server(
const re = /add_(\w+?)Servicer_to_server\s*\(/g;
let m: RegExpExecArray | null;
while ((m = re.exec(content)) !== null) {
const serviceName = m[1];
out.push(
makeContract(
serviceOnlyContractId(serviceName),
'provider',
filePath,
`add_${serviceName}Servicer_to_server`,
0.8,
{ service: serviceName, source: 'python_servicer' },
),
);
}
return out;
}
private scanPythonConsumers(content: string, filePath: string): ExtractedContract[] {
const out: ExtractedContract[] = [];
// XxxStub(
const re = /(\w+)Stub\s*\(/g;
let m: RegExpExecArray | null;
while ((m = re.exec(content)) !== null) {
const name = m[1];
// Filter out common false positives
if (['Mock', 'Test', 'Fake', 'Stub'].includes(name)) continue;
out.push(
makeContract(serviceOnlyContractId(name), 'consumer', filePath, `${name}Stub`, 0.7, {
service: name,
source: 'python_stub',
}),
);
}
return out;
}
private scanTsProviders(content: string, filePath: string): ExtractedContract[] {
const out: ExtractedContract[] = [];
// @GrpcMethod('ServiceName', 'MethodName')
const re = /@GrpcMethod\s*\(\s*['"](\w+)['"]\s*,\s*['"](\w+)['"]\s*\)/g;
let m: RegExpExecArray | null;
while ((m = re.exec(content)) !== null) {
const serviceName = m[1];
const methodName = m[2];
const cid = contractId('', serviceName, methodName);
out.push(
makeContract(cid, 'provider', filePath, `${serviceName}.${methodName}`, 0.8, {
service: serviceName,
method: methodName,
source: 'ts_grpc_method',
}),
);
}
return out;
/**
* Convert a plugin `GrpcDetection` into a concrete `ExtractedContract`
* by resolving the short service name against the proto map, building
* either a service-level (`grpc::pkg.Svc/*`) or method-level
* (`grpc::pkg.Svc/Method`) contract id, and selecting confidence
* based on whether the proto map had an entry.
*/
private detectionToContract(
d: GrpcDetection,
filePath: string,
protoMap: Map<string, ProtoServiceInfo[]>,
): ExtractedContract {
const candidates = protoMap.get(d.serviceName);
const proto = resolveProtoConflict(d.serviceName, filePath, candidates ?? []);
const pkg = proto?.package ?? '';
const cid = d.methodName
? contractId(pkg, d.serviceName, d.methodName)
: proto
? serviceContractId(pkg, d.serviceName)
: serviceOnlyContractId(d.serviceName);
const confidence = proto ? d.confidenceWithProto : d.confidenceWithoutProto;
const meta: Record<string, unknown> = {
service: d.serviceName,
source: d.source,
};
if (d.methodName) meta.method = d.methodName;
return makeContract(cid, d.role, filePath, d.symbolName, confidence, meta);
}
private dedupe(items: ExtractedContract[]): ExtractedContract[] {
const seen = new Set<string>();
const out: ExtractedContract[] = [];
const byKey = new Map<string, ExtractedContract>();
for (const c of items) {
const k = `${c.contractId}|${c.role}|${c.symbolRef.filePath}`;
if (seen.has(k)) continue;
seen.add(k);
out.push(c);
const existing = byKey.get(k);
if (
!existing ||
c.confidence > existing.confidence ||
(c.confidence === existing.confidence &&
String(c.meta.source) < String(existing.meta.source))
) {
byKey.set(k, c);
}
}
return out;
return Array.from(byKey.values());
}
}

109
gitnexus/src/core/group/extractors/grpc-patterns/go.ts Normal file
View file

@ -0,0 +1,109 @@
import Go from 'tree-sitter-go';
import {
compilePatterns,
runCompiledPatterns,
type LanguagePatterns,
} from '../tree-sitter-scanner.js';
import type { GrpcDetection, GrpcLanguagePlugin } from './types.js';
/**
* Go gRPC plugin. Detects:
* - Provider: `pb.RegisterXxxServer(...)` calls
* - Provider: `pb.UnimplementedXxxServer` embedded in a struct
* - Consumer: `pb.NewXxxClient(conn)` calls
*/
const REGISTER_RE = /^Register(\w+)Server$/;
const UNIMPLEMENTED_RE = /^Unimplemented(\w+)Server$/;
const NEW_CLIENT_RE = /^New(\w+)Client$/;
// Any `xxx.<fn>(...)` call — plugin filters the field identifier text.
const SELECTOR_CALL_PATTERNS = compilePatterns({
name: 'go-grpc-selector-call',
language: Go,
patterns: [
{
meta: {},
query: `
(call_expression
function: (selector_expression
field: (field_identifier) @fn))
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
// Any `qualified_type` used as a struct field — for `pb.UnimplementedXxxServer`.
const STRUCT_EMBEDDING_PATTERNS = compilePatterns({
name: 'go-grpc-struct-embedding',
language: Go,
patterns: [
{
meta: {},
query: `
(struct_type
(field_declaration_list
(field_declaration
type: (qualified_type
name: (type_identifier) @field_type))))
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
export const GO_GRPC_PLUGIN: GrpcLanguagePlugin = {
name: 'go-grpc',
language: Go,
scan(tree) {
const out: GrpcDetection[] = [];
for (const match of runCompiledPatterns(SELECTOR_CALL_PATTERNS, tree)) {
const fnNode = match.captures.fn;
if (!fnNode) continue;
const fnText = fnNode.text;
const registerMatch = REGISTER_RE.exec(fnText);
if (registerMatch) {
out.push({
role: 'provider',
serviceName: registerMatch[1],
symbolName: fnText,
source: 'go_register',
confidenceWithProto: 0.8,
confidenceWithoutProto: 0.65,
});
continue;
}
const newClientMatch = NEW_CLIENT_RE.exec(fnText);
if (newClientMatch) {
out.push({
role: 'consumer',
serviceName: newClientMatch[1],
symbolName: fnText,
source: 'go_client',
confidenceWithProto: 0.75,
confidenceWithoutProto: 0.55,
});
continue;
}
}
for (const match of runCompiledPatterns(STRUCT_EMBEDDING_PATTERNS, tree)) {
const fieldNode = match.captures.field_type;
if (!fieldNode) continue;
const unimpl = UNIMPLEMENTED_RE.exec(fieldNode.text);
if (!unimpl) continue;
out.push({
role: 'provider',
serviceName: unimpl[1],
symbolName: fieldNode.text,
source: 'go_unimplemented',
confidenceWithProto: 0.8,
confidenceWithoutProto: 0.65,
});
}
return out;
},
};

53
gitnexus/src/core/group/extractors/grpc-patterns/index.ts Normal file
View file

@ -0,0 +1,53 @@
import * as path from 'node:path';
import type { GrpcLanguagePlugin } from './types.js';
import { GO_GRPC_PLUGIN } from './go.js';
import { JAVA_GRPC_PLUGIN } from './java.js';
import { PYTHON_GRPC_PLUGIN } from './python.js';
import { JAVASCRIPT_GRPC_PLUGIN, TYPESCRIPT_GRPC_PLUGIN, TSX_GRPC_PLUGIN } from './node.js';
import { PROTO_GRPC_PLUGIN } from './proto.js';
export type { GrpcDetection, GrpcLanguagePlugin, GrpcRole } from './types.js';
export { PROTO_GRPC_PLUGIN, extractPackageFromTree } from './proto.js';
/**
* File-extension → gRPC language plugin registry. Mirrors the shape
* of `http-patterns/index.ts` and `topic-patterns/index.ts`.
*
* `.proto` files are registered only when `tree-sitter-proto` is
* available (it's an optionalDependency). When absent, the orchestrator
* falls back to the built-in manual proto parser.
*/
const REGISTRY: Record<string, GrpcLanguagePlugin> = {
'.go': GO_GRPC_PLUGIN,
'.java': JAVA_GRPC_PLUGIN,
'.py': PYTHON_GRPC_PLUGIN,
'.js': JAVASCRIPT_GRPC_PLUGIN,
'.jsx': JAVASCRIPT_GRPC_PLUGIN,
'.ts': TYPESCRIPT_GRPC_PLUGIN,
'.tsx': TSX_GRPC_PLUGIN,
...(PROTO_GRPC_PLUGIN ? { '.proto': PROTO_GRPC_PLUGIN } : {}),
};
/**
* Glob for source files worth scanning for gRPC server/client patterns.
* Includes `.proto` when the grammar is available.
*/
export const GRPC_SCAN_GLOB = PROTO_GRPC_PLUGIN
? '**/*.{go,java,py,ts,tsx,js,jsx,proto}'
: '**/*.{go,java,py,ts,tsx,js,jsx}';
/**
* Whether the tree-sitter proto plugin is available. The orchestrator
* uses this to decide between the tree-sitter path and the fallback
* manual parser for `.proto` files.
*/
export const hasProtoPlugin = PROTO_GRPC_PLUGIN !== null;
/**
* Return the gRPC plugin registered for the given file's extension,
* or `undefined` if the extension is not registered.
*/
export function getPluginForFile(rel: string): GrpcLanguagePlugin | undefined {
const ext = path.extname(rel).toLowerCase();
return REGISTRY[ext];
}

179
gitnexus/src/core/group/extractors/grpc-patterns/java.ts Normal file
View file

@ -0,0 +1,179 @@
import Parser from 'tree-sitter';
import Java from 'tree-sitter-java';
import {
compilePatterns,
runCompiledPatterns,
type LanguagePatterns,
} from '../tree-sitter-scanner.js';
import type { GrpcDetection, GrpcLanguagePlugin } from './types.js';
/**
* Java gRPC plugin. Detects:
* - Provider: classes extending `XxxServiceGrpc.XxxServiceImplBase`
* (with or without a `@GrpcService` annotation; the annotation
* only affects confidence labelling in the original regex version
* — here we emit a single detection per class and pick the source
* label based on whether the annotation is present).
* - Consumer: `XxxServiceGrpc.newBlockingStub(ch)` /
* `XxxServiceGrpc.newStub(ch)` calls.
*/
const IMPL_BASE_RE = /^(\w+)ImplBase$/;
const GRPC_SUFFIX_RE = /^(\w+)Grpc$/;
// Classes extending `ScopedType.ScopedType` where the inner name ends
// in ImplBase. Covers `XxxServiceGrpc.XxxServiceImplBase`.
// Note: tree-sitter-java's `scoped_type_identifier` exposes its two
// segments as positional `type_identifier` children, NOT as named
// `scope:`/`name:` fields. We match positionally here and rely on the
// grammar's left-to-right ordering: first child = outer, second = inner.
const SCOPED_IMPL_BASE_PATTERNS = compilePatterns({
name: 'java-grpc-scoped-impl-base',
language: Java,
patterns: [
{
meta: {},
query: `
(class_declaration
name: (identifier) @class_name
superclass: (superclass
(scoped_type_identifier
(type_identifier) @outer
(type_identifier) @inner (#match? @inner "ImplBase$")))) @class
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
// Classes extending a simple `XxxImplBase` identifier (no scope).
const PLAIN_IMPL_BASE_PATTERNS = compilePatterns({
name: 'java-grpc-plain-impl-base',
language: Java,
patterns: [
{
meta: {},
query: `
(class_declaration
name: (identifier) @class_name
superclass: (superclass
(type_identifier) @plain_type (#match? @plain_type "ImplBase$"))) @class
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
// gRPC stub factories: `XxxGrpc.newStub(ch)` / `XxxGrpc.newBlockingStub(ch)`.
const STUB_PATTERNS = compilePatterns({
name: 'java-grpc-stub',
language: Java,
patterns: [
{
meta: {},
query: `
(method_invocation
object: (identifier) @grpc_cls
name: (identifier) @method (#match? @method "^new(Blocking)?Stub$"))
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
/**
* Check whether a `class_declaration` node has a `@GrpcService`
* annotation in its modifiers list. In tree-sitter-java, class-level
* annotations live under `(class_declaration (modifiers (marker_annotation|annotation)))`.
*/
function hasGrpcServiceAnnotation(classNode: Parser.SyntaxNode): boolean {
for (let i = 0; i < classNode.namedChildCount; i++) {
const child = classNode.namedChild(i);
if (!child || child.type !== 'modifiers') continue;
for (let j = 0; j < child.namedChildCount; j++) {
const mod = child.namedChild(j);
if (!mod) continue;
if (mod.type !== 'marker_annotation' && mod.type !== 'annotation') continue;
const nameNode = mod.childForFieldName('name');
if (nameNode?.text === 'GrpcService') return true;
}
}
return false;
}
/**
* Given the inner type_identifier text like `AuthServiceImplBase`,
* return the service name (`AuthService`), or null if the text
* doesn't end in `ImplBase`.
*/
function extractServiceFromImplBase(text: string): string | null {
const m = IMPL_BASE_RE.exec(text);
if (!m) return null;
// Strip a trailing `Grpc` on the service name too — the original
// regex replaces `Grpc$` on the extracted prefix.
return m[1].replace(/Grpc$/, '');
}
export const JAVA_GRPC_PLUGIN: GrpcLanguagePlugin = {
name: 'java-grpc',
language: Java,
scan(tree) {
const out: GrpcDetection[] = [];
const emittedClassIds = new Set<number>();
// ─── Providers: scoped form (`...Grpc.XxxImplBase`) ─────────────
for (const match of runCompiledPatterns(SCOPED_IMPL_BASE_PATTERNS, tree)) {
const classNode = match.captures.class;
const innerNode = match.captures.inner;
if (!classNode || !innerNode) continue;
const serviceName = extractServiceFromImplBase(innerNode.text);
if (!serviceName) continue;
emittedClassIds.add(classNode.id);
const annotated = hasGrpcServiceAnnotation(classNode);
out.push({
role: 'provider',
serviceName,
symbolName: serviceName,
source: annotated ? 'java_grpc_service' : 'java_impl_base',
confidenceWithProto: 0.8,
confidenceWithoutProto: 0.65,
});
}
// ─── Providers: plain form (`XxxImplBase`) ──────────────────────
for (const match of runCompiledPatterns(PLAIN_IMPL_BASE_PATTERNS, tree)) {
const classNode = match.captures.class;
const plainNode = match.captures.plain_type;
if (!classNode || !plainNode) continue;
if (emittedClassIds.has(classNode.id)) continue;
const serviceName = extractServiceFromImplBase(plainNode.text);
if (!serviceName) continue;
emittedClassIds.add(classNode.id);
const annotated = hasGrpcServiceAnnotation(classNode);
out.push({
role: 'provider',
serviceName,
symbolName: serviceName,
source: annotated ? 'java_grpc_service' : 'java_impl_base',
confidenceWithProto: 0.8,
confidenceWithoutProto: 0.65,
});
}
// ─── Consumers: `XxxGrpc.newBlockingStub(...)` / `newStub(...)` ─
for (const match of runCompiledPatterns(STUB_PATTERNS, tree)) {
const grpcClsNode = match.captures.grpc_cls;
if (!grpcClsNode) continue;
const grpcMatch = GRPC_SUFFIX_RE.exec(grpcClsNode.text);
if (!grpcMatch) continue;
const serviceName = grpcMatch[1];
out.push({
role: 'consumer',
serviceName,
symbolName: `${serviceName}Stub`,
source: 'java_stub',
confidenceWithProto: 0.75,
confidenceWithoutProto: 0.55,
});
}
return out;
},
};

314
gitnexus/src/core/group/extractors/grpc-patterns/node.ts Normal file
View file

@ -0,0 +1,314 @@
import Parser from 'tree-sitter';
import JavaScript from 'tree-sitter-javascript';
import TypeScript from 'tree-sitter-typescript';
import {
compilePatterns,
runCompiledPatterns,
unquoteLiteral,
type CompiledPatterns,
type LanguagePatterns,
type PatternSpec,
} from '../tree-sitter-scanner.js';
import type { GrpcDetection, GrpcLanguagePlugin } from './types.js';
/**
* Node.js / TypeScript gRPC plugin family. Detects:
* - Provider: NestJS `@GrpcMethod('Service', 'Method')` decorators
* - Consumer: NestJS `@GrpcClient(...) readonly x!: XxxServiceClient`
* - Consumer: `client.getService<X>('AuthService')`
* - Consumer: `new XxxServiceClient(...)` (generated client constructor)
* - Consumer: `new foo.bar.Xxx(...)` when the file uses
* `loadPackageDefinition` (gRPC dynamic proto loader)
*
* As with the HTTP `node.ts`, pattern sources are defined once and
* compiled against three grammar variants (JS / TS / TSX) because
* `Parser.Query` is not portable across grammar objects.
*/
const SERVICE_CLIENT_RE = /^(\w+Service)Client$/;
const CAPITALIZED_SERVICE_RE = /^[A-Z]\w+$/;
// @GrpcMethod('Service', 'Method')
const GRPC_METHOD_SPEC: PatternSpec<Record<string, never>> = {
meta: {},
query: `
(decorator
(call_expression
function: (identifier) @dec (#eq? @dec "GrpcMethod")
arguments: (arguments
. [(string) (template_string)] @service
. [(string) (template_string)] @method)))
`,
};
// @GrpcClient(...) standalone decorator — the plugin walks to the next
// sibling (a field definition) to read its type annotation.
const GRPC_CLIENT_SPEC: PatternSpec<Record<string, never>> = {
meta: {},
query: `
(decorator
(call_expression
function: (identifier) @dec (#eq? @dec "GrpcClient"))) @grpc_client_decorator
`,
};
// `.getService<X>('AuthService')` / `.getService('AuthService')`
const GET_SERVICE_SPEC: PatternSpec<Record<string, never>> = {
meta: {},
query: `
(call_expression
function: (member_expression
property: (property_identifier) @method (#eq? @method "getService"))
arguments: (arguments . [(string) (template_string)] @service))
`,
};
// `new XxxServiceClient(...)` — bare identifier constructor.
const NEW_SIMPLE_CTOR_SPEC: PatternSpec<Record<string, never>> = {
meta: {},
query: `
(new_expression
constructor: (identifier) @ctor)
`,
};
// `new foo.bar.XxxService(...)` — qualified constructor.
const NEW_QUALIFIED_CTOR_SPEC: PatternSpec<Record<string, never>> = {
meta: {},
query: `
(new_expression
constructor: (member_expression
property: (property_identifier) @ctor))
`,
};
// Detect whether the file uses `loadPackageDefinition` (gRPC dynamic
// proto loader). Matches either a bare call or an `obj.loadPackageDefinition(...)`
// call. Plugin gates the qualified-constructor consumer on this —
// structural check avoids materializing `tree.rootNode.text` for every file.
const LOAD_PACKAGE_DEFINITION_SPEC: PatternSpec<Record<string, never>> = {
meta: {},
query: `
(call_expression
function: [
(identifier) @fn (#eq? @fn "loadPackageDefinition")
(member_expression property: (property_identifier) @fn (#eq? @fn "loadPackageDefinition"))
])
`,
};
interface NodeGrpcPatternBundle {
grpcMethod: CompiledPatterns<Record<string, never>>;
grpcClient: CompiledPatterns<Record<string, never>>;
getService: CompiledPatterns<Record<string, never>>;
newSimpleCtor: CompiledPatterns<Record<string, never>>;
newQualifiedCtor: CompiledPatterns<Record<string, never>>;
loadPackageDefinition: CompiledPatterns<Record<string, never>>;
}
function compileBundle(language: unknown, name: string): NodeGrpcPatternBundle {
const mk = (spec: PatternSpec<Record<string, never>>, suffix: string) =>
compilePatterns({
name: `${name}-${suffix}`,
language,
patterns: [spec],
} satisfies LanguagePatterns<Record<string, never>>);
return {
grpcMethod: mk(GRPC_METHOD_SPEC, 'grpc-method'),
grpcClient: mk(GRPC_CLIENT_SPEC, 'grpc-client'),
getService: mk(GET_SERVICE_SPEC, 'get-service'),
newSimpleCtor: mk(NEW_SIMPLE_CTOR_SPEC, 'new-simple-ctor'),
newQualifiedCtor: mk(NEW_QUALIFIED_CTOR_SPEC, 'new-qualified-ctor'),
loadPackageDefinition: mk(LOAD_PACKAGE_DEFINITION_SPEC, 'load-package-definition'),
};
}
const JAVASCRIPT_BUNDLE = compileBundle(JavaScript, 'javascript-grpc');
const TYPESCRIPT_BUNDLE = compileBundle(TypeScript.typescript, 'typescript-grpc');
const TSX_BUNDLE = compileBundle(TypeScript.tsx, 'tsx-grpc');
/**
* Given a `@GrpcClient(...)` decorator node, find the type annotation
* text of the field it decorates (e.g. `AuthServiceClient`).
*
* In tree-sitter-typescript, decorators on class fields can appear in
* two configurations:
* - As a CHILD of `public_field_definition` alongside the field's
* type annotation (the common case for NestJS `@GrpcClient`).
* - As a SIBLING of the field in `class_body` (for method
* decorators, but kept for resilience against grammar variants).
* We walk the parent container and search for a type annotation.
*/
function resolveGrpcClientFieldType(decoratorNode: Parser.SyntaxNode): string | null {
const parent = decoratorNode.parent;
if (!parent) return null;
// Case 1: decorator is a child of the field definition — search
// the parent itself (which is the field definition) for a
// type_annotation child.
if (parent.type === 'public_field_definition' || parent.type.endsWith('field_definition')) {
return findFirstTypeAnnotationText(parent);
}
// Case 2: decorator is a sibling of the field in a class_body — walk
// forward through subsequent siblings until we find a node containing
// a type annotation.
for (let i = 0; i < parent.namedChildCount; i++) {
const child = parent.namedChild(i);
if (child && child.id === decoratorNode.id) {
for (let j = i + 1; j < parent.namedChildCount; j++) {
const next = parent.namedChild(j);
if (!next) continue;
if (next.type === 'decorator') continue;
const typeText = findFirstTypeAnnotationText(next);
if (typeText) return typeText;
return null;
}
return null;
}
}
return null;
}
/**
* Recursively search `node` for the first `type_annotation` child and
* return the text of its inner `type_identifier`, or null. Handles
* both `public_field_definition` and its variants.
*/
function findFirstTypeAnnotationText(node: Parser.SyntaxNode): string | null {
if (node.type === 'type_annotation') {
for (let i = 0; i < node.namedChildCount; i++) {
const child = node.namedChild(i);
if (!child) continue;
if (child.type === 'type_identifier') return child.text;
}
return null;
}
for (let i = 0; i < node.namedChildCount; i++) {
const child = node.namedChild(i);
if (!child) continue;
const found = findFirstTypeAnnotationText(child);
if (found) return found;
}
return null;
}
function scanBundle(bundle: NodeGrpcPatternBundle, tree: Parser.Tree): GrpcDetection[] {
const out: GrpcDetection[] = [];
// ─── Provider: @GrpcMethod('Service', 'Method') ──────────────────
for (const match of runCompiledPatterns(bundle.grpcMethod, tree)) {
const svcNode = match.captures.service;
const methodNode = match.captures.method;
if (!svcNode || !methodNode) continue;
const svc = unquoteLiteral(svcNode.text);
const mth = unquoteLiteral(methodNode.text);
if (!svc || !mth) continue;
out.push({
role: 'provider',
serviceName: svc,
symbolName: `${svc}.${mth}`,
source: 'ts_grpc_method',
methodName: mth,
// @GrpcMethod hard-coded confidence 0.8 in the original code
// regardless of whether the proto map has a match.
confidenceWithProto: 0.8,
confidenceWithoutProto: 0.8,
});
}
// ─── Consumer: @GrpcClient() field with XxxServiceClient type ────
for (const match of runCompiledPatterns(bundle.grpcClient, tree)) {
const decoratorNode = match.captures.grpc_client_decorator;
if (!decoratorNode) continue;
const typeText = resolveGrpcClientFieldType(decoratorNode);
if (!typeText) continue;
const svcMatch = SERVICE_CLIENT_RE.exec(typeText);
if (!svcMatch) continue;
const serviceName = svcMatch[1];
out.push({
role: 'consumer',
serviceName,
symbolName: `${serviceName}Client`,
source: 'ts_grpc_client_decorator',
confidenceWithProto: 0.75,
confidenceWithoutProto: 0.55,
});
}
// ─── Consumer: client.getService<X>('Service') ───────────────────
for (const match of runCompiledPatterns(bundle.getService, tree)) {
const svcNode = match.captures.service;
if (!svcNode) continue;
const svc = unquoteLiteral(svcNode.text);
if (!svc) continue;
out.push({
role: 'consumer',
serviceName: svc,
symbolName: `${svc}Client`,
source: 'ts_client_grpc_get_service',
confidenceWithProto: 0.75,
confidenceWithoutProto: 0.55,
});
}
// ─── Consumer: new XxxServiceClient(...) ─────────────────────────
for (const match of runCompiledPatterns(bundle.newSimpleCtor, tree)) {
const ctorNode = match.captures.ctor;
if (!ctorNode) continue;
const svcMatch = SERVICE_CLIENT_RE.exec(ctorNode.text);
if (!svcMatch) continue;
const serviceName = svcMatch[1];
out.push({
role: 'consumer',
serviceName,
symbolName: `${serviceName}Client`,
source: 'ts_generated_client',
confidenceWithProto: 0.75,
confidenceWithoutProto: 0.55,
});
}
// ─── Consumer: loadPackageDefinition dynamic proto loader ────────
// Only emit when the file uses loadPackageDefinition, otherwise a
// generic `new foo.bar.Something()` in unrelated code would falsely
// register as a gRPC consumer. Check structurally via a dedicated
// query — avoids materializing `tree.rootNode.text` for the whole
// file (expensive on large files).
const usesLoadPackage = runCompiledPatterns(bundle.loadPackageDefinition, tree).length > 0;
if (usesLoadPackage) {
for (const match of runCompiledPatterns(bundle.newQualifiedCtor, tree)) {
const ctorNode = match.captures.ctor;
if (!ctorNode) continue;
if (!CAPITALIZED_SERVICE_RE.test(ctorNode.text)) continue;
out.push({
role: 'consumer',
serviceName: ctorNode.text,
symbolName: `${ctorNode.text}Client`,
source: 'ts_load_package_definition',
confidenceWithProto: 0.75,
confidenceWithoutProto: 0.55,
});
}
}
return out;
}
export const JAVASCRIPT_GRPC_PLUGIN: GrpcLanguagePlugin = {
name: 'javascript-grpc',
language: JavaScript,
scan: (tree) => scanBundle(JAVASCRIPT_BUNDLE, tree),
};
export const TYPESCRIPT_GRPC_PLUGIN: GrpcLanguagePlugin = {
name: 'typescript-grpc',
language: TypeScript.typescript,
scan: (tree) => scanBundle(TYPESCRIPT_BUNDLE, tree),
};
export const TSX_GRPC_PLUGIN: GrpcLanguagePlugin = {
name: 'tsx-grpc',
language: TypeScript.tsx,
scan: (tree) => scanBundle(TSX_BUNDLE, tree),
};

147
gitnexus/src/core/group/extractors/grpc-patterns/proto.ts Normal file
View file

@ -0,0 +1,147 @@
import { createRequire } from 'node:module';
import {
compilePatterns,
runCompiledPatterns,
type CompiledPatterns,
type LanguagePatterns,
} from '../tree-sitter-scanner.js';
import type { GrpcDetection, GrpcLanguagePlugin } from './types.js';
/**
* Protobuf (.proto) tree-sitter plugin for gRPC contract extraction.
*
* Uses `tree-sitter-proto` (coder3101/tree-sitter-proto) as an
* optionalDependency — if the grammar is not installed (e.g. native
* compilation failed on an unusual platform), the plugin exports
* `null` and the orchestrator falls back to the existing manual
* string-sanitizing parser.
*
* The grammar is vendored in `vendor/tree-sitter-proto/` with
* parser.c regenerated against tree-sitter-cli 0.24 (ABI version 14)
* so it is compatible with the project's tree-sitter 0.25 runtime.
*/
const _require = createRequire(import.meta.url);
let ProtoGrammar: unknown = null;
try {
ProtoGrammar = _require('tree-sitter-proto');
} catch {
// Grammar not installed — PROTO_GRPC_PLUGIN will be null.
}
let PACKAGE_PATTERNS: CompiledPatterns<Record<string, never>> | null = null;
let SERVICE_PATTERNS: CompiledPatterns<Record<string, never>> | null = null;
if (ProtoGrammar) {
try {
// Validate that the grammar actually loads end-to-end: compile queries
// AND parse + walk a trivial proto file. tree-sitter's internal
// `initializeLanguageNodeClasses` can fail with a TDZ error in some
// test runners (vitest forks) when SyntaxNode isn't fully initialized
// yet. Catching that here ensures `PROTO_GRPC_PLUGIN` stays null and
// the orchestrator falls back to the manual parser.
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const _Parser = _require('tree-sitter') as any;
// Smoke-test: parse + setLanguage to verify the grammar is
// end-to-end compatible with this tree-sitter runtime.
const _testParser = new _Parser();
_testParser.setLanguage(ProtoGrammar);
_testParser.parse('service X { rpc Y (R) returns (R); }');
PACKAGE_PATTERNS = compilePatterns({
name: 'proto-package',
language: ProtoGrammar,
patterns: [
{
meta: {},
query: `(package (full_ident) @pkg)`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
SERVICE_PATTERNS = compilePatterns({
name: 'proto-service',
language: ProtoGrammar,
patterns: [
{
meta: {},
query: `
(service
(service_name) @service_name
(rpc
(rpc_name) @rpc_name))
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
} catch {
// Compilation failed (grammar ABI mismatch?) — fall back to null.
PACKAGE_PATTERNS = null;
SERVICE_PATTERNS = null;
ProtoGrammar = null;
}
}
function buildPlugin(): GrpcLanguagePlugin | null {
if (!ProtoGrammar || !PACKAGE_PATTERNS || !SERVICE_PATTERNS) return null;
const pkgPatterns = PACKAGE_PATTERNS;
const svcPatterns = SERVICE_PATTERNS;
return {
name: 'proto-grpc',
language: ProtoGrammar,
scan(tree) {
const out: GrpcDetection[] = [];
// Extract `package` declaration (first match wins).
let pkg = '';
for (const match of runCompiledPatterns(pkgPatterns, tree)) {
const pkgNode = match.captures.pkg;
if (pkgNode) {
pkg = pkgNode.text;
break;
}
}
// Extract `service → rpc` pairs. The query returns one match per
// (service, rpc) combination thanks to the nested structure.
for (const match of runCompiledPatterns(svcPatterns, tree)) {
const serviceNode = match.captures.service_name;
const rpcNode = match.captures.rpc_name;
if (!serviceNode || !rpcNode) continue;
const serviceName = serviceNode.text;
const methodName = rpcNode.text;
out.push({
role: 'provider',
serviceName,
symbolName: `${serviceName}.${methodName}`,
source: 'proto',
methodName,
// Proto definitions are the canonical source of truth — always
// high confidence regardless of cross-referencing.
confidenceWithProto: 0.85,
confidenceWithoutProto: 0.85,
});
}
return out;
},
};
}
/**
* The proto plugin, or `null` if tree-sitter-proto is not available.
* The orchestrator checks this at import time and decides whether to
* use the tree-sitter path or the fallback manual parser.
*/
export const PROTO_GRPC_PLUGIN: GrpcLanguagePlugin | null = buildPlugin();
/** The package declaration text from a proto file's tree. */
export function extractPackageFromTree(tree: import('tree-sitter').Tree): string {
if (!PACKAGE_PATTERNS) return '';
for (const match of runCompiledPatterns(PACKAGE_PATTERNS, tree)) {
const pkgNode = match.captures.pkg;
if (pkgNode) return pkgNode.text;
}
return '';
}

77
gitnexus/src/core/group/extractors/grpc-patterns/python.ts Normal file
View file

@ -0,0 +1,77 @@
import Python from 'tree-sitter-python';
import {
compilePatterns,
runCompiledPatterns,
type LanguagePatterns,
} from '../tree-sitter-scanner.js';
import type { GrpcDetection, GrpcLanguagePlugin } from './types.js';
/**
* Python gRPC plugin. Detects:
* - Provider: `add_XxxServicer_to_server(...)` calls (bare identifier
* or qualified attribute form `auth_pb2_grpc.add_XxxServicer_to_server`)
* - Consumer: `XxxStub(channel)` calls (bare or `auth_pb2_grpc.XxxStub`)
*/
const ADD_SERVICER_RE = /^add_(\w+)Servicer_to_server$/;
const STUB_RE = /^(\w+)Stub$/;
/** Reserved names that would produce garbage service names. */
const STUB_IGNORE = new Set(['Mock', 'Test', 'Fake', 'Stub']);
// Any call whose target is either a bare identifier or an attribute
// access (`obj.method`). The plugin filters the function name in JS.
const CALL_PATTERNS = compilePatterns({
name: 'python-grpc-call',
language: Python,
patterns: [
{
meta: {},
query: `
(call
function: [
(identifier) @fn
(attribute attribute: (identifier) @fn)
])
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
export const PYTHON_GRPC_PLUGIN: GrpcLanguagePlugin = {
name: 'python-grpc',
language: Python,
scan(tree) {
const out: GrpcDetection[] = [];
for (const match of runCompiledPatterns(CALL_PATTERNS, tree)) {
const fnNode = match.captures.fn;
if (!fnNode) continue;
const fnText = fnNode.text;
const addServicer = ADD_SERVICER_RE.exec(fnText);
if (addServicer) {
out.push({
role: 'provider',
serviceName: addServicer[1],
symbolName: fnText,
source: 'python_servicer',
confidenceWithProto: 0.8,
confidenceWithoutProto: 0.65,
});
continue;
}
const stubMatch = STUB_RE.exec(fnText);
if (stubMatch && !STUB_IGNORE.has(stubMatch[1])) {
out.push({
role: 'consumer',
serviceName: stubMatch[1],
symbolName: fnText,
source: 'python_stub',
confidenceWithProto: 0.75,
confidenceWithoutProto: 0.55,
});
}
}
return out;
},
};

54
gitnexus/src/core/group/extractors/grpc-patterns/types.ts Normal file
View file

@ -0,0 +1,54 @@
import type Parser from 'tree-sitter';
/**
* Shared types for the grpc-extractor language plugins.
*
* Each plugin lives in its own file (java.ts, go.ts, ...) and owns the
* tree-sitter grammar import + query sources. The top-level
* `grpc-extractor.ts` orchestrator only knows about this type module
* and the plugin registry (`./index.ts`). It MUST NOT import any
* grammar or query text directly.
*/
export type GrpcRole = 'provider' | 'consumer';
/**
* One raw gRPC detection produced by a plugin's `scan()` function. The
* orchestrator uses the proto map to resolve the full package-qualified
* contract id and choose a confidence based on whether the proto was
* found.
*
* Most patterns produce service-level detections; `TS @GrpcMethod` is
* the only pattern that captures an explicit `methodName`, producing
* a method-level contract (`grpc::pkg.Service/Method`).
*/
export interface GrpcDetection {
role: GrpcRole;
/** Short service name, e.g. `"AuthService"`. */
serviceName: string;
/** Symbol name emitted into the contract's symbolRef. */
symbolName: string;
/** Metadata source label (goes into `meta.source`). */
source: string;
/** Explicit method name; set only by TS `@GrpcMethod`. */
methodName?: string;
/** Confidence when the proto map resolves the service. */
confidenceWithProto: number;
/** Confidence when the proto map has no entry. */
confidenceWithoutProto: number;
}
/**
* One language-scoped gRPC plugin. Plugins own the tree-sitter grammar
* and a `scan(tree)` function that returns zero or more
* `GrpcDetection`s. The plugin is free to run multiple compiled query
* bundles and walk the AST to cross-reference captures.
*
* `language` is typed `unknown` for the same reason as in
* `tree-sitter-scanner.ts`.
*/
export interface GrpcLanguagePlugin {
name: string;
language: unknown;
scan(tree: Parser.Tree): GrpcDetection[];
}

224
gitnexus/src/core/group/extractors/http-patterns/go.ts Normal file
View file

@ -0,0 +1,224 @@
import Go from 'tree-sitter-go';
import {
compilePatterns,
runCompiledPatterns,
unquoteLiteral,
type LanguagePatterns,
} from '../tree-sitter-scanner.js';
import type { HttpDetection, HttpLanguagePlugin } from './types.js';
/**
* Go HTTP plugin. Handles:
* - gin / echo / chi framework routing — `r.GET("/path", handler)`
* - net/http stdlib — `http.HandleFunc("/path", handler)`
* - net/http consumer — `http.Get(...)`, `http.NewRequest("METHOD", ...)`
* - resty consumer — `client.R().Delete("/path")`
*/
// ─── Provider: framework routing ──────────────────────────────────────
// Matches `\w+\.GET(...)` etc. (gin, echo, chi all share this shape).
// Captures the HTTP method (field name), path literal, and handler
// identifier passed as the second argument.
const FRAMEWORK_ROUTE_PATTERNS = compilePatterns({
name: 'go-framework-route',
language: Go,
patterns: [
{
meta: {},
query: `
(call_expression
function: (selector_expression
field: (field_identifier) @http_method (#match? @http_method "^(GET|POST|PUT|DELETE|PATCH)$"))
arguments: (argument_list
(interpreted_string_literal) @path
(identifier) @handler))
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
// ─── Provider: net/http `http.HandleFunc("/p", handler)` ─────────────
const HANDLE_FUNC_PATTERNS = compilePatterns({
name: 'go-handle-func',
language: Go,
patterns: [
{
meta: {},
query: `
(call_expression
function: (selector_expression
operand: (identifier) @pkg (#eq? @pkg "http")
field: (field_identifier) @fn (#eq? @fn "HandleFunc"))
arguments: (argument_list
(interpreted_string_literal) @path
(identifier) @handler))
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
// ─── Consumer: net/http stdlib Get / Post / Head ─────────────────────
const HTTP_CLIENT_METHOD_TO_HTTP: Record<string, string> = {
Get: 'GET',
Post: 'POST',
Head: 'GET', // HEAD has no body semantics we care about — treat as GET for contract matching
};
const HTTP_CLIENT_PATTERNS = compilePatterns({
name: 'go-http-client',
language: Go,
patterns: [
{
meta: {},
query: `
(call_expression
function: (selector_expression
operand: (identifier) @pkg (#eq? @pkg "http")
field: (field_identifier) @fn (#match? @fn "^(Get|Post|Head)$"))
arguments: (argument_list . (interpreted_string_literal) @path))
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
// ─── Consumer: net/http `http.NewRequest("METHOD", "/path", ...)` ────
const NEW_REQUEST_PATTERNS = compilePatterns({
name: 'go-new-request',
language: Go,
patterns: [
{
meta: {},
query: `
(call_expression
function: (selector_expression
operand: (identifier) @pkg (#eq? @pkg "http")
field: (field_identifier) @fn (#eq? @fn "NewRequest"))
arguments: (argument_list
.
(interpreted_string_literal) @http_method
(interpreted_string_literal) @path))
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
// ─── Consumer: resty `client.R().Delete("/path")` ─────────────────────
// Matches any chained call whose receiver is `something.R()` and whose
// method name is an HTTP verb. This is how go-resty's fluent API looks.
const RESTY_PATTERNS = compilePatterns({
name: 'go-resty',
language: Go,
patterns: [
{
meta: {},
query: `
(call_expression
function: (selector_expression
operand: (call_expression
function: (selector_expression
field: (field_identifier) @r (#eq? @r "R")))
field: (field_identifier) @http_method (#match? @http_method "^(Get|Post|Put|Delete|Patch)$"))
arguments: (argument_list . (interpreted_string_literal) @path))
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
export const GO_HTTP_PLUGIN: HttpLanguagePlugin = {
name: 'go-http',
language: Go,
scan(tree) {
const out: HttpDetection[] = [];
// Framework providers: r.GET/POST/... with handler identifier
for (const match of runCompiledPatterns(FRAMEWORK_ROUTE_PATTERNS, tree)) {
const methodNode = match.captures.http_method;
const pathNode = match.captures.path;
const handlerNode = match.captures.handler;
if (!methodNode || !pathNode) continue;
const path = unquoteLiteral(pathNode.text);
if (path === null) continue;
out.push({
role: 'provider',
framework: 'go-framework',
method: methodNode.text.toUpperCase(),
path,
name: handlerNode?.text ?? null,
confidence: 0.8,
});
}
// net/http HandleFunc: default method GET
for (const match of runCompiledPatterns(HANDLE_FUNC_PATTERNS, tree)) {
const pathNode = match.captures.path;
const handlerNode = match.captures.handler;
if (!pathNode) continue;
const path = unquoteLiteral(pathNode.text);
if (path === null) continue;
out.push({
role: 'provider',
framework: 'go-stdlib',
method: 'GET',
path,
name: handlerNode?.text ?? null,
confidence: 0.8,
});
}
// net/http client: http.Get/Post/Head
for (const match of runCompiledPatterns(HTTP_CLIENT_PATTERNS, tree)) {
const fnNode = match.captures.fn;
const pathNode = match.captures.path;
if (!fnNode || !pathNode) continue;
const httpMethod = HTTP_CLIENT_METHOD_TO_HTTP[fnNode.text];
if (!httpMethod) continue;
const path = unquoteLiteral(pathNode.text);
if (path === null) continue;
out.push({
role: 'consumer',
framework: 'go-stdlib',
method: httpMethod,
path,
name: null,
confidence: 0.7,
});
}
// net/http NewRequest
for (const match of runCompiledPatterns(NEW_REQUEST_PATTERNS, tree)) {
const methodNode = match.captures.http_method;
const pathNode = match.captures.path;
if (!methodNode || !pathNode) continue;
const method = unquoteLiteral(methodNode.text);
const path = unquoteLiteral(pathNode.text);
if (method === null || path === null) continue;
out.push({
role: 'consumer',
framework: 'go-stdlib',
method: method.toUpperCase(),
path,
name: null,
confidence: 0.7,
});
}
// resty
for (const match of runCompiledPatterns(RESTY_PATTERNS, tree)) {
const methodNode = match.captures.http_method;
const pathNode = match.captures.path;
if (!methodNode || !pathNode) continue;
const path = unquoteLiteral(pathNode.text);
if (path === null) continue;
out.push({
role: 'consumer',
framework: 'go-resty',
method: methodNode.text.toUpperCase(),
path,
name: null,
confidence: 0.7,
});
}
return out;
},
};

50
gitnexus/src/core/group/extractors/http-patterns/index.ts Normal file
View file

@ -0,0 +1,50 @@
import * as path from 'node:path';
import type { HttpLanguagePlugin } from './types.js';
import { JAVA_HTTP_PLUGIN } from './java.js';
import { GO_HTTP_PLUGIN } from './go.js';
import { PYTHON_HTTP_PLUGIN } from './python.js';
import { PHP_HTTP_PLUGIN } from './php.js';
import { JAVASCRIPT_HTTP_PLUGIN, TYPESCRIPT_HTTP_PLUGIN, TSX_HTTP_PLUGIN } from './node.js';
export type { HttpDetection, HttpLanguagePlugin, HttpRole } from './types.js';
/**
* File-extension → HTTP language plugin registry. The top-level
* orchestrator (`http-route-extractor.ts`) looks up the plugin for each
* file it visits and delegates the tree-sitter scanning to the plugin.
*
* Keys are lowercase extensions including the leading dot. To add a
* new language, drop a `http-patterns/<lang>.ts` that exports a
* `HttpLanguagePlugin`, import it here and register the extension(s).
* No edits to `http-route-extractor.ts` are required.
*/
const REGISTRY: Record<string, HttpLanguagePlugin> = {
'.java': JAVA_HTTP_PLUGIN,
'.go': GO_HTTP_PLUGIN,
'.py': PYTHON_HTTP_PLUGIN,
'.php': PHP_HTTP_PLUGIN,
'.js': JAVASCRIPT_HTTP_PLUGIN,
'.jsx': JAVASCRIPT_HTTP_PLUGIN,
'.ts': TYPESCRIPT_HTTP_PLUGIN,
'.tsx': TSX_HTTP_PLUGIN,
};
/**
* Glob for files worth scanning for HTTP routes. Kept alongside the
* registry so adding a new language widens the glob in one edit.
*
* `.vue` / `.svelte` files are intentionally omitted for the source-scan
* path — they need their own grammar-aware extraction and the existing
* regex fallback for them was never very accurate. The graph-assisted
* Strategy A still handles them via the ingestion pipeline.
*/
export const HTTP_SCAN_GLOB = '**/*.{ts,tsx,js,jsx,java,go,py,php}';
/**
* Return the HTTP plugin registered for the given file's extension,
* or `undefined` if the extension is not registered.
*/
export function getPluginForFile(rel: string): HttpLanguagePlugin | undefined {
const ext = path.extname(rel).toLowerCase();
return REGISTRY[ext];
}

267
gitnexus/src/core/group/extractors/http-patterns/java.ts Normal file
View file

@ -0,0 +1,267 @@
import Parser from 'tree-sitter';
import Java from 'tree-sitter-java';
import {
compilePatterns,
runCompiledPatterns,
unquoteLiteral,
type LanguagePatterns,
} from '../tree-sitter-scanner.js';
import type { HttpDetection, HttpLanguagePlugin } from './types.js';
/**
* Java HTTP plugin. Handles:
* - Spring `@RequestMapping` class prefixes + `@(Get|Post|...)Mapping` method annotations
* - Spring `RestTemplate.getForObject/...`, `WebClient.method(HttpMethod.X, ...)`
* - OkHttp `new Request.Builder().url("...")`
*
* The plugin runs two pattern bundles: one to collect class-level
* `@RequestMapping` prefixes keyed by the enclosing class node, and a
* second to match method-level annotations. The `scan` function walks
* up from each matched annotation to find its enclosing class and
* combines the prefix with the method path.
*/
const METHOD_ANNOTATION_TO_HTTP: Record<string, string> = {
GetMapping: 'GET',
PostMapping: 'POST',
PutMapping: 'PUT',
DeleteMapping: 'DELETE',
PatchMapping: 'PATCH',
};
// ─── Provider: Spring class-level @RequestMapping prefix ──────────────
const SPRING_CLASS_PREFIX_PATTERNS = compilePatterns({
name: 'java-spring-class-prefix',
language: Java,
patterns: [
{
meta: {},
query: `
(class_declaration
(modifiers
(annotation
name: (identifier) @ann (#eq? @ann "RequestMapping")
arguments: (annotation_argument_list (string_literal) @prefix)))) @class
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
// ─── Provider: Spring @(Get|Post|...)Mapping method annotations ───────
const SPRING_METHOD_ROUTE_PATTERNS = compilePatterns({
name: 'java-spring-method-route',
language: Java,
patterns: [
{
meta: {},
query: `
(method_declaration
(modifiers
(annotation
name: (identifier) @ann (#match? @ann "^(Get|Post|Put|Delete|Patch)Mapping$")
arguments: (annotation_argument_list (string_literal) @path)))
name: (identifier) @method_name) @method
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
// ─── Consumer: Spring RestTemplate (object-named + method-named) ──────
// RestTemplate.getForObject / getForEntity → GET
// RestTemplate.postForObject / postForEntity → POST
// RestTemplate.put → PUT
// RestTemplate.delete → DELETE
// RestTemplate.patchForObject → PATCH
const REST_TEMPLATE_TO_HTTP: Record<string, string> = {
getForObject: 'GET',
getForEntity: 'GET',
postForObject: 'POST',
postForEntity: 'POST',
put: 'PUT',
delete: 'DELETE',
patchForObject: 'PATCH',
};
interface RestTemplateMeta {
framework: 'spring-rest-template';
}
const REST_TEMPLATE_PATTERNS = compilePatterns({
name: 'java-rest-template',
language: Java,
patterns: [
{
meta: { framework: 'spring-rest-template' },
query: `
(method_invocation
object: (identifier) @obj (#eq? @obj "restTemplate")
name: (identifier) @method
arguments: (argument_list . (string_literal) @path))
`,
},
],
} satisfies LanguagePatterns<RestTemplateMeta>);
// ─── Consumer: Spring WebClient — webClient.method(HttpMethod.X, "path") ─
const WEB_CLIENT_PATTERNS = compilePatterns({
name: 'java-web-client',
language: Java,
patterns: [
{
meta: {},
query: `
(method_invocation
object: (identifier) @obj (#eq? @obj "webClient")
name: (identifier) @method (#eq? @method "method")
arguments: (argument_list
(field_access
object: (identifier) @httpMethodCls (#eq? @httpMethodCls "HttpMethod")
field: (identifier) @http_method)
(string_literal) @path))
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
// ─── Consumer: OkHttp `new Request.Builder().url("path")` ─────────────
// Note: `Request.Builder` is a `scoped_type_identifier` whose text includes
// the dot, so `#eq?` against the literal string matches cleanly (no need
// to escape a regex dot).
const OK_HTTP_PATTERNS = compilePatterns({
name: 'java-okhttp',
language: Java,
patterns: [
{
meta: {},
query: `
(method_invocation
object: (object_creation_expression
type: (scoped_type_identifier) @type (#eq? @type "Request.Builder"))
name: (identifier) @method (#eq? @method "url")
arguments: (argument_list . (string_literal) @path))
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
/**
* Find the nearest enclosing class_declaration ancestor for a node, or
* null if the node is top-level. Tree-sitter's SyntaxNode.parent walks
* one level at a time.
*/
function findEnclosingClass(node: Parser.SyntaxNode): Parser.SyntaxNode | null {
let cur: Parser.SyntaxNode | null = node.parent;
while (cur) {
if (cur.type === 'class_declaration') return cur;
cur = cur.parent;
}
return null;
}
/**
* Join a class-level prefix and a method-level path into a single URL
* path. Mirrors the semantics of the original regex implementation:
* strip trailing slashes on the prefix, then ensure a single slash
* between prefix and method path.
*/
function joinPath(prefix: string, methodPath: string): string {
const cleanPrefix = prefix.replace(/^\/+/, '').replace(/\/+$/, '');
const cleanSub = methodPath.replace(/^\/+/, '');
if (!cleanPrefix) return `/${cleanSub}`;
return `/${cleanPrefix}/${cleanSub}`;
}
export const JAVA_HTTP_PLUGIN: HttpLanguagePlugin = {
name: 'java-http',
language: Java,
scan(tree) {
const out: HttpDetection[] = [];
// ─── Providers: Spring class prefix + method annotations ────────
const prefixByClassId = new Map<number, string>();
for (const match of runCompiledPatterns(SPRING_CLASS_PREFIX_PATTERNS, tree)) {
const prefixNode = match.captures.prefix;
const classNode = match.captures.class;
if (!prefixNode || !classNode) continue;
const prefix = unquoteLiteral(prefixNode.text);
if (prefix !== null) prefixByClassId.set(classNode.id, prefix);
}
for (const match of runCompiledPatterns(SPRING_METHOD_ROUTE_PATTERNS, tree)) {
const annNode = match.captures.ann;
const pathNode = match.captures.path;
const nameNode = match.captures.method_name;
const methodNode = match.captures.method;
if (!annNode || !pathNode || !methodNode) continue;
const httpMethod = METHOD_ANNOTATION_TO_HTTP[annNode.text];
if (!httpMethod) continue;
const rawPath = unquoteLiteral(pathNode.text);
if (rawPath === null) continue;
const enclosingClass = findEnclosingClass(methodNode);
const prefix = enclosingClass ? (prefixByClassId.get(enclosingClass.id) ?? '') : '';
const fullPath = joinPath(prefix, rawPath);
out.push({
role: 'provider',
framework: 'spring',
method: httpMethod,
path: fullPath,
name: nameNode?.text ?? null,
confidence: 0.8,
});
}
// ─── Consumers: RestTemplate ────────────────────────────────────
for (const match of runCompiledPatterns(REST_TEMPLATE_PATTERNS, tree)) {
const methodNode = match.captures.method;
const pathNode = match.captures.path;
if (!methodNode || !pathNode) continue;
const httpMethod = REST_TEMPLATE_TO_HTTP[methodNode.text];
if (!httpMethod) continue;
const path = unquoteLiteral(pathNode.text);
if (path === null) continue;
out.push({
role: 'consumer',
framework: 'spring-rest-template',
method: httpMethod,
path,
name: null,
confidence: 0.7,
});
}
// ─── Consumers: WebClient.method(HttpMethod.X, "path") ──────────
for (const match of runCompiledPatterns(WEB_CLIENT_PATTERNS, tree)) {
const httpMethodNode = match.captures.http_method;
const pathNode = match.captures.path;
if (!httpMethodNode || !pathNode) continue;
const path = unquoteLiteral(pathNode.text);
if (path === null) continue;
out.push({
role: 'consumer',
framework: 'spring-web-client',
method: httpMethodNode.text.toUpperCase(),
path,
name: null,
confidence: 0.7,
});
}
// ─── Consumers: OkHttp Request.Builder().url("path") ────────────
for (const match of runCompiledPatterns(OK_HTTP_PATTERNS, tree)) {
const pathNode = match.captures.path;
if (!pathNode) continue;
const path = unquoteLiteral(pathNode.text);
if (path === null) continue;
out.push({
role: 'consumer',
framework: 'okhttp',
method: 'GET',
path,
name: null,
confidence: 0.7,
});
}
return out;
},
};

373
gitnexus/src/core/group/extractors/http-patterns/node.ts Normal file
View file

@ -0,0 +1,373 @@
import Parser from 'tree-sitter';
import JavaScript from 'tree-sitter-javascript';
import TypeScript from 'tree-sitter-typescript';
import {
compilePatterns,
runCompiledPatterns,
unquoteLiteral,
type CompiledPatterns,
type LanguagePatterns,
type PatternSpec,
} from '../tree-sitter-scanner.js';
import type { HttpDetection, HttpLanguagePlugin } from './types.js';
/**
* Node.js / TypeScript HTTP plugin family. Handles:
* - NestJS `@Controller('prefix')` classes with `@Get(':id')` methods
* - Express `router.get(...)` / `app.post(...)` providers
* - `fetch(url)` / `fetch(url, { method: 'POST' })` consumers
* - `axios.get(url)` / `axios.delete(url)` consumers
*
* Because the JavaScript and TypeScript tree-sitter grammars share
* node type names for every construct we query, pattern sources are
* defined once and compiled against each grammar variant. The plugin
* exports three `HttpLanguagePlugin`s (JS, TS, TSX) that share the
* same `scan` function but bind to different grammars.
*/
// ─── Provider: NestJS — class-level @Controller('prefix') ────────────
// In tree-sitter-typescript decorators are NOT children of
// class_declaration / method_definition — they're siblings in the
// surrounding class_body / program node. We therefore match the
// decorator standalone and walk to its related class/method in JS.
const NEST_CONTROLLER_SPEC: PatternSpec<Record<string, never>> = {
meta: {},
query: `
(decorator
(call_expression
function: (identifier) @dec (#eq? @dec "Controller")
arguments: (arguments . [(string) (template_string)] @prefix))) @ctrl_decorator
`,
};
// ─── Provider: NestJS — method-level @Get/@Post/... decorators ───────
// Matches either `@Get('path')` or `@Get()`. The `@path` capture is
// optional — when the first argument isn't a string, the plugin falls
// back to '/' for the method-level path.
const NEST_METHOD_SPEC: PatternSpec<Record<string, never>> = {
meta: {},
query: `
(decorator
(call_expression
function: (identifier) @dec (#match? @dec "^(Get|Post|Put|Delete|Patch)$")
arguments: (arguments) @args)) @method_decorator
`,
};
// ─── Provider: Express — router.get/app.post/... ─────────────────────
const EXPRESS_SPEC: PatternSpec<Record<string, never>> = {
meta: {},
query: `
(call_expression
function: (member_expression
object: (identifier) @obj (#match? @obj "^(router|app)$")
property: (property_identifier) @http_method (#match? @http_method "^(get|post|put|delete|patch)$"))
arguments: (arguments . [(string) (template_string)] @path))
`,
};
// ─── Consumer: fetch(url) with NO options ─────────────────────────────
const FETCH_NO_OPTIONS_SPEC: PatternSpec<Record<string, never>> = {
meta: {},
query: `
(call_expression
function: (identifier) @fn (#eq? @fn "fetch")
arguments: (arguments . [(string) (template_string)] @path .))
`,
};
// ─── Consumer: fetch(url, { method: 'X', ... }) ──────────────────────
const FETCH_WITH_OPTIONS_SPEC: PatternSpec<Record<string, never>> = {
meta: {},
query: `
(call_expression
function: (identifier) @fn (#eq? @fn "fetch")
arguments: (arguments
. [(string) (template_string)] @path
(object
(pair
key: (property_identifier) @key (#eq? @key "method")
value: (string) @http_method))))
`,
};
// ─── Consumer: axios.get/post/... ────────────────────────────────────
const AXIOS_SPEC: PatternSpec<Record<string, never>> = {
meta: {},
query: `
(call_expression
function: (member_expression
object: (identifier) @obj (#eq? @obj "axios")
property: (property_identifier) @http_method (#match? @http_method "^(get|post|put|delete|patch)$"))
arguments: (arguments . [(string) (template_string)] @path))
`,
};
interface NodePatternBundle {
controller: CompiledPatterns<Record<string, never>>;
methodDecorator: CompiledPatterns<Record<string, never>>;
express: CompiledPatterns<Record<string, never>>;
fetchNoOptions: CompiledPatterns<Record<string, never>>;
fetchWithOptions: CompiledPatterns<Record<string, never>>;
axios: CompiledPatterns<Record<string, never>>;
}
function compileBundle(language: unknown, name: string): NodePatternBundle {
const mk = (spec: PatternSpec<Record<string, never>>, suffix: string) =>
compilePatterns({
name: `${name}-${suffix}`,
language,
patterns: [spec],
} satisfies LanguagePatterns<Record<string, never>>);
return {
controller: mk(NEST_CONTROLLER_SPEC, 'nest-controller'),
methodDecorator: mk(NEST_METHOD_SPEC, 'nest-method-decorator'),
express: mk(EXPRESS_SPEC, 'express'),
fetchNoOptions: mk(FETCH_NO_OPTIONS_SPEC, 'fetch-no-options'),
fetchWithOptions: mk(FETCH_WITH_OPTIONS_SPEC, 'fetch-with-options'),
axios: mk(AXIOS_SPEC, 'axios'),
};
}
const JAVASCRIPT_BUNDLE = compileBundle(JavaScript, 'javascript-http');
const TYPESCRIPT_BUNDLE = compileBundle(TypeScript.typescript, 'typescript-http');
const TSX_BUNDLE = compileBundle(TypeScript.tsx, 'tsx-http');
const NEST_DECORATOR_TO_HTTP: Record<string, string> = {
Get: 'GET',
Post: 'POST',
Put: 'PUT',
Delete: 'DELETE',
Patch: 'PATCH',
};
/**
* Find the nearest enclosing class_declaration for a node, or null.
*/
function findEnclosingClass(node: Parser.SyntaxNode): Parser.SyntaxNode | null {
let cur: Parser.SyntaxNode | null = node.parent;
while (cur) {
if (cur.type === 'class_declaration') return cur;
cur = cur.parent;
}
return null;
}
function joinPath(prefix: string, sub: string): string {
const cleanPrefix = prefix.replace(/^\/+/, '').replace(/\/+$/, '');
const cleanSub = sub.replace(/^\/+/, '');
if (!cleanPrefix) return `/${cleanSub}`;
return `/${cleanPrefix}/${cleanSub}`;
}
/**
* For a standalone `decorator` node (child of class_body / program),
* find the related `class_declaration` node that it decorates. In
* tree-sitter-typescript the decorator is placed before the class
* declaration as a sibling (when decorating a class) or inside the
* class_body before a method_definition (when decorating a method);
* we walk the parent chain until we find the enclosing class.
*/
function findDecoratedClass(decoratorNode: Parser.SyntaxNode): Parser.SyntaxNode | null {
const parent = decoratorNode.parent;
if (!parent) return null;
// Case 1: decorator is a sibling of the class_declaration at program /
// export_statement level. Walk forward through siblings until we find
// the class_declaration this decorator belongs to.
for (let i = 0; i < parent.namedChildCount; i++) {
const child = parent.namedChild(i);
if (child && child.id === decoratorNode.id) {
for (let j = i + 1; j < parent.namedChildCount; j++) {
const next = parent.namedChild(j);
if (!next) continue;
if (next.type === 'decorator') continue; // adjacent decorators stack
if (next.type === 'class_declaration') return next;
if (next.type === 'export_statement') {
// `export class Foo { ... }` wraps the declaration.
for (let k = 0; k < next.namedChildCount; k++) {
const inner = next.namedChild(k);
if (inner?.type === 'class_declaration') return inner;
}
}
break;
}
break;
}
}
// Case 2: decorator is inside a class_body (decorating a method) —
// walk up to the enclosing class_declaration.
return findEnclosingClass(decoratorNode);
}
/**
* For a method-level decorator node (child of class_body before a
* method_definition), find the method_definition it decorates.
*/
function findDecoratedMethod(decoratorNode: Parser.SyntaxNode): Parser.SyntaxNode | null {
const parent = decoratorNode.parent;
if (!parent || parent.type !== 'class_body') return null;
for (let i = 0; i < parent.namedChildCount; i++) {
const child = parent.namedChild(i);
if (child && child.id === decoratorNode.id) {
for (let j = i + 1; j < parent.namedChildCount; j++) {
const next = parent.namedChild(j);
if (!next) continue;
if (next.type === 'decorator') continue;
if (next.type === 'method_definition') return next;
return null;
}
return null;
}
}
return null;
}
function scanBundle(bundle: NodePatternBundle, tree: Parser.Tree): HttpDetection[] {
const out: HttpDetection[] = [];
// NestJS: collect `@Controller('prefix')` class decorators, keyed by
// the `class_declaration` they decorate.
const prefixByClassId = new Map<number, string>();
for (const match of runCompiledPatterns(bundle.controller, tree)) {
const prefixNode = match.captures.prefix;
const decoratorNode = match.captures.ctrl_decorator;
if (!prefixNode || !decoratorNode) continue;
const prefix = unquoteLiteral(prefixNode.text);
if (prefix === null) continue;
const classNode = findDecoratedClass(decoratorNode);
if (!classNode) continue;
prefixByClassId.set(classNode.id, prefix);
}
// NestJS: method-level @Get/@Post/... decorators. The decorator's
// arguments list may be empty (`@Get()`), a string (`@Get('path')`),
// or something else (which we skip).
for (const match of runCompiledPatterns(bundle.methodDecorator, tree)) {
const decNode = match.captures.dec;
const argsNode = match.captures.args;
const decoratorNode = match.captures.method_decorator;
if (!decNode || !argsNode || !decoratorNode) continue;
const httpMethod = NEST_DECORATOR_TO_HTTP[decNode.text];
if (!httpMethod) continue;
const methodNode = findDecoratedMethod(decoratorNode);
if (!methodNode) continue;
const enclosingClass = findEnclosingClass(methodNode);
// Only emit NestJS detections when the class actually has a
// @Controller decorator — without it, the match is almost certainly
// something else (e.g. an unrelated library using similar names).
if (!enclosingClass || !prefixByClassId.has(enclosingClass.id)) continue;
const prefix = prefixByClassId.get(enclosingClass.id) ?? '';
let rawPath = '/';
const firstArg = argsNode.namedChild(0);
if (firstArg && (firstArg.type === 'string' || firstArg.type === 'template_string')) {
const unquoted = unquoteLiteral(firstArg.text);
if (unquoted !== null) rawPath = unquoted;
}
// Get the method name from the decorated method_definition.
const methodNameNode = methodNode.childForFieldName('name');
const name = methodNameNode?.text ?? null;
out.push({
role: 'provider',
framework: 'nest',
method: httpMethod,
path: joinPath(prefix, rawPath),
name,
confidence: 0.8,
});
}
// Express: router/app.<verb>(...)
for (const match of runCompiledPatterns(bundle.express, tree)) {
const methodNode = match.captures.http_method;
const pathNode = match.captures.path;
if (!methodNode || !pathNode) continue;
const path = unquoteLiteral(pathNode.text);
if (path === null) continue;
out.push({
role: 'provider',
framework: 'express',
method: methodNode.text.toUpperCase(),
path,
name: 'handler',
confidence: 0.8,
});
}
// Consumer: fetch with options { method: 'X' }
const fetchSeen = new Set<number>();
for (const match of runCompiledPatterns(bundle.fetchWithOptions, tree)) {
const pathNode = match.captures.path;
const methodNode = match.captures.http_method;
if (!pathNode || !methodNode) continue;
const path = unquoteLiteral(pathNode.text);
const method = unquoteLiteral(methodNode.text);
if (path === null || method === null) continue;
fetchSeen.add(pathNode.id);
out.push({
role: 'consumer',
framework: 'fetch',
method: method.toUpperCase(),
path,
name: null,
confidence: 0.7,
});
}
// Consumer: plain fetch(path) — default GET. Skip path nodes we already
// matched with the options variant so we don't double-emit.
for (const match of runCompiledPatterns(bundle.fetchNoOptions, tree)) {
const pathNode = match.captures.path;
if (!pathNode) continue;
if (fetchSeen.has(pathNode.id)) continue;
const path = unquoteLiteral(pathNode.text);
if (path === null) continue;
out.push({
role: 'consumer',
framework: 'fetch',
method: 'GET',
path,
name: null,
confidence: 0.7,
});
}
// Consumer: axios.<verb>(url)
for (const match of runCompiledPatterns(bundle.axios, tree)) {
const methodNode = match.captures.http_method;
const pathNode = match.captures.path;
if (!methodNode || !pathNode) continue;
const path = unquoteLiteral(pathNode.text);
if (path === null) continue;
out.push({
role: 'consumer',
framework: 'axios',
method: methodNode.text.toUpperCase(),
path,
name: null,
confidence: 0.7,
});
}
return out;
}
export const JAVASCRIPT_HTTP_PLUGIN: HttpLanguagePlugin = {
name: 'javascript-http',
language: JavaScript,
scan: (tree) => scanBundle(JAVASCRIPT_BUNDLE, tree),
};
export const TYPESCRIPT_HTTP_PLUGIN: HttpLanguagePlugin = {
name: 'typescript-http',
language: TypeScript.typescript,
scan: (tree) => scanBundle(TYPESCRIPT_BUNDLE, tree),
};
export const TSX_HTTP_PLUGIN: HttpLanguagePlugin = {
name: 'tsx-http',
language: TypeScript.tsx,
scan: (tree) => scanBundle(TSX_BUNDLE, tree),
};

79
gitnexus/src/core/group/extractors/http-patterns/php.ts Normal file
View file

@ -0,0 +1,79 @@
import PHP from 'tree-sitter-php';
import {
compilePatterns,
runCompiledPatterns,
unquoteLiteral,
type LanguagePatterns,
} from '../tree-sitter-scanner.js';
import type { HttpDetection, HttpLanguagePlugin } from './types.js';
/**
* PHP HTTP plugin — Laravel `Route::get/post/...` declarations.
*
* The pipeline already uses `PHP.php_only` for ingesting plain `.php`
* files (see `core/tree-sitter/parser-loader.ts`), and we do the same
* here so Laravel route files are parsed with the right grammar dialect.
*/
const LARAVEL_PATTERNS = compilePatterns({
name: 'php-laravel',
language: PHP.php_only,
patterns: [
{
meta: {},
query: `
(scoped_call_expression
scope: (name) @scope (#eq? @scope "Route")
name: (name) @method (#match? @method "^(get|post|put|delete|patch)$")
arguments: (arguments . (argument (string) @path)))
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
/**
* Extract the inner text of a PHP `string` node. The tree-sitter-php
* grammar wraps single / double-quoted literals differently depending
* on content; we try both the raw `text` (with quotes) through
* `unquoteLiteral`, and a fallback via the `string_value` / `string_content`
* child nodes.
*/
function phpStringText(node: import('tree-sitter').SyntaxNode): string | null {
// Most single-quoted strings expose their inner content through the
// full node text (including quotes), which unquoteLiteral strips.
const direct = unquoteLiteral(node.text);
if (direct !== null && direct !== node.text) return direct;
// Fall back to child string_content / string_value node if present.
for (const child of node.children) {
if (child.type === 'string_content' || child.type === 'string_value') {
return child.text;
}
}
return direct;
}
export const PHP_HTTP_PLUGIN: HttpLanguagePlugin = {
name: 'php-http',
language: PHP.php_only,
scan(tree) {
const out: HttpDetection[] = [];
for (const match of runCompiledPatterns(LARAVEL_PATTERNS, tree)) {
const methodNode = match.captures.method;
const pathNode = match.captures.path;
if (!methodNode || !pathNode) continue;
const path = phpStringText(pathNode);
if (path === null) continue;
out.push({
role: 'provider',
framework: 'laravel',
method: methodNode.text.toUpperCase(),
path,
name: 'route',
confidence: 0.8,
});
}
return out;
},
};

142
gitnexus/src/core/group/extractors/http-patterns/python.ts Normal file
View file

@ -0,0 +1,142 @@
import Python from 'tree-sitter-python';
import {
compilePatterns,
runCompiledPatterns,
unquoteLiteral,
type LanguagePatterns,
} from '../tree-sitter-scanner.js';
import type { HttpDetection, HttpLanguagePlugin } from './types.js';
/**
* Python HTTP plugin. Handles:
* - FastAPI `@app.get("/path")` provider decorators
* - `requests.get/post/...("url")` consumer calls
* - Generic `requests.request("METHOD", "url")` consumer calls
*/
const FASTAPI_VERBS: Record<string, string> = {
get: 'GET',
post: 'POST',
put: 'PUT',
delete: 'DELETE',
patch: 'PATCH',
};
// ─── Provider: FastAPI @app.get/... ──────────────────────────────────
const FASTAPI_PATTERNS = compilePatterns({
name: 'python-fastapi',
language: Python,
patterns: [
{
meta: {},
query: `
(decorator
(call
function: (attribute
object: (identifier) @obj (#eq? @obj "app")
attribute: (identifier) @method (#match? @method "^(get|post|put|delete|patch)$"))
arguments: (argument_list . (string) @path)))
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
// ─── Consumer: requests.get/post/... ──────────────────────────────────
const REQUESTS_VERB_PATTERNS = compilePatterns({
name: 'python-requests-verb',
language: Python,
patterns: [
{
meta: {},
query: `
(call
function: (attribute
object: (identifier) @obj (#eq? @obj "requests")
attribute: (identifier) @method (#match? @method "^(get|post|put|delete|patch)$"))
arguments: (argument_list . (string) @path))
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
// ─── Consumer: requests.request("METHOD", "url") ─────────────────────
const REQUESTS_GENERIC_PATTERNS = compilePatterns({
name: 'python-requests-generic',
language: Python,
patterns: [
{
meta: {},
query: `
(call
function: (attribute
object: (identifier) @obj (#eq? @obj "requests")
attribute: (identifier) @method (#eq? @method "request"))
arguments: (argument_list . (string) @http_method (string) @path))
`,
},
],
} satisfies LanguagePatterns<Record<string, never>>);
export const PYTHON_HTTP_PLUGIN: HttpLanguagePlugin = {
name: 'python-http',
language: Python,
scan(tree) {
const out: HttpDetection[] = [];
// Providers: FastAPI
for (const match of runCompiledPatterns(FASTAPI_PATTERNS, tree)) {
const methodNode = match.captures.method;
const pathNode = match.captures.path;
if (!methodNode || !pathNode) continue;
const httpMethod = FASTAPI_VERBS[methodNode.text];
if (!httpMethod) continue;
const path = unquoteLiteral(pathNode.text);
if (path === null) continue;
out.push({
role: 'provider',
framework: 'fastapi',
method: httpMethod,
path,
name: null,
confidence: 0.8,
});
}
// Consumers: requests.<verb>
for (const match of runCompiledPatterns(REQUESTS_VERB_PATTERNS, tree)) {
const methodNode = match.captures.method;
const pathNode = match.captures.path;
if (!methodNode || !pathNode) continue;
const path = unquoteLiteral(pathNode.text);
if (path === null) continue;
out.push({
role: 'consumer',
framework: 'python-requests',
method: methodNode.text.toUpperCase(),
path,
name: null,
confidence: 0.7,
});
}
// Consumers: requests.request("METHOD", "url")
for (const match of runCompiledPatterns(REQUESTS_GENERIC_PATTERNS, tree)) {
const methodNode = match.captures.http_method;
const pathNode = match.captures.path;
if (!methodNode || !pathNode) continue;
const methodRaw = unquoteLiteral(methodNode.text);
const path = unquoteLiteral(pathNode.text);
if (methodRaw === null || path === null) continue;
out.push({
role: 'consumer',
framework: 'python-requests',
method: methodRaw.toUpperCase(),
path,
name: null,
confidence: 0.7,
});
}
return out;
},
};

65
gitnexus/src/core/group/extractors/http-patterns/types.ts Normal file
View file

@ -0,0 +1,65 @@
import type Parser from 'tree-sitter';
/**
* Shared types for the http-route-extractor language plugins.
*
* Each plugin lives in its own file (java.ts, node.ts, ...) and owns
* the tree-sitter grammar import + queries. The top-level
* `http-route-extractor.ts` orchestrator only knows about this type
* module and the plugin registry (`./index.ts`). It MUST NOT import
* any grammar or query text directly — language-specific knowledge
* belongs in the plugins.
*/
export type HttpRole = 'provider' | 'consumer';
/**
* One raw HTTP detection produced by a plugin's `scan()` function. The
* orchestrator converts this into a full `ExtractedContract` by running
* path normalization and building the contract id.
*
* `path` is the raw literal string as it appeared in source (with
* `${...}` template placeholders still in place); the orchestrator
* runs the appropriate normalizer for provider vs. consumer paths.
*/
export interface HttpDetection {
role: HttpRole;
/** Short framework label, e.g. `'spring'`, `'nest'`, `'express'`. */
framework: string;
/** HTTP method in upper case (`'GET'`, `'POST'`, ...). */
method: string;
/** Raw path literal as seen in source (template placeholders intact). */
path: string;
/**
* Symbol name of the handler (for providers) or calling function
* (for consumers) when the plugin can determine it structurally.
* Null when no good candidate is available.
*/
name: string | null;
/** Confidence in (0, 1]. Source-scan plugins typically use 0.7–0.8. */
confidence: number;
}
/**
* One language-scoped HTTP plugin. The plugin owns the tree-sitter
* grammar and the `scan` function that translates a parsed tree into
* zero or more `HttpDetection`s. Plugins are free to run multiple
* compiled pattern bundles internally (see the shared scanner's
* `runCompiledPatterns` helper).
*
* `language` is typed as `unknown` for the same reason as
* `LanguagePatterns.language` in `tree-sitter-scanner.ts` — the
* grammar modules export different shapes.
*/
export interface HttpLanguagePlugin {
/** Human-readable plugin name for diagnostics. */
name: string;
/** tree-sitter grammar object (passed to the shared parser). */
language: unknown;
/**
* Scan a parsed tree and return zero or more HTTP detections. Plugins
* must not throw — they should swallow per-match errors so a single
* malformed construct does not abort the whole file.
*/
scan(tree: Parser.Tree): HttpDetection[];
}

486
gitnexus/src/core/group/extractors/http-route-extractor.ts
View file

@ -1,8 +1,34 @@
import * as fs from 'node:fs';
import * as path from 'node:path';
import { glob } from 'glob';
import Parser from 'tree-sitter';
import type { ContractExtractor, CypherExecutor } from '../contract-extractor.js';
import type { ExtractedContract, RepoHandle } from '../types.js';
import { readSafe } from './fs-utils.js';
import { getPluginForFile, HTTP_SCAN_GLOB, type HttpDetection } from './http-patterns/index.js';
/**
* Language-agnostic orchestrator for HTTP route (provider + consumer)
* contract extraction. Two strategies, in order of preference per role:
*
* 1. **Graph-assisted (Strategy A)** — if a per-repo LadybugDB executor
* is available, read `HANDLES_ROUTE` / `FETCHES` Cypher edges that
* the ingestion pipeline already produced via tree-sitter. This is
* the preferred path because the graph has richer symbol metadata
* (real uids, class/method structure, etc.).
*
* 2. **Source-scan fallback (Strategy B)** — parse files directly with
* the per-language plugin registry in `./http-patterns/`. Used when
* the graph has no routes/fetches for this repo (e.g. a repo that
* hasn't been indexed yet, or whose indexer doesn't know the
* framework). Each plugin owns its tree-sitter grammar and query
* sources — this orchestrator imports NO grammars or query strings.
*
* Adding a new language for Strategy B is a one-file edit in
* `http-patterns/index.ts`: register a new `HttpLanguagePlugin` and
* widen `HTTP_SCAN_GLOB` if needed.
*/
// ─── Graph-assisted queries ──────────────────────────────────────────
const HANDLES_ROUTE_QUERY = `
MATCH (handlerFile:File)-[r:CodeRelation {type: 'HANDLES_ROUTE'}]->(route:Route)
@ -23,14 +49,56 @@ WHERE sym.startLine IS NOT NULL
RETURN sym.id AS uid, sym.name AS name, sym.filePath AS filePath, labels(sym) AS labels
ORDER BY sym.startLine`;
// ─── Path normalization (shared between provider / consumer paths) ──
/**
* Canonicalize a provider-side HTTP path for contract-id generation:
* - strip query string
* - lower-case
* - drop trailing slash
* - collapse `:id`, `{id}`, `[id]` path params into a single `{param}`
*/
export function normalizeHttpPath(p: string): string {
let s = p.trim().split('?')[0].toLowerCase().replace(/\/+$/, '');
s = s.replace(/:\w+/g, '{param}');
s = s.replace(/\{[^}]+\}/g, '{param}');
s = s.replace(/\[[^\]]+\]/g, '{param}');
return s;
// Preserve root: after stripping trailing slashes, the root "/"
// collapses to "" which would produce malformed contract ids like
// `http::GET::`. Restore a single slash for the root case.
return s === '' ? '/' : s;
}
/**
* Consumer-side normalization is more aggressive:
* - template literals (`${x}`) → `{param}`
* - strip protocol + host if the URL is absolute
* - numeric segments → `{param}` (so `/api/orders/42` → `/api/orders/{param}`)
*/
function normalizeConsumerPath(url: string): string {
const templated = url.replace(/\$\{[^}]+\}/g, '{param}').trim();
let pathOnly = templated;
if (/^https?:\/\//i.test(templated)) {
try {
pathOnly = new URL(templated).pathname;
} catch {
pathOnly = templated.replace(/^https?:\/\/[^/]+/i, '');
}
}
const normalized = normalizeHttpPath(pathOnly || '/');
const segments = normalized
.split('/')
.filter(Boolean)
.map((segment) => (/^\d+$/.test(segment) ? '{param}' : segment));
return `/${segments.join('/')}`.replace(/\/+$/, '') || '/';
}
function contractIdFor(method: string, pathNorm: string): string {
return `http::${method.toUpperCase()}::${pathNorm}`;
}
// ─── Graph row helpers ───────────────────────────────────────────────
function methodFromRouteReason(reason: string): string | null {
const r = reason || '';
if (/GetMapping|decorator-Get/i.test(r)) return 'GET';
@ -41,50 +109,6 @@ function methodFromRouteReason(reason: string): string | null {
return null;
}
function contractIdFor(method: string, pathNorm: string): string {
return `http::${method.toUpperCase()}::${pathNorm}`;
}
function readSafe(repoPath: string, rel: string): string | null {
const abs = path.resolve(repoPath, rel);
const base = path.resolve(repoPath);
const relToBase = path.relative(base, abs);
if (relToBase.startsWith('..') || path.isAbsolute(relToBase)) return null;
try {
return fs.readFileSync(abs, 'utf-8');
} catch {
return null;
}
}
function pickJavaHandlerName(
content: string,
routePath: string,
httpMethod: string,
): string | null {
const tail = routePath.split('/').filter(Boolean).pop() || '';
const mapNames: Record<string, string> = {
GET: 'GetMapping',
POST: 'PostMapping',
PUT: 'PutMapping',
DELETE: 'DeleteMapping',
PATCH: 'PatchMapping',
};
const ann = mapNames[httpMethod] || 'GetMapping';
const lines = content.split(/\r?\n/);
for (let i = 0; i < lines.length; i++) {
const line = lines[i];
if (!line.includes(`@${ann}`)) continue;
if (!line.includes(`"${tail}"`) && !line.includes(`'${tail}'`) && tail && !line.includes(tail))
continue;
for (let j = i + 1; j < Math.min(i + 8, lines.length); j++) {
const m = lines[j].match(/(?:public|protected|private)\s+[\w<>,\s\[\]]+\s+(\w+)\s*\(/);
if (m) return m[1];
}
}
return null;
}
function pickSymbolUid(
rows: Record<string, unknown>[],
preferredName: string | null,
@ -114,6 +138,8 @@ function pickSymbolUid(
};
}
// ─── Orchestrator ────────────────────────────────────────────────────
export class HttpRouteExtractor implements ContractExtractor {
type = 'http' as const;
@ -124,20 +150,76 @@ export class HttpRouteExtractor implements ContractExtractor {
async extract(
dbExecutor: CypherExecutor | null,
repoPath: string,
repo: RepoHandle,
_repo: RepoHandle,
): Promise<ExtractedContract[]> {
const graphP = dbExecutor != null ? await this.extractProvidersGraph(dbExecutor, repoPath) : [];
const providers = graphP.length > 0 ? graphP : await this.extractProvidersSourceScan(repoPath);
// Parse each file at most once and reuse the plugin results across
// both graph-assisted enrichment and source-scan emission.
const parser = new Parser();
const cachedDetections = new Map<string, HttpDetection[]>();
const getDetections = (rel: string): HttpDetection[] => {
const cached = cachedDetections.get(rel);
if (cached) return cached;
const plugin = getPluginForFile(rel);
if (!plugin) {
cachedDetections.set(rel, []);
return [];
}
const content = readSafe(repoPath, rel);
if (!content) {
cachedDetections.set(rel, []);
return [];
}
try {
parser.setLanguage(plugin.language);
const tree = parser.parse(content);
const detections = plugin.scan(tree);
cachedDetections.set(rel, detections);
return detections;
} catch {
cachedDetections.set(rel, []);
return [];
}
};
const graphC = dbExecutor != null ? await this.extractConsumersGraph(dbExecutor, repoPath) : [];
const consumers = graphC.length > 0 ? graphC : await this.extractConsumersSourceScan(repoPath);
// Glob the source-scan file list at most once per extract() —
// both provider and consumer fallback paths share the same list.
let scannedFiles: string[] | null = null;
const getScannedFiles = async (): Promise<string[]> => {
if (scannedFiles) return scannedFiles;
scannedFiles = await this.scanFiles(repoPath);
return scannedFiles;
};
const graphProviders =
dbExecutor != null ? await this.extractProvidersGraph(dbExecutor, getDetections) : [];
const providers =
graphProviders.length > 0
? graphProviders
: this.extractProvidersSourceScan(await getScannedFiles(), getDetections);
const graphConsumers =
dbExecutor != null ? await this.extractConsumersGraph(dbExecutor, getDetections) : [];
const consumers =
graphConsumers.length > 0
? graphConsumers
: this.extractConsumersSourceScan(await getScannedFiles(), getDetections);
return [...providers, ...consumers];
}
private async scanFiles(repoPath: string): Promise<string[]> {
return glob(HTTP_SCAN_GLOB, {
cwd: repoPath,
ignore: ['**/node_modules/**', '**/.git/**', '**/dist/**', '**/build/**', '**/vendor/**'],
nodir: true,
});
}
// ─── Graph-assisted providers ──────────────────────────────────────
private async extractProvidersGraph(
db: CypherExecutor,
repoPath: string,
getDetections: (rel: string) => HttpDetection[],
): Promise<ExtractedContract[]> {
const out: ExtractedContract[] = [];
let rows: Record<string, unknown>[];
@ -152,16 +234,26 @@ export class HttpRouteExtractor implements ContractExtractor {
const routePath = String(row.routePath ?? '');
const routeSource = String(row.routeSource ?? row.routeReason ?? '');
let method = methodFromRouteReason(routeSource);
const content = readSafe(repoPath, filePath);
if (!method && content) {
method = this.inferMethodFromFileScan(content, routePath, 'provider');
// Look up handler name (and backfill method if missing) from the
// plugin's scan of the handler file. This replaces the old
// regex-based `inferMethodFromFileScan` and `pickJavaHandlerName`
// helpers — tree-sitter gives both pieces of information
// structurally. Always run the lookup: even when method is set by
// `methodFromRouteReason`, we still need the handler name.
const detections = filePath ? getDetections(filePath) : [];
const providerDetections = detections.filter((d) => d.role === 'provider');
let handlerName: string | null = null;
const normalizedRoute = normalizeHttpPath(routePath);
const match = providerDetections.find((d) => normalizeHttpPath(d.path) === normalizedRoute);
if (match) {
if (!method) method = match.method;
handlerName = match.name;
}
if (!method) method = 'GET';
const pathNorm = normalizeHttpPath(routePath);
const cid = contractIdFor(method, pathNorm);
const handlerName =
content && routePath ? pickJavaHandlerName(content, routePath, method) : null;
let symbolUid = '';
let symbolName = path.basename(filePath) || 'handler';
@ -201,157 +293,44 @@ export class HttpRouteExtractor implements ContractExtractor {
return out;
}
private inferMethodFromFileScan(
content: string,
routePath: string,
_role: string,
): string | null {
const tail = routePath.split('/').filter(Boolean).pop() || '';
for (const m of ['GET', 'POST', 'PUT', 'DELETE', 'PATCH'] as const) {
const mapNames: Record<string, string> = {
GET: 'GetMapping',
POST: 'PostMapping',
PUT: 'PutMapping',
DELETE: 'DeleteMapping',
PATCH: 'PatchMapping',
};
if (
content.includes(`@${mapNames[m]}`) &&
(content.includes(tail) || routePath.includes(tail))
) {
return m;
}
}
return null;
}
// ─── Source-scan providers ─────────────────────────────────────────
private async extractProvidersSourceScan(repoPath: string): Promise<ExtractedContract[]> {
const files = await glob('**/*.{ts,tsx,js,jsx,java,vue,svelte,php,py}', {
cwd: repoPath,
ignore: ['**/node_modules/**', '**/.git/**', '**/dist/**', '**/build/**'],
nodir: true,
});
private extractProvidersSourceScan(
files: string[],
getDetections: (rel: string) => HttpDetection[],
): ExtractedContract[] {
const out: ExtractedContract[] = [];
for (const rel of files) {
const content = readSafe(repoPath, rel);
if (!content) continue;
out.push(...this.scanSpringProviders(content, rel));
out.push(...this.scanExpressProviders(content, rel));
out.push(...this.scanLaravelProviders(content, rel));
out.push(...this.scanFastApiProviders(content, rel));
const detections = getDetections(rel);
for (const d of detections) {
if (d.role !== 'provider') continue;
const pathNorm = normalizeHttpPath(d.path);
out.push({
contractId: contractIdFor(d.method, pathNorm),
type: 'http',
role: 'provider',
symbolUid: '',
symbolRef: { filePath: rel, name: d.name ?? 'handler' },
symbolName: d.name ?? 'handler',
confidence: d.confidence,
meta: {
method: d.method,
path: pathNorm,
pathSegments: pathNorm.split('/').filter(Boolean),
extractionStrategy: 'source_scan',
framework: d.framework,
},
});
}
}
return this.dedupeContracts(out);
}
private dedupeContracts(items: ExtractedContract[]): ExtractedContract[] {
const seen = new Set<string>();
const out: ExtractedContract[] = [];
for (const c of items) {
const k = `${c.contractId}|${c.symbolRef.filePath}|${c.symbolRef.name}`;
if (seen.has(k)) continue;
seen.add(k);
out.push(c);
}
return out;
}
private scanSpringProviders(content: string, filePath: string): ExtractedContract[] {
const out: ExtractedContract[] = [];
// Skip Feign/client interfaces — annotated methods in interfaces are
// consumers (Feign, JAX-RS proxies), not provider endpoints.
// Anchored to line start (with optional access modifier) so we do not
// match "interface" inside comments or string literals.
if (
/^\s*(?:public\s+)?interface\s+\w+/m.test(content) &&
!/@(?:Rest)?Controller\b/.test(content)
) {
return out;
}
let classPrefix = '';
const classRm = content.match(/@RequestMapping\s*\(\s*"([^"]+)"/);
if (classRm) classPrefix = classRm[1].replace(/\/+$/, '');
const re = /@(Get|Post|Put|Delete|Patch)Mapping\s*\(\s*"([^"]+)"/gi;
let m: RegExpExecArray | null;
while ((m = re.exec(content)) !== null) {
const method = m[1].toUpperCase();
let p = m[2];
if (classPrefix) p = `${classPrefix}/${p.replace(/^\//, '')}`;
const pathNorm = normalizeHttpPath(p);
const sub = content.slice(m.index);
const nameM = sub.match(/(?:public|protected|private)\s+[\w<>,\s\[\]]+\s+(\w+)\s*\(/);
const name = nameM ? nameM[1] : m[0];
out.push(this.makeProvider(filePath, method, pathNorm, name, 0.8));
}
return out;
}
private scanExpressProviders(content: string, filePath: string): ExtractedContract[] {
const out: ExtractedContract[] = [];
const re = /(?:router|app)\.(get|post|put|delete|patch)\s*\(\s*['"]([^'"]+)['"]/gi;
let m: RegExpExecArray | null;
while ((m = re.exec(content)) !== null) {
const method = m[1].toUpperCase();
const pathNorm = normalizeHttpPath(m[2]);
out.push(this.makeProvider(filePath, method, pathNorm, 'handler', 0.8));
}
return out;
}
private scanLaravelProviders(content: string, filePath: string): ExtractedContract[] {
const out: ExtractedContract[] = [];
const re = /Route::(get|post|put|delete|patch)\s*\(\s*['"]([^'"]+)['"]/gi;
let m: RegExpExecArray | null;
while ((m = re.exec(content)) !== null) {
const method = m[1].toUpperCase();
const pathNorm = normalizeHttpPath(m[2]);
out.push(this.makeProvider(filePath, method, pathNorm, 'route', 0.8));
}
return out;
}
private scanFastApiProviders(content: string, filePath: string): ExtractedContract[] {
const out: ExtractedContract[] = [];
const re = /@app\.(get|post|put|delete|patch)\s*\(\s*['"]([^'"]+)['"]/gi;
let m: RegExpExecArray | null;
while ((m = re.exec(content)) !== null) {
const method = m[1].toUpperCase();
const pathNorm = normalizeHttpPath(m[2]);
out.push(this.makeProvider(filePath, method, pathNorm, 'handler', 0.8));
}
return out;
}
private makeProvider(
filePath: string,
method: string,
pathNorm: string,
name: string,
confidence: number,
): ExtractedContract {
const cid = contractIdFor(method, pathNorm);
return {
contractId: cid,
type: 'http',
role: 'provider',
symbolUid: '',
symbolRef: { filePath, name },
symbolName: name,
confidence,
meta: {
method,
path: pathNorm,
pathSegments: pathNorm.split('/').filter(Boolean),
extractionStrategy: 'source_scan',
},
};
}
// ─── Graph-assisted consumers ──────────────────────────────────────
private async extractConsumersGraph(
db: CypherExecutor,
repoPath: string,
getDetections: (rel: string) => HttpDetection[],
): Promise<ExtractedContract[]> {
const out: ExtractedContract[] = [];
let rows: Record<string, unknown>[];
@ -365,11 +344,14 @@ export class HttpRouteExtractor implements ContractExtractor {
const routePath = String(row.routePath ?? '');
const pathNorm = normalizeHttpPath(routePath);
let method = 'GET';
const content = readSafe(repoPath, filePath);
if (content) {
const inferred = this.inferFetchMethod(content, pathNorm);
if (inferred) method = inferred;
}
// Prefer the plugin's detected method if we can find a matching
// fetch/axios call in the same file.
const detections = filePath ? getDetections(filePath) : [];
const inferred = detections.find(
(d) => d.role === 'consumer' && normalizeConsumerPath(d.path) === pathNorm,
);
if (inferred) method = inferred.method;
const cid = contractIdFor(method, pathNorm);
let symbolUid = '';
let symbolName = 'fetch';
@ -407,81 +389,47 @@ export class HttpRouteExtractor implements ContractExtractor {
return out;
}
private inferFetchMethod(content: string, pathNorm: string): string | null {
const esc = pathNorm.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
const fetchRe = new RegExp(
`fetch\\s*\\(\\s*['"\`]([^'"\`]*${esc}[^'"\`]*)['"\`]\\s*,\\s*\\{[^}]*method:\\s*['"](\\w+)['"]`,
'i',
);
const m = content.match(fetchRe);
if (m) return m[2].toUpperCase();
return null;
}
// ─── Source-scan consumers ─────────────────────────────────────────
private async extractConsumersSourceScan(repoPath: string): Promise<ExtractedContract[]> {
const files = await glob('**/*.{ts,tsx,js,jsx,vue,svelte}', {
cwd: repoPath,
ignore: ['**/node_modules/**', '**/.git/**'],
nodir: true,
});
private extractConsumersSourceScan(
files: string[],
getDetections: (rel: string) => HttpDetection[],
): ExtractedContract[] {
const out: ExtractedContract[] = [];
for (const rel of files) {
const content = readSafe(repoPath, rel);
if (!content) continue;
out.push(...this.scanFetchConsumers(content, rel));
out.push(...this.scanAxiosConsumers(content, rel));
const detections = getDetections(rel);
for (const d of detections) {
if (d.role !== 'consumer') continue;
const pathNorm = normalizeConsumerPath(d.path);
out.push({
contractId: contractIdFor(d.method, pathNorm),
type: 'http',
role: 'consumer',
symbolUid: '',
symbolRef: { filePath: rel, name: 'fetch' },
symbolName: 'fetch',
confidence: d.confidence,
meta: {
method: d.method,
path: pathNorm,
extractionStrategy: 'source_scan',
framework: d.framework,
},
});
}
}
return this.dedupeContracts(out);
}
private scanFetchConsumers(content: string, filePath: string): ExtractedContract[] {
private dedupeContracts(items: ExtractedContract[]): ExtractedContract[] {
const seen = new Set<string>();
const out: ExtractedContract[] = [];
const re =
/fetch\s*\(\s*['"`]([^'"`]+)['"`](?:\s*,\s*\{[^}]*method:\s*['"](\w+)['"][^}]*\})?\s*\)/gi;
let m: RegExpExecArray | null;
while ((m = re.exec(content)) !== null) {
const pathNorm = normalizeHttpPath(this.templateToPattern(m[1]));
const method = (m[2] || 'GET').toUpperCase();
out.push(this.makeConsumer(filePath, method, pathNorm, 0.7));
for (const c of items) {
const k = `${c.contractId}|${c.symbolRef.filePath}|${c.symbolRef.name}`;
if (seen.has(k)) continue;
seen.add(k);
out.push(c);
}
return out;
}
private templateToPattern(url: string): string {
return url.replace(/\$\{[^}]+\}/g, '{param}');
}
private scanAxiosConsumers(content: string, filePath: string): ExtractedContract[] {
const out: ExtractedContract[] = [];
const re = /axios\.(get|post|put|delete|patch)\s*\(\s*[`'"]([^`'"]+)[`'"]/gi;
let m: RegExpExecArray | null;
while ((m = re.exec(content)) !== null) {
const method = m[1].toUpperCase();
const pathNorm = normalizeHttpPath(this.templateToPattern(m[2]));
out.push(this.makeConsumer(filePath, method, pathNorm, 0.7));
}
return out;
}
private makeConsumer(
filePath: string,
method: string,
pathNorm: string,
confidence: number,
): ExtractedContract {
return {
contractId: contractIdFor(method, pathNorm),
type: 'http',
role: 'consumer',
symbolUid: '',
symbolRef: { filePath, name: 'fetch' },
symbolName: 'fetch',
confidence,
meta: {
method,
path: pathNorm,
extractionStrategy: 'source_scan',
},
};
}
}

268
gitnexus/src/core/group/extractors/manifest-extractor.ts Normal file
View file

@ -0,0 +1,268 @@
import type { ContractType, CrossLink, GroupManifestLink, StoredContract } from '../types.js';
import type { CypherExecutor } from '../contract-extractor.js';
export interface ManifestExtractResult {
contracts: StoredContract[];
crossLinks: CrossLink[];
}
/**
* Canonicalize an HTTP path for matching against Route.name in the graph.
* Mirrors core/ingestion/pipeline.ts ensureSlash semantics:
* - Ensures a leading slash.
* - Strips trailing slashes (except the root "/").
* - Normalizes consecutive slashes.
* - Does NOT lowercase (route matching is case-sensitive).
*/
function normalizeRoutePath(raw: string): string {
const trimmed = raw.trim();
if (!trimmed) return '/';
const withLeading = trimmed.startsWith('/') ? trimmed : `/${trimmed}`;
const collapsed = withLeading.replace(/\/+/g, '/');
if (collapsed === '/') return '/';
return collapsed.replace(/\/+$/, '');
}
/**
* Stable synthetic symbolUid for a manifest-declared contract whose target
* symbol could not be resolved against the per-repo graph (resolveSymbol
* returned null). Two reasons we don't leave the uid empty:
*
* 1. The bridge stores Contract nodes keyed in part by symbolUid; an empty
* uid means downstream Cypher queries that anchor on `provider.symbolUid`
* can't tell two different unresolved manifest contracts apart.
* 2. The cross-impact bridge query in cross-impact.ts joins local impact
* results to bridge contracts via `WHERE provider.symbolUid IN $localUids`.
* If the local impact engine produces a deterministic identifier for the
* unresolved target, it must agree with the value the bridge stored. A
* synthetic uid keyed off (repo, contractId) is the only thing both sides
* can derive without knowing about each other.
*
* Format: `manifest::<repo>::<contractId>`. Stable across syncs, scoped to a
* single repo within a group, and never collides with real indexer uids
* (which never start with `manifest::`).
*/
export function manifestSymbolUid(repo: string, contractId: string): string {
return `manifest::${repo}::${contractId}`;
}
export class ManifestExtractor {
async extractFromManifest(
links: GroupManifestLink[],
dbExecutors?: Map<string, CypherExecutor>,
): Promise<ManifestExtractResult> {
const contracts: StoredContract[] = [];
const crossLinks: CrossLink[] = [];
for (const link of links) {
const contractId = this.buildContractId(link.type, link.contract);
const providerRepo = link.role === 'provider' ? link.from : link.to;
const consumerRepo = link.role === 'provider' ? link.to : link.from;
const providerSymbol = await this.resolveSymbol(providerRepo, link, dbExecutors);
const consumerSymbol = await this.resolveSymbol(consumerRepo, link, dbExecutors);
const providerRef = providerSymbol || { filePath: '', name: link.contract };
const consumerRef = consumerSymbol || { filePath: '', name: link.contract };
// When the resolver finds a real graph symbol we keep its uid, otherwise
// fall back to the deterministic synthetic uid (see manifestSymbolUid).
const providerUid = providerSymbol?.uid || manifestSymbolUid(providerRepo, contractId);
const consumerUid = consumerSymbol?.uid || manifestSymbolUid(consumerRepo, contractId);
contracts.push({
contractId,
type: link.type,
role: 'provider',
symbolUid: providerUid,
symbolRef: providerRef,
symbolName: link.contract,
confidence: 1.0,
meta: { source: 'manifest' },
repo: providerRepo,
});
contracts.push({
contractId,
type: link.type,
role: 'consumer',
symbolUid: consumerUid,
symbolRef: consumerRef,
symbolName: link.contract,
confidence: 1.0,
meta: { source: 'manifest' },
repo: consumerRepo,
});
crossLinks.push({
from: { repo: consumerRepo, symbolUid: consumerUid, symbolRef: consumerRef },
to: { repo: providerRepo, symbolUid: providerUid, symbolRef: providerRef },
type: link.type,
contractId,
matchType: 'manifest',
confidence: 1.0,
});
}
return { contracts, crossLinks };
}
private async resolveSymbol(
repoPathKey: string,
link: GroupManifestLink,
dbExecutors?: Map<string, CypherExecutor>,
): Promise<{ filePath: string; name: string; uid: string } | null> {
const executor = dbExecutors?.get(repoPathKey);
if (!executor) return null;
// NOTE: All lookups use EXACT equality on the relevant name field and
// deterministic ORDER BY before LIMIT 1. Previous versions used CONTAINS
// for fuzzy matching (plus an unconditional ".proto" fallback for gRPC)
// which produced silent false positives: e.g. manifest "/orders" would
// match "/suborders", and a gRPC manifest entry in a repo with any
// .proto file would attach to a random proto symbol.
//
// If resolveSymbol returns null, the extractor falls back to a
// deterministic synthetic uid via `manifestSymbolUid(repo, contractId)`
// (see the function's docstring for why synthetic rather than empty).
// Cross-impact still works: the bridge query joins on the synthetic
// uid, and the local impact engine derives the same uid for the
// unresolved symbol — name-based hints are the additional safety net.
try {
let rows: Record<string, unknown>[];
if (link.type === 'http') {
// Route.name is the canonicalized URL path (see
// core/ingestion/pipeline.ts ensureSlash + generateId('Route', ...)).
// Normalize the manifest contract the same way so a user-written
// "/api/orders" matches "api/orders" in the graph.
const normalized = normalizeRoutePath(link.contract);
rows = await executor(
`MATCH (handler)-[r:CodeRelation {type: 'HANDLES_ROUTE'}]->(route:Route)
WHERE route.name = $normalized
RETURN handler.id AS uid, handler.name AS name, handler.filePath AS filePath
ORDER BY handler.filePath ASC
LIMIT 1`,
{ normalized },
);
} else if (link.type === 'topic') {
// Topic names aren't a first-class NodeLabel in the graph —
// topics are referenced by function/method symbols (Kafka
// listeners, publishers). Restrict to symbol-like labels to
// avoid cross-matching Files/Variables/Imports that happen to
// share the topic name.
rows = await executor(
`MATCH (n:Function|Method|Class|Interface) WHERE n.name = $contract
RETURN n.id AS uid, n.name AS name, n.filePath AS filePath
ORDER BY n.filePath ASC
LIMIT 1`,
{ contract: link.contract },
);
} else if (link.type === 'grpc') {
// Contract is "Service/Method" or just "Service" (or package.Service
// variants). Prefer matching by method name when present, otherwise
// by service name. NO .proto path fallback — that's guaranteed to
// return a wrong symbol in any repo with more than one proto file.
// Label filters scope lookups: methods → Function|Method, services
// → Class|Interface (no label match = no silent wrong hits on
// File/Variable nodes that happen to share the name).
const parts = link.contract.split('/');
const serviceName = parts[0]?.trim() ?? '';
const methodName = parts[1]?.trim() ?? '';
if (methodName) {
rows = await executor(
`MATCH (n:Function|Method) WHERE n.name = $methodName
RETURN n.id AS uid, n.name AS name, n.filePath AS filePath
ORDER BY n.filePath ASC
LIMIT 1`,
{ methodName },
);
} else if (serviceName) {
rows = await executor(
`MATCH (n:Class|Interface) WHERE n.name = $serviceName
RETURN n.id AS uid, n.name AS name, n.filePath AS filePath
ORDER BY n.filePath ASC
LIMIT 1`,
{ serviceName },
);
} else {
rows = [];
}
} else if (link.type === 'lib') {
// Only exact match on the symbol's name. Previous fallback to
// CONTAINS on n.filePath would promote "react" to "react-native"
// or "@types/react" — silent wrong attribution. Restrict to
// package-level labels so we don't return arbitrary symbols
// named after a library.
rows = await executor(
`MATCH (n:Package|Module) WHERE n.name = $contract
RETURN n.id AS uid, n.name AS name, n.filePath AS filePath
ORDER BY n.filePath ASC
LIMIT 1`,
{ contract: link.contract },
);
} else {
return null;
}
if (rows.length > 0) {
return {
filePath: rows[0].filePath as string,
name: rows[0].name as string,
uid: String(rows[0].uid ?? ''),
};
}
} catch (err) {
// Log but don't throw: a broken graph query in one repo shouldn't
// fail the whole manifest extraction. Unresolved contracts still
// get a synthetic symbolUid below, so cross-impact can proceed.
const message = err instanceof Error ? err.message : String(err);
console.warn(
`[manifest-extractor] resolveSymbol failed for ${link.type}:${link.contract} ` +
`in ${repoPathKey}: ${message}`,
);
}
return null;
}
/**
* Build a canonical contract id for a manifest link.
*
* HTTP is the only type with two valid forms:
* - Explicit method: `"GET::/api/orders"` → `"http::GET::/api/orders"`
* (matches exactly against `HttpRouteExtractor` provider/consumer
* contracts, which are also keyed by `http::<METHOD>::<path>`).
* - Method-agnostic: `"/api/orders"` → `"http::*::/api/orders"`
* — the `*` is a wildcard and is intended to match any concrete
* HTTP method on that path. Wildcard-aware matching is the
* responsibility of the sync / cross-impact layer (see #793);
* downstream code should treat `http::*::<path>` as matching
* every `http::<METHOD>::<path>` for the same path.
*
* Recommend the explicit-method form in group.yaml whenever the
* manifest author knows the method — it round-trips through exact
* equality matching without requiring wildcard logic downstream.
*
* NOTE on exhaustiveness: the switch covers every current
* `ContractType` variant and falls through to a `never` assertion so
* TypeScript fails the build if a new variant is added without a
* corresponding case.
*/
private buildContractId(type: ContractType, contract: string): string {
switch (type) {
case 'http': {
if (/^[A-Za-z]+::/.test(contract)) return `http::${contract}`;
return `http::*::${contract}`;
}
case 'grpc':
return `grpc::${contract}`;
case 'topic':
return `topic::${contract}`;
case 'lib':
return `lib::${contract}`;
case 'custom':
return `custom::${contract}`;
default: {
const _exhaustive: never = type;
throw new Error(`Unhandled ContractType: ${String(_exhaustive)}`);
}
}
}
}

283
gitnexus/src/core/group/extractors/topic-extractor.ts
View file

@ -1,214 +1,49 @@
import * as fs from 'node:fs';
import * as path from 'node:path';
import { glob } from 'glob';
import Parser from 'tree-sitter';
import type { ContractExtractor, CypherExecutor } from '../contract-extractor.js';
import type { ExtractedContract, RepoHandle } from '../types.js';
import { readSafe } from './fs-utils.js';
import { scanFile, unquoteLiteral } from './tree-sitter-scanner.js';
import {
TOPIC_SCAN_GLOB,
getProviderForFile,
type Broker,
type TopicMeta,
} from './topic-patterns/index.js';
type Broker = 'kafka' | 'rabbitmq' | 'nats';
/**
* Language-agnostic orchestrator for topic (message broker) contract
* extraction. All grammar-specific knowledge lives in `topic-patterns/*`
* — this file must not import any tree-sitter grammar directly.
*
* Flow per file:
* 1. `getProviderForFile(rel)` → compiled plugin (or `undefined` if the
* file's extension isn't registered, in which case we skip it).
* 2. `scanFile(parser, provider, content)` → list of `{meta, valueText}`
* pairs, one per matched literal.
* 3. `unquoteLiteral(valueText)` → the raw topic string.
* 4. `makeContract(topic, meta, relPath)` → `ExtractedContract`.
*
* Adding a new language is a one-file edit in `topic-patterns/index.ts`.
*/
function readSafe(repoPath: string, rel: string): string | null {
const abs = path.resolve(repoPath, rel);
const base = path.resolve(repoPath);
const relToBase = path.relative(base, abs);
if (relToBase.startsWith('..') || path.isAbsolute(relToBase)) return null;
try {
return fs.readFileSync(abs, 'utf-8');
} catch {
return null;
}
}
function makeContract(
topicName: string,
role: 'provider' | 'consumer',
filePath: string,
symbolName: string,
confidence: number,
broker: Broker,
): ExtractedContract {
function makeContract(topicName: string, meta: TopicMeta, filePath: string): ExtractedContract {
return {
contractId: `topic::${topicName}`,
type: 'topic',
role,
role: meta.role,
symbolUid: '',
symbolRef: { filePath: filePath.replace(/\\/g, '/'), name: symbolName },
symbolName,
confidence,
symbolRef: { filePath: filePath.replace(/\\/g, '/'), name: meta.symbolName },
symbolName: meta.symbolName,
confidence: meta.confidence,
meta: {
broker,
broker: meta.broker satisfies Broker,
topicName,
extractionStrategy: 'source_scan',
extractionStrategy: 'tree_sitter',
},
};
}
interface PatternDef {
regex: RegExp;
role: 'provider' | 'consumer';
broker: Broker;
confidence: number;
topicGroup: number;
symbolName: string;
}
// --- Kafka patterns ---
const KAFKA_PATTERNS: PatternDef[] = [
// Java: @KafkaListener(topics = "xxx")
{
regex: /@KafkaListener\s*\(\s*topics\s*=\s*"([^"]+)"/g,
role: 'consumer',
broker: 'kafka',
confidence: 0.8,
topicGroup: 1,
symbolName: 'kafkaListener',
},
// Java: kafkaTemplate.send("xxx"
{
regex: /kafkaTemplate\.send\s*\(\s*"([^"]+)"/gi,
role: 'provider',
broker: 'kafka',
confidence: 0.8,
topicGroup: 1,
symbolName: 'kafkaTemplate.send',
},
// Node: producer.send({ topic: 'xxx'
{
regex: /producer\.send\s*\(\s*\{\s*topic:\s*['"]([^'"]+)['"]/g,
role: 'provider',
broker: 'kafka',
confidence: 0.8,
topicGroup: 1,
symbolName: 'producer.send',
},
// Node: consumer.subscribe({ topic: 'xxx'
{
regex: /consumer\.subscribe\s*\(\s*\{\s*topic:\s*['"]([^'"]+)['"]/g,
role: 'consumer',
broker: 'kafka',
confidence: 0.8,
topicGroup: 1,
symbolName: 'consumer.subscribe',
},
// Go: consumer.ConsumePartition("xxx"
{
regex: /\.ConsumePartition\s*\(\s*"([^"]+)"/g,
role: 'consumer',
broker: 'kafka',
confidence: 0.7,
topicGroup: 1,
symbolName: 'ConsumePartition',
},
// Python: KafkaConsumer('xxx'
{
regex: /KafkaConsumer\s*\(\s*['"]([^'"]+)['"]/g,
role: 'consumer',
broker: 'kafka',
confidence: 0.7,
topicGroup: 1,
symbolName: 'KafkaConsumer',
},
// Python: producer.send('xxx' or producer.produce('xxx'
{
regex: /producer\.(?:send|produce)\s*\(\s*['"]([^'"]+)['"]/g,
role: 'provider',
broker: 'kafka',
confidence: 0.7,
topicGroup: 1,
symbolName: 'producer.send',
},
];
// --- RabbitMQ patterns ---
const RABBITMQ_PATTERNS: PatternDef[] = [
// Java: @RabbitListener(queues = "xxx")
{
regex: /@RabbitListener\s*\(\s*queues\s*=\s*"([^"]+)"/g,
role: 'consumer',
broker: 'rabbitmq',
confidence: 0.8,
topicGroup: 1,
symbolName: 'rabbitListener',
},
// Java: rabbitTemplate.convertAndSend("xxx"
{
regex: /rabbitTemplate\.convertAndSend\s*\(\s*"([^"]+)"/gi,
role: 'provider',
broker: 'rabbitmq',
confidence: 0.8,
topicGroup: 1,
symbolName: 'rabbitTemplate.convertAndSend',
},
// Node: channel.consume("xxx"
{
regex: /channel\.consume\s*\(\s*"([^"]+)"/g,
role: 'consumer',
broker: 'rabbitmq',
confidence: 0.8,
topicGroup: 1,
symbolName: 'channel.consume',
},
// Node: channel.publish("xxx"
{
regex: /channel\.publish\s*\(\s*"([^"]+)"/g,
role: 'provider',
broker: 'rabbitmq',
confidence: 0.8,
topicGroup: 1,
symbolName: 'channel.publish',
},
// Node: channel.sendToQueue("xxx"
{
regex: /channel\.sendToQueue\s*\(\s*"([^"]+)"/g,
role: 'provider',
broker: 'rabbitmq',
confidence: 0.8,
topicGroup: 1,
symbolName: 'channel.sendToQueue',
},
// Python: channel.basic_consume(queue='xxx'
{
regex: /channel\.basic_consume\s*\(\s*queue\s*=\s*['"]([^'"]+)['"]/g,
role: 'consumer',
broker: 'rabbitmq',
confidence: 0.7,
topicGroup: 1,
symbolName: 'basic_consume',
},
// Python: channel.basic_publish(exchange='xxx'
{
regex: /channel\.basic_publish\s*\([^)]*exchange\s*=\s*['"]([^'"]+)['"]/g,
role: 'provider',
broker: 'rabbitmq',
confidence: 0.7,
topicGroup: 1,
symbolName: 'basic_publish',
},
];
// --- NATS patterns ---
const NATS_PATTERNS: PatternDef[] = [
// Go/Node: nc.Subscribe("xxx" or nc.subscribe("xxx"
{
regex: /nc\.(?:S|s)ubscribe\s*\(\s*"([^"]+)"/g,
role: 'consumer',
broker: 'nats',
confidence: 0.8,
topicGroup: 1,
symbolName: 'nc.Subscribe',
},
// Go/Node: nc.Publish("xxx" or nc.publish("xxx"
{
regex: /nc\.(?:P|p)ublish\s*\(\s*"([^"]+)"/g,
role: 'provider',
broker: 'nats',
confidence: 0.8,
topicGroup: 1,
symbolName: 'nc.Publish',
},
];
const ALL_PATTERNS: PatternDef[] = [...KAFKA_PATTERNS, ...RABBITMQ_PATTERNS, ...NATS_PATTERNS];
export class TopicExtractor implements ContractExtractor {
type = 'topic' as const;
@ -221,46 +56,48 @@ export class TopicExtractor implements ContractExtractor {
repoPath: string,
_repo: RepoHandle,
): Promise<ExtractedContract[]> {
const files = await glob('**/*.{ts,tsx,js,jsx,java,go,py}', {
const files = await glob(TOPIC_SCAN_GLOB, {
cwd: repoPath,
ignore: ['**/node_modules/**', '**/.git/**', '**/vendor/**', '**/dist/**', '**/build/**'],
ignore: [
'**/node_modules/**',
'**/.git/**',
'**/vendor/**',
'**/dist/**',
'**/build/**',
// Language-level test file conventions. Go test files
// `*_test.go` live next to source; other languages either use
// separate test directories (Python's `tests/`, Java's
// `src/test/`) or are already covered by the dist/build ignores.
// Pushed to the glob level so the orchestrator stays
// language-agnostic.
'**/*_test.go',
],
nodir: true,
});
// One parser reused across files; the scanner calls `setLanguage` per
// file based on which plugin the registry returns.
const parser = new Parser();
const out: ExtractedContract[] = [];
for (const rel of files) {
const provider = getProviderForFile(rel);
if (!provider) continue;
const content = readSafe(repoPath, rel);
if (!content) continue;
out.push(...this.scanFile(content, rel));
}
return this.dedupe(out);
}
private scanFile(content: string, filePath: string): ExtractedContract[] {
const out: ExtractedContract[] = [];
for (const pattern of ALL_PATTERNS) {
// Reset regex state for each file
const re = new RegExp(pattern.regex.source, pattern.regex.flags);
let m: RegExpExecArray | null;
while ((m = re.exec(content)) !== null) {
const topicName = m[pattern.topicGroup];
const matches = scanFile(parser, provider, content);
for (const match of matches) {
const valueNode = match.captures.value;
if (!valueNode) continue;
const topicName = unquoteLiteral(valueNode.text);
if (!topicName) continue;
out.push(
makeContract(
topicName,
pattern.role,
filePath,
pattern.symbolName,
pattern.confidence,
pattern.broker,
),
);
out.push(makeContract(topicName, match.meta, rel));
}
}
return out;
return this.dedupe(out);
}
private dedupe(items: ExtractedContract[]): ExtractedContract[] {

123
gitnexus/src/core/group/extractors/topic-patterns/go.ts Normal file
View file

@ -0,0 +1,123 @@
import Go from 'tree-sitter-go';
import { compilePatterns, type LanguagePatterns } from '../tree-sitter-scanner.js';
import type { TopicMeta } from './types.js';
/**
* Go topic extraction patterns.
*
* Detects Sarama, segmentio/kafka-go and nats.go producer/consumer APIs:
* - `X.ConsumePartition("topic", ...)`
* - `sarama.ProducerMessage{Topic: "xxx"}`
* - `kafka.Writer{Topic: "xxx"}` / `kafka.WriterConfig{Topic: ...}`
* - `kafka.Reader{Topic: "xxx"}` / `kafka.ReaderConfig{Topic: ...}`
* - `nc.Subscribe("topic", ...)` / `js.Subscribe("topic", ...)`
* - `nc.Publish("topic", ...)` / `js.Publish("topic", ...)`
*
* Every query MUST bind `@value` to the topic literal node.
*/
const GO_TOPIC_SPEC: LanguagePatterns<TopicMeta> = {
name: 'go-topic',
language: Go,
patterns: [
{
meta: {
role: 'consumer',
broker: 'kafka',
confidence: 0.7,
symbolName: 'ConsumePartition',
},
query: `
(call_expression
function: (selector_expression
field: (field_identifier) @method (#eq? @method "ConsumePartition"))
arguments: (argument_list . (interpreted_string_literal) @value))
`,
},
{
meta: {
role: 'provider',
broker: 'kafka',
confidence: 0.75,
symbolName: 'sarama.ProducerMessage',
},
query: `
(composite_literal
type: (qualified_type
package: (package_identifier) @pkg (#eq? @pkg "sarama")
name: (type_identifier) @ty (#eq? @ty "ProducerMessage"))
body: (literal_value
(keyed_element
(literal_element (identifier) @field (#eq? @field "Topic"))
(literal_element (interpreted_string_literal) @value))))
`,
},
{
meta: {
role: 'provider',
broker: 'kafka',
confidence: 0.75,
symbolName: 'kafka.Writer',
},
query: `
(composite_literal
type: (qualified_type
package: (package_identifier) @pkg (#eq? @pkg "kafka")
name: (type_identifier) @ty (#match? @ty "^(Writer|WriterConfig)$"))
body: (literal_value
(keyed_element
(literal_element (identifier) @field (#eq? @field "Topic"))
(literal_element (interpreted_string_literal) @value))))
`,
},
{
meta: {
role: 'consumer',
broker: 'kafka',
confidence: 0.75,
symbolName: 'kafka.Reader',
},
query: `
(composite_literal
type: (qualified_type
package: (package_identifier) @pkg (#eq? @pkg "kafka")
name: (type_identifier) @ty (#match? @ty "^(Reader|ReaderConfig)$"))
body: (literal_value
(keyed_element
(literal_element (identifier) @field (#eq? @field "Topic"))
(literal_element (interpreted_string_literal) @value))))
`,
},
{
meta: {
role: 'consumer',
broker: 'nats',
confidence: 0.8,
symbolName: 'nc.Subscribe',
},
query: `
(call_expression
function: (selector_expression
operand: (identifier) @obj (#match? @obj "^(nc|js)$")
field: (field_identifier) @method (#match? @method "^[Ss]ubscribe$"))
arguments: (argument_list . (interpreted_string_literal) @value))
`,
},
{
meta: {
role: 'provider',
broker: 'nats',
confidence: 0.8,
symbolName: 'nc.Publish',
},
query: `
(call_expression
function: (selector_expression
operand: (identifier) @obj (#match? @obj "^(nc|js)$")
field: (field_identifier) @method (#match? @method "^[Pp]ublish$"))
arguments: (argument_list . (interpreted_string_literal) @value))
`,
},
],
};
export const GO_TOPIC_PROVIDER = compilePatterns(GO_TOPIC_SPEC);

49
gitnexus/src/core/group/extractors/topic-patterns/index.ts Normal file
View file

@ -0,0 +1,49 @@
import * as path from 'node:path';
import type { CompiledPatterns } from '../tree-sitter-scanner.js';
import type { TopicMeta } from './types.js';
import { JAVA_TOPIC_PROVIDER } from './java.js';
import { GO_TOPIC_PROVIDER } from './go.js';
import { PYTHON_TOPIC_PROVIDER } from './python.js';
import {
JAVASCRIPT_TOPIC_PROVIDER,
TYPESCRIPT_TOPIC_PROVIDER,
TSX_TOPIC_PROVIDER,
} from './node.js';
export type { TopicMeta, Broker } from './types.js';
/**
* File-extension → compiled-plugin registry for topic extraction. The
* top-level orchestrator (`topic-extractor.ts`) looks up the plugin for
* each file it visits and delegates the scanning to `tree-sitter-scanner`.
*
* Keys are lowercase extensions including the leading dot. To add a new
* language, drop a `topic-patterns/<lang>.ts` that exports a compiled
* provider, import it here and register the extension(s). No edits to
* `topic-extractor.ts` are required.
*/
const REGISTRY: Record<string, CompiledPatterns<TopicMeta>> = {
'.java': JAVA_TOPIC_PROVIDER,
'.go': GO_TOPIC_PROVIDER,
'.py': PYTHON_TOPIC_PROVIDER,
'.js': JAVASCRIPT_TOPIC_PROVIDER,
'.jsx': JAVASCRIPT_TOPIC_PROVIDER,
'.ts': TYPESCRIPT_TOPIC_PROVIDER,
'.tsx': TSX_TOPIC_PROVIDER,
};
/**
* Glob pattern for files worth scanning. Kept here so adding a new
* language to the registry also widens the glob automatically via a
* single edit.
*/
export const TOPIC_SCAN_GLOB = '**/*.{ts,tsx,js,jsx,java,go,py}';
/**
* Return the compiled provider registered for the given file's
* extension, or `undefined` if the extension is not registered.
*/
export function getProviderForFile(rel: string): CompiledPatterns<TopicMeta> | undefined {
const ext = path.extname(rel).toLowerCase();
return REGISTRY[ext];
}

83
gitnexus/src/core/group/extractors/topic-patterns/java.ts Normal file
View file

@ -0,0 +1,83 @@
import Java from 'tree-sitter-java';
import { compilePatterns, type LanguagePatterns } from '../tree-sitter-scanner.js';
import type { TopicMeta } from './types.js';
/**
* Java topic extraction patterns.
*
* Detects Kafka and RabbitMQ (Spring conventions) producer/consumer APIs:
* - `@KafkaListener(topics = "xxx")`
* - `@RabbitListener(queues = "xxx")`
* - `kafkaTemplate.send("xxx", ...)`
* - `rabbitTemplate.convertAndSend("xxx", ...)`
*
* Every query MUST bind `@value` to the topic literal node.
*/
const JAVA_TOPIC_SPEC: LanguagePatterns<TopicMeta> = {
name: 'java-topic',
language: Java,
patterns: [
{
meta: {
role: 'consumer',
broker: 'kafka',
confidence: 0.8,
symbolName: 'kafkaListener',
},
query: `
(annotation
name: (identifier) @name (#eq? @name "KafkaListener")
arguments: (annotation_argument_list
(element_value_pair
key: (identifier) @key (#eq? @key "topics")
value: (string_literal) @value)))
`,
},
{
meta: {
role: 'consumer',
broker: 'rabbitmq',
confidence: 0.8,
symbolName: 'rabbitListener',
},
query: `
(annotation
name: (identifier) @name (#eq? @name "RabbitListener")
arguments: (annotation_argument_list
(element_value_pair
key: (identifier) @key (#eq? @key "queues")
value: (string_literal) @value)))
`,
},
{
meta: {
role: 'provider',
broker: 'kafka',
confidence: 0.8,
symbolName: 'kafkaTemplate.send',
},
query: `
(method_invocation
object: (identifier) @obj (#eq? @obj "kafkaTemplate")
name: (identifier) @method (#eq? @method "send")
arguments: (argument_list . (string_literal) @value))
`,
},
{
meta: {
role: 'provider',
broker: 'rabbitmq',
confidence: 0.8,
symbolName: 'rabbitTemplate.convertAndSend',
},
query: `
(method_invocation
object: (identifier) @obj (#eq? @obj "rabbitTemplate")
name: (identifier) @method (#eq? @method "convertAndSend")
arguments: (argument_list . (string_literal) @value))
`,
},
],
};
export const JAVA_TOPIC_PROVIDER = compilePatterns(JAVA_TOPIC_SPEC);

165
gitnexus/src/core/group/extractors/topic-patterns/node.ts Normal file
View file

@ -0,0 +1,165 @@
import JavaScript from 'tree-sitter-javascript';
import TypeScript from 'tree-sitter-typescript';
import {
compilePatterns,
type LanguagePatterns,
type PatternSpec,
} from '../tree-sitter-scanner.js';
import type { TopicMeta } from './types.js';
/**
* Node.js / TypeScript topic extraction patterns.
*
* Detects kafkajs, amqplib (RabbitMQ), and nats.js producer/consumer APIs:
* - `producer.send({ topic: 'xxx', ... })` (kafkajs)
* - `consumer.subscribe({ topic: 'xxx', ... })` (kafkajs)
* - `channel.consume("queue", ...)` / `channel.publish(...)` / `channel.sendToQueue(...)`
* - `nc.subscribe("topic")` / `js.subscribe("topic")`
* - `nc.publish("topic", ...)` / `js.publish("topic", ...)`
*
* The JavaScript and TypeScript tree-sitter grammars share node type
* names for every construct we query here, so the pattern sources are
* defined once and compiled against each grammar variant. We export three
* providers because Parser.Query objects are NOT portable across grammar
* instances — `.js` files use the JavaScript grammar, `.ts` uses
* TypeScript.typescript, and `.tsx` uses TypeScript.tsx.
*
* Every query MUST bind `@value` to the topic literal node.
*/
const NODE_TOPIC_PATTERNS: PatternSpec<TopicMeta>[] = [
{
meta: {
role: 'provider',
broker: 'kafka',
confidence: 0.8,
symbolName: 'producer.send',
},
query: `
(call_expression
function: (member_expression
object: (identifier) @obj (#eq? @obj "producer")
property: (property_identifier) @prop (#eq? @prop "send"))
arguments: (arguments
(object
(pair
key: (property_identifier) @key (#eq? @key "topic")
value: [(string) (template_string)] @value))))
`,
},
{
meta: {
role: 'consumer',
broker: 'kafka',
confidence: 0.8,
symbolName: 'consumer.subscribe',
},
query: `
(call_expression
function: (member_expression
object: (identifier) @obj (#eq? @obj "consumer")
property: (property_identifier) @prop (#eq? @prop "subscribe"))
arguments: (arguments
(object
(pair
key: (property_identifier) @key (#eq? @key "topic")
value: [(string) (template_string)] @value))))
`,
},
{
meta: {
role: 'consumer',
broker: 'rabbitmq',
confidence: 0.8,
symbolName: 'channel.consume',
},
query: `
(call_expression
function: (member_expression
object: (identifier) @obj (#eq? @obj "channel")
property: (property_identifier) @prop (#eq? @prop "consume"))
arguments: (arguments . [(string) (template_string)] @value))
`,
},
{
meta: {
role: 'provider',
broker: 'rabbitmq',
confidence: 0.8,
symbolName: 'channel.publish',
},
query: `
(call_expression
function: (member_expression
object: (identifier) @obj (#eq? @obj "channel")
property: (property_identifier) @prop (#eq? @prop "publish"))
arguments: (arguments . [(string) (template_string)] @value))
`,
},
{
meta: {
role: 'provider',
broker: 'rabbitmq',
confidence: 0.8,
symbolName: 'channel.sendToQueue',
},
query: `
(call_expression
function: (member_expression
object: (identifier) @obj (#eq? @obj "channel")
property: (property_identifier) @prop (#eq? @prop "sendToQueue"))
arguments: (arguments . [(string) (template_string)] @value))
`,
},
{
meta: {
role: 'consumer',
broker: 'nats',
confidence: 0.8,
symbolName: 'nc.subscribe',
},
query: `
(call_expression
function: (member_expression
object: (identifier) @obj (#match? @obj "^(nc|js)$")
property: (property_identifier) @prop (#match? @prop "^[Ss]ubscribe$"))
arguments: (arguments . [(string) (template_string)] @value))
`,
},
{
meta: {
role: 'provider',
broker: 'nats',
confidence: 0.8,
symbolName: 'nc.publish',
},
query: `
(call_expression
function: (member_expression
object: (identifier) @obj (#match? @obj "^(nc|js)$")
property: (property_identifier) @prop (#match? @prop "^[Pp]ublish$"))
arguments: (arguments . [(string) (template_string)] @value))
`,
},
];
const JAVASCRIPT_TOPIC_SPEC: LanguagePatterns<TopicMeta> = {
name: 'javascript-topic',
language: JavaScript,
patterns: NODE_TOPIC_PATTERNS,
};
const TYPESCRIPT_TOPIC_SPEC: LanguagePatterns<TopicMeta> = {
name: 'typescript-topic',
language: TypeScript.typescript,
patterns: NODE_TOPIC_PATTERNS,
};
const TSX_TOPIC_SPEC: LanguagePatterns<TopicMeta> = {
name: 'tsx-topic',
language: TypeScript.tsx,
patterns: NODE_TOPIC_PATTERNS,
};
export const JAVASCRIPT_TOPIC_PROVIDER = compilePatterns(JAVASCRIPT_TOPIC_SPEC);
export const TYPESCRIPT_TOPIC_PROVIDER = compilePatterns(TYPESCRIPT_TOPIC_SPEC);
export const TSX_TOPIC_PROVIDER = compilePatterns(TSX_TOPIC_SPEC);

119
gitnexus/src/core/group/extractors/topic-patterns/python.ts Normal file
View file

@ -0,0 +1,119 @@
import Python from 'tree-sitter-python';
import { compilePatterns, type LanguagePatterns } from '../tree-sitter-scanner.js';
import type { TopicMeta } from './types.js';
/**
* Python topic extraction patterns.
*
* Detects kafka-python, pika (RabbitMQ), and nats-py producer/consumer APIs:
* - `KafkaConsumer('topic', ...)`
* - `producer.send('topic', ...)` / `producer.produce('topic', ...)`
* - `channel.basic_consume(queue='xxx', ...)`
* - `channel.basic_publish(exchange='xxx', ...)`
* - `await nc.subscribe('topic')`
* - `await nc.publish('topic', ...)`
*
* Every query MUST bind `@value` to the topic literal node.
*/
const PYTHON_TOPIC_SPEC: LanguagePatterns<TopicMeta> = {
name: 'python-topic',
language: Python,
patterns: [
{
meta: {
role: 'consumer',
broker: 'kafka',
confidence: 0.7,
symbolName: 'KafkaConsumer',
},
query: `
(call
function: (identifier) @func (#eq? @func "KafkaConsumer")
arguments: (argument_list . (string) @value))
`,
},
{
meta: {
role: 'provider',
broker: 'kafka',
confidence: 0.7,
symbolName: 'producer.send',
},
query: `
(call
function: (attribute
object: (identifier) @obj (#eq? @obj "producer")
attribute: (identifier) @method (#match? @method "^(send|produce)$"))
arguments: (argument_list . (string) @value))
`,
},
{
meta: {
role: 'consumer',
broker: 'rabbitmq',
confidence: 0.7,
symbolName: 'basic_consume',
},
query: `
(call
function: (attribute
object: (identifier) @obj (#eq? @obj "channel")
attribute: (identifier) @method (#eq? @method "basic_consume"))
arguments: (argument_list
(keyword_argument
name: (identifier) @kw (#eq? @kw "queue")
value: (string) @value)))
`,
},
{
meta: {
role: 'provider',
broker: 'rabbitmq',
confidence: 0.7,
symbolName: 'basic_publish',
},
query: `
(call
function: (attribute
object: (identifier) @obj (#eq? @obj "channel")
attribute: (identifier) @method (#eq? @method "basic_publish"))
arguments: (argument_list
(keyword_argument
name: (identifier) @kw (#eq? @kw "exchange")
value: (string) @value)))
`,
},
{
meta: {
role: 'consumer',
broker: 'nats',
confidence: 0.75,
symbolName: 'nc.subscribe',
},
query: `
(call
function: (attribute
object: (identifier) @obj (#eq? @obj "nc")
attribute: (identifier) @method (#eq? @method "subscribe"))
arguments: (argument_list . (string) @value))
`,
},
{
meta: {
role: 'provider',
broker: 'nats',
confidence: 0.75,
symbolName: 'nc.publish',
},
query: `
(call
function: (attribute
object: (identifier) @obj (#eq? @obj "nc")
attribute: (identifier) @method (#eq? @method "publish"))
arguments: (argument_list . (string) @value))
`,
},
],
};
export const PYTHON_TOPIC_PROVIDER = compilePatterns(PYTHON_TOPIC_SPEC);

27
gitnexus/src/core/group/extractors/topic-patterns/types.ts Normal file
View file

@ -0,0 +1,27 @@
/**
* Shared types for the topic-extractor language plugins.
*
* Each plugin lives in its own file (java.ts, go.ts, ...) and owns the
* tree-sitter grammar import + query sources. The top-level
* `topic-extractor.ts` orchestrator only knows about this type module and
* the plugin registry (`./index.ts`). It MUST NOT import any grammar or
* query text directly — that's the whole point of the split.
*/
export type Broker = 'kafka' | 'rabbitmq' | 'nats';
/**
* Per-pattern payload every topic plugin attaches to its query. Whatever
* the pattern matches, the orchestrator receives this object verbatim
* and uses it to build an `ExtractedContract`.
*
* Plugins produce one `TopicMeta` per pattern (not per match) because a
* single query uniquely identifies its broker/role/confidence triple.
*/
export interface TopicMeta {
role: 'provider' | 'consumer';
broker: Broker;
confidence: number;
/** Short human-readable label of the API being detected. */
symbolName: string;
}

193
gitnexus/src/core/group/extractors/tree-sitter-scanner.ts Normal file
View file

@ -0,0 +1,193 @@
import Parser from 'tree-sitter';
/**
* Shared, language-agnostic tree-sitter scanning utilities used by group
* extractors (topic, http, grpc, ...).
*
* Design goals:
* - The top-level extractors must not import any tree-sitter grammar.
* - Per-language plugins own their grammar import, their query sources,
* and the mapping from capture → meta.
* - This module provides the plumbing: compile queries once per plugin,
* parse a file with a given grammar, run all patterns, and return the
* captured `string_literal`-style nodes together with the plugin's meta.
*/
/**
* One pattern owned by a language plugin. Each pattern owns a tree-sitter
* S-expression query. Plugins can freely choose which capture names to
* use — the scanner exposes every capture in the returned `captures`
* map and does not privilege any particular name.
*
* `TMeta` is the plugin-specific payload the orchestrator receives back
* when this pattern matches — e.g. for topic extraction it carries the
* broker name, role, confidence, symbol name.
*/
export interface PatternSpec<TMeta> {
/** Tree-sitter S-expression. */
query: string;
/** Plugin-specific payload returned on every match. */
meta: TMeta;
}
/**
* A set of patterns owned by one language plugin, bound to a specific
* tree-sitter grammar.
*
* `language` is typed as `unknown` because tree-sitter's TypeScript
* declarations use `any` for the grammar object, and the grammar modules
* export different shapes (plain grammar vs. namespace with `typescript`
* / `tsx` members). Callers pass the concrete grammar object; this
* module forwards it to `parser.setLanguage` / `new Parser.Query`.
*/
export interface LanguagePatterns<TMeta> {
/** Human-readable plugin name for diagnostics. */
name: string;
/** tree-sitter grammar object. */
language: unknown;
/** Patterns authored against `language`. */
patterns: PatternSpec<TMeta>[];
}
/**
* Compiled form of a `LanguagePatterns` bundle. Queries are compiled
* eagerly at module load time so a broken grammar/query pair fails
* loudly the first time the plugin is imported, instead of silently
* at scan time when no contract is produced.
*/
export interface CompiledPatterns<TMeta> {
name: string;
language: unknown;
patterns: CompiledPattern<TMeta>[];
}
export interface CompiledPattern<TMeta> {
query: Parser.Query;
meta: TMeta;
}
/**
* Map from capture name → syntax node. Every named capture the query
* binds is exposed as an entry. If a query captures the same name more
* than once (unusual), the first occurrence wins — plugins that need
* all occurrences should use distinct capture names or fall back to
* `match.captures` array directly by iterating `query.matches()`
* themselves.
*/
export type CaptureMap = Record<string, Parser.SyntaxNode>;
/**
* One match returned by `scanFile` / `runCompiledPatterns`. The caller
* receives the full capture map plus the plugin meta, and is
* responsible for turning it into a domain object.
*/
export interface ScanMatch<TMeta> {
meta: TMeta;
captures: CaptureMap;
}
/**
* Compile a LanguagePatterns bundle. Call this once per plugin, at
* module load time, and export the result. Throws if any pattern
* fails to compile against the grammar — that's a bug in the plugin
* author's query, not a runtime condition.
*/
export function compilePatterns<TMeta>(bundle: LanguagePatterns<TMeta>): CompiledPatterns<TMeta> {
const compiled: CompiledPattern<TMeta>[] = [];
for (const spec of bundle.patterns) {
try {
const query = new Parser.Query(bundle.language, spec.query);
compiled.push({ query, meta: spec.meta });
} catch (err) {
const message = err instanceof Error ? err.message : String(err);
throw new Error(
`[tree-sitter-scanner] Failed to compile pattern in ${bundle.name}: ${message}\n` +
`Query source:\n${spec.query}`,
);
}
}
return { name: bundle.name, language: bundle.language, patterns: compiled };
}
/**
* Run every compiled pattern in `plugin` against an already-parsed
* tree. Use this when a plugin needs multiple query bundles against
* the same file (e.g. one query for class-level prefixes and another
* for method-level annotations) and wants to avoid re-parsing.
*/
export function runCompiledPatterns<TMeta>(
plugin: CompiledPatterns<TMeta>,
tree: Parser.Tree,
): ScanMatch<TMeta>[] {
const out: ScanMatch<TMeta>[] = [];
for (const compiled of plugin.patterns) {
let matches: Parser.QueryMatch[];
try {
matches = compiled.query.matches(tree.rootNode);
} catch {
continue;
}
for (const match of matches) {
const captures: CaptureMap = {};
for (const cap of match.captures) {
if (!(cap.name in captures)) captures[cap.name] = cap.node;
}
out.push({ meta: compiled.meta, captures });
}
}
return out;
}
/**
* Parse `content` with the plugin's grammar and run every compiled
* pattern against the AST. Returns one `ScanMatch` per matched query
* occurrence, carrying the plugin's meta payload.
*
* Errors are swallowed at the file level (malformed file must not abort
* the whole extract). Individual pattern failures are swallowed too so
* a single unusable query doesn't block the rest of the plugin.
*/
export function scanFile<TMeta>(
parser: Parser,
plugin: CompiledPatterns<TMeta>,
content: string,
): ScanMatch<TMeta>[] {
let tree: Parser.Tree;
try {
parser.setLanguage(plugin.language);
tree = parser.parse(content);
} catch {
return [];
}
return runCompiledPatterns(plugin, tree);
}
/**
* Strip enclosing quotes from a tree-sitter string literal node's text.
* Handles single / double / template quotes, Python triple-quoted strings,
* and Go raw string literals (backticks).
*
* Returns null for empty/nullish input so callers can uniformly skip
* captures whose value is missing.
*/
export function unquoteLiteral(raw: string): string | null {
if (!raw) return null;
// Python triple-quoted
if (
(raw.startsWith('"""') && raw.endsWith('"""')) ||
(raw.startsWith("'''") && raw.endsWith("'''"))
) {
return raw.slice(3, -3);
}
const first = raw[0];
const last = raw[raw.length - 1];
if ((first === '"' || first === "'" || first === '`') && last === first && raw.length >= 2) {
return raw.slice(1, -1);
}
// Some grammars expose the string content without quotes already (e.g.
// Python `string_content` child). Return as-is.
return raw;
}

539
gitnexus/test/unit/group/grpc-extractor.test.ts
View file

@ -1,17 +1,23 @@
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
import * as fs from 'node:fs';
import fsp from 'node:fs/promises';
import * as path from 'node:path';
import * as os from 'node:os';
import { GrpcExtractor } from '../../../src/core/group/extractors/grpc-extractor.js';
import {
GrpcExtractor,
buildProtoMap,
resolveProtoConflict,
serviceContractId,
} from '../../../src/core/group/extractors/grpc-extractor.js';
import type { ProtoServiceInfo } from '../../../src/core/group/extractors/grpc-extractor.js';
import type { RepoHandle } from '../../../src/core/group/types.js';
describe('GrpcExtractor', () => {
let tmpDir: string;
let extractor: GrpcExtractor;
beforeEach(() => {
tmpDir = path.join(os.tmpdir(), `gitnexus-grpc-${Date.now()}`);
fs.mkdirSync(tmpDir, { recursive: true });
beforeEach(async () => {
tmpDir = await fsp.mkdtemp(path.join(os.tmpdir(), 'gitnexus-grpc-'));
extractor = new GrpcExtractor();
});
@ -205,6 +211,66 @@ service IncompleteService {
// The old regex would find partial match; the new parser should skip it
expect(providers).toHaveLength(0);
});
it('test_extract_proto_ignores_braces_inside_string_literals', async () => {
// Regression for a known parser limitation: braces inside string
// literals used to be counted as real service-body braces, which
// would terminate the service early and drop methods after the
// offending string.
writeFile(
'api/strings.proto',
`syntax = "proto3";
package strings;
service TrickyService {
rpc First (Req) returns (Res) {
option (google.api.http).additional_bindings = {
post: "/v1/first";
};
}
// Previously the "{" inside this literal would close the service body.
option deprecated_reason = "use NewService { instead";
rpc Second (Req) returns (Res);
rpc Third (Req) returns (Res);
}
`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const protoProviders = contracts.filter(
(c) => c.role === 'provider' && c.symbolRef.filePath === 'api/strings.proto',
);
// All three methods must be extracted even though a string literal
// contains an unbalanced "{".
expect(protoProviders.map((c) => c.symbolName).sort()).toEqual([
'TrickyService.First',
'TrickyService.Second',
'TrickyService.Third',
]);
});
it('test_extract_proto_ignores_braces_inside_comments', async () => {
writeFile(
'api/commented.proto',
`syntax = "proto3";
package commented;
service Svc {
// TODO: move { or } from this comment — parser used to count them
/* A block comment with { unbalanced braces } */
rpc Alpha (Req) returns (Res);
// }} end of the method block (in comment)
rpc Beta (Req) returns (Res);
}
`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const protoProviders = contracts.filter(
(c) => c.role === 'provider' && c.symbolRef.filePath === 'api/commented.proto',
);
expect(protoProviders.map((c) => c.symbolName).sort()).toEqual(['Svc.Alpha', 'Svc.Beta']);
});
});
describe('Go server detection', () => {
@ -228,7 +294,7 @@ func main() {
expect(providers.length).toBeGreaterThanOrEqual(1);
expect(providers[0].contractId).toContain('grpc::');
expect(providers[0].contractId).toContain('AuthService');
expect(providers[0].confidence).toBe(0.8);
expect(providers[0].confidence).toBe(0.65);
});
it('test_extract_go_unimplemented_server_returns_provider', async () => {
@ -267,7 +333,7 @@ func NewAuthClient(conn *grpc.ClientConn) pb.AuthServiceClient {
expect(consumers.length).toBeGreaterThanOrEqual(1);
expect(consumers[0].contractId).toContain('AuthService');
expect(consumers[0].confidence).toBe(0.7);
expect(consumers[0].confidence).toBe(0.55);
});
});
@ -287,7 +353,7 @@ public class AuthGrpcService extends AuthServiceGrpc.AuthServiceImplBase {
expect(providers.length).toBeGreaterThanOrEqual(1);
expect(providers[0].contractId).toContain('AuthService');
expect(providers[0].confidence).toBe(0.8);
expect(providers[0].confidence).toBe(0.65);
});
it('test_extract_java_blocking_stub_returns_consumer', async () => {
@ -306,7 +372,7 @@ public class AuthGrpcService extends AuthServiceGrpc.AuthServiceImplBase {
expect(consumers.length).toBeGreaterThanOrEqual(1);
expect(consumers[0].contractId).toContain('AuthService');
expect(consumers[0].confidence).toBe(0.7);
expect(consumers[0].confidence).toBe(0.55);
});
});
@ -328,7 +394,7 @@ def serve():
expect(providers.length).toBeGreaterThanOrEqual(1);
expect(providers[0].contractId).toContain('AuthService');
expect(providers[0].confidence).toBe(0.8);
expect(providers[0].confidence).toBe(0.65);
});
it('test_extract_python_stub_returns_consumer', async () => {
@ -346,7 +412,7 @@ stub = auth_pb2_grpc.AuthServiceStub(channel)`,
expect(consumers.length).toBeGreaterThanOrEqual(1);
expect(consumers[0].contractId).toContain('AuthService');
expect(consumers[0].confidence).toBe(0.7);
expect(consumers[0].confidence).toBe(0.55);
});
});
@ -372,6 +438,165 @@ export class AuthController {
expect(providers[0].contractId).toContain('Login');
expect(providers[0].confidence).toBe(0.8);
});
it('test_extract_ts_grpc_client_decorator_returns_consumer', async () => {
writeFile(
'proto/auth.proto',
`syntax = "proto3";
package auth.v1;
service AuthService {
rpc Login (LoginRequest) returns (LoginResponse);
}`,
);
writeFile(
'src/auth.client.ts',
`import { GrpcClient } from '@nestjs/microservices';
import type { AuthServiceClient } from './generated/auth';
export class AuthGateway {
@GrpcClient({ package: 'auth.v1', protoPath: 'proto/auth.proto' })
private readonly authClient!: AuthServiceClient;
}`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const consumers = contracts.filter((c) => c.role === 'consumer');
expect(consumers).toHaveLength(1);
expect(consumers[0].contractId).toBe('grpc::auth.v1.AuthService/*');
});
it('test_extract_ts_getService_without_decorator_returns_consumer', async () => {
writeFile(
'proto/auth.proto',
`syntax = "proto3";
package auth.v1;
service AuthService {
rpc Login (LoginRequest) returns (LoginResponse);
}`,
);
writeFile(
'src/auth.client.ts',
`import type { ClientGrpc } from '@nestjs/microservices';
export function createAuthClient(client: ClientGrpc) {
return client.getService<AuthService>('AuthService');
}`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const consumers = contracts.filter((c) => c.role === 'consumer');
expect(consumers).toHaveLength(1);
expect(consumers[0].contractId).toBe('grpc::auth.v1.AuthService/*');
});
it('test_extract_ts_generated_client_constructor_returns_consumer', async () => {
writeFile(
'proto/auth.proto',
`syntax = "proto3";
package auth.v1;
service AuthService {
rpc Login (LoginRequest) returns (LoginResponse);
}`,
);
writeFile(
'src/auth.client.ts',
`import { credentials } from '@grpc/grpc-js';
import { AuthServiceClient } from './generated/auth';
export const authClient = new AuthServiceClient('localhost:50051', credentials.createInsecure());`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const consumers = contracts.filter((c) => c.role === 'consumer');
expect(consumers).toHaveLength(1);
expect(consumers[0].contractId).toBe('grpc::auth.v1.AuthService/*');
});
it('test_extract_ts_non_service_client_constructor_is_ignored', async () => {
writeFile(
'proto/auth.proto',
`syntax = "proto3";
package auth.v1;
service AuthService {
rpc Login (LoginRequest) returns (LoginResponse);
}`,
);
writeFile(
'src/auth.client.ts',
`import { AuthClient } from './generated/auth';
export const authClient = new AuthClient('localhost:50051');`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const consumers = contracts.filter((c) => c.role === 'consumer');
expect(consumers).toHaveLength(0);
});
it('test_extract_ts_loadPackageDefinition_constructor_returns_consumer', async () => {
writeFile(
'proto/auth.proto',
`syntax = "proto3";
package auth.v1;
service AuthService {
rpc Login (LoginRequest) returns (LoginResponse);
}`,
);
writeFile(
'src/auth.client.ts',
`import * as grpc from '@grpc/grpc-js';
import * as protoLoader from '@grpc/proto-loader';
const definition = protoLoader.loadSync('proto/auth.proto');
const authProto = grpc.loadPackageDefinition(definition) as any;
export const authClient = new authProto.auth.v1.AuthService(
'localhost:50051',
grpc.credentials.createInsecure(),
);`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const consumers = contracts.filter((c) => c.role === 'consumer');
expect(consumers).toHaveLength(1);
expect(consumers[0].contractId).toBe('grpc::auth.v1.AuthService/*');
});
it('test_extract_ts_duplicate_consumer_patterns_in_one_file_dedupes_deterministically', async () => {
writeFile(
'proto/auth.proto',
`syntax = "proto3";
package auth.v1;
service AuthService {
rpc Login (LoginRequest) returns (LoginResponse);
}`,
);
writeFile(
'src/auth.client.ts',
`import * as grpc from '@grpc/grpc-js';
import type { ClientGrpc } from '@nestjs/microservices';
import { AuthServiceClient } from './generated/auth';
export class AuthGateway {
constructor(private readonly client: ClientGrpc) {}
connect() {
this.client.getService<AuthService>('AuthService');
return new AuthServiceClient('localhost:50051', grpc.credentials.createInsecure());
}
}`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const consumers = contracts.filter((c) => c.role === 'consumer');
expect(consumers).toHaveLength(1);
expect(consumers[0].contractId).toBe('grpc::auth.v1.AuthService/*');
});
});
describe('edge cases', () => {
@ -389,3 +614,297 @@ export class AuthController {
});
});
});
describe('buildProtoMap', () => {
let tmpDir: string;
beforeEach(async () => {
tmpDir = await fsp.mkdtemp(path.join(os.tmpdir(), 'proto-test-'));
});
afterEach(async () => {
await fsp.rm(tmpDir, { recursive: true, force: true });
});
it('test_buildProtoMap_single_proto_parses_package_service_methods', async () => {
const protoContent = `
syntax = "proto3";
package com.example;
service UserService {
rpc GetUser (GetUserRequest) returns (GetUserResponse);
rpc ListUsers (ListUsersRequest) returns (ListUsersResponse);
}`;
await fsp.mkdir(path.join(tmpDir, 'proto'), { recursive: true });
await fsp.writeFile(path.join(tmpDir, 'proto', 'user.proto'), protoContent);
const map = await buildProtoMap(tmpDir);
expect(map.has('UserService')).toBe(true);
const entries = map.get('UserService')!;
expect(entries).toHaveLength(1);
expect(entries[0].package).toBe('com.example');
expect(entries[0].serviceName).toBe('UserService');
expect(entries[0].methods).toEqual(['GetUser', 'ListUsers']);
expect(entries[0].protoPath).toBe('proto/user.proto');
});
it('test_buildProtoMap_no_package_declaration', async () => {
const protoContent = `
syntax = "proto3";
service Foo { rpc Bar (Req) returns (Res); }`;
await fsp.writeFile(path.join(tmpDir, 'foo.proto'), protoContent);
const map = await buildProtoMap(tmpDir);
const entries = map.get('Foo')!;
expect(entries[0].package).toBe('');
});
it('test_buildProtoMap_no_protos_returns_empty', async () => {
const map = await buildProtoMap(tmpDir);
expect(map.size).toBe(0);
});
it('test_buildProtoMap_conflicting_names', async () => {
await fsp.mkdir(path.join(tmpDir, 'a'), { recursive: true });
await fsp.mkdir(path.join(tmpDir, 'b'), { recursive: true });
await fsp.writeFile(
path.join(tmpDir, 'a', 'svc.proto'),
'package pkg.a;\nservice Svc { rpc Do (R) returns (R); }',
);
await fsp.writeFile(
path.join(tmpDir, 'b', 'svc.proto'),
'package pkg.b;\nservice Svc { rpc Do (R) returns (R); }',
);
const map = await buildProtoMap(tmpDir);
expect(map.get('Svc')).toHaveLength(2);
});
it('test_buildProtoMap_imported_package_is_inherited_for_split_service_definition', async () => {
await fsp.mkdir(path.join(tmpDir, 'proto', 'shared'), { recursive: true });
await fsp.mkdir(path.join(tmpDir, 'proto', 'services'), { recursive: true });
await fsp.writeFile(
path.join(tmpDir, 'proto', 'shared', 'package.proto'),
'package auth.v1;\nmessage LoginRequest {}',
);
await fsp.writeFile(
path.join(tmpDir, 'proto', 'services', 'auth.proto'),
'import "../shared/package.proto";\nservice AuthService { rpc Login (LoginRequest) returns (LoginRequest); }',
);
const map = await buildProtoMap(tmpDir);
const entries = map.get('AuthService')!;
expect(entries).toHaveLength(1);
expect(entries[0].package).toBe('auth.v1');
});
});
describe('resolveProtoConflict', () => {
const makeInfo = (pkg: string, protoPath: string): ProtoServiceInfo => ({
package: pkg,
serviceName: 'Svc',
methods: ['Do'],
protoPath,
});
it('test_single_candidate_returns_it', () => {
const result = resolveProtoConflict('Svc', 'src/main.go', [makeInfo('pkg', 'proto/svc.proto')]);
expect(result?.package).toBe('pkg');
});
it('test_multiple_candidates_picks_closest_directory', () => {
const candidates = [
makeInfo('far', 'other/dir/svc.proto'),
makeInfo('close', 'src/proto/svc.proto'),
];
const result = resolveProtoConflict('Svc', 'src/server.go', candidates);
expect(result?.package).toBe('close');
});
it('test_centralized_proto_layout_prefers_shared_path_segments_over_prefix_only', () => {
const candidates = [
makeInfo('billing', 'proto/services/billing/svc.proto'),
makeInfo('auth', 'proto/services/auth/svc.proto'),
];
const result = resolveProtoConflict('Svc', 'services/auth/src/server.ts', candidates);
expect(result?.package).toBe('auth');
});
it('test_no_candidates_returns_null', () => {
expect(resolveProtoConflict('Svc', 'src/main.go', [])).toBeNull();
});
});
describe('serviceContractId', () => {
it('test_with_package', () => {
expect(serviceContractId('com.example', 'UserService')).toBe('grpc::com.example.UserService/*');
});
it('test_without_package', () => {
expect(serviceContractId('', 'UserService')).toBe('grpc::UserService/*');
});
});
describe('proto-aware source scanners', () => {
let tmpDir: string;
let extractor: GrpcExtractor;
beforeEach(async () => {
tmpDir = await fsp.mkdtemp(path.join(os.tmpdir(), 'scanner-test-'));
extractor = new GrpcExtractor();
});
afterEach(async () => {
await fsp.rm(tmpDir, { recursive: true, force: true });
});
const makeRepo = (repoPath: string): RepoHandle => ({
id: 'test-repo',
path: '',
repoPath,
storagePath: '',
});
it('test_go_provider_with_proto_uses_canonical_service_id', async () => {
await fsp.mkdir(path.join(tmpDir, 'proto'), { recursive: true });
await fsp.writeFile(
path.join(tmpDir, 'proto', 'user.proto'),
'package com.example;\nservice UserService { rpc GetUser (R) returns (R); }',
);
await fsp.mkdir(path.join(tmpDir, 'src'), { recursive: true });
await fsp.writeFile(
path.join(tmpDir, 'src', 'server.go'),
'package main\nfunc init() { pb.RegisterUserServiceServer(srv, &impl{}) }',
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const goProvider = contracts.find((c) => c.meta.source === 'go_register');
expect(goProvider).toBeDefined();
expect(goProvider!.contractId).toBe('grpc::com.example.UserService/*');
expect(goProvider!.confidence).toBe(0.8);
});
it('test_go_provider_without_proto_reduced_confidence', async () => {
await fsp.mkdir(path.join(tmpDir, 'src'), { recursive: true });
await fsp.writeFile(
path.join(tmpDir, 'src', 'server.go'),
'package main\nfunc init() { pb.RegisterFooServer(srv, &impl{}) }',
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const goProvider = contracts.find((c) => c.meta.source === 'go_register');
expect(goProvider).toBeDefined();
expect(goProvider!.contractId).toBe('grpc::Foo/*');
expect(goProvider!.confidence).toBe(0.65);
});
it('test_go_consumer_with_proto_uses_canonical_service_id', async () => {
await fsp.mkdir(path.join(tmpDir, 'proto'), { recursive: true });
await fsp.writeFile(
path.join(tmpDir, 'proto', 'user.proto'),
'package com.example;\nservice UserService { rpc GetUser (R) returns (R); }',
);
await fsp.mkdir(path.join(tmpDir, 'src'), { recursive: true });
await fsp.writeFile(
path.join(tmpDir, 'src', 'client.go'),
'package main\nfunc init() { client := pb.NewUserServiceClient(conn) }',
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const goConsumer = contracts.find((c) => c.meta.source === 'go_client');
expect(goConsumer).toBeDefined();
expect(goConsumer!.contractId).toBe('grpc::com.example.UserService/*');
expect(goConsumer!.confidence).toBe(0.75);
});
it('test_java_provider_with_proto_uses_canonical_service_id', async () => {
await fsp.mkdir(path.join(tmpDir, 'proto'), { recursive: true });
await fsp.writeFile(
path.join(tmpDir, 'proto', 'user.proto'),
'package com.example;\nservice UserService { rpc GetUser (R) returns (R); }',
);
await fsp.mkdir(path.join(tmpDir, 'src', 'main', 'java'), { recursive: true });
await fsp.writeFile(
path.join(tmpDir, 'src', 'main', 'java', 'UserGrpcService.java'),
`@GrpcService
public class UserGrpcService extends UserServiceGrpc.UserServiceImplBase {
@Override
public void getUser(GetUserRequest req, StreamObserver<GetUserResponse> obs) {}
}`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const javaProvider = contracts.find((c) => c.meta.source === 'java_grpc_service');
expect(javaProvider).toBeDefined();
expect(javaProvider!.contractId).toBe('grpc::com.example.UserService/*');
expect(javaProvider!.confidence).toBe(0.8);
});
it('test_python_consumer_with_proto_uses_canonical_service_id', async () => {
await fsp.mkdir(path.join(tmpDir, 'proto'), { recursive: true });
await fsp.writeFile(
path.join(tmpDir, 'proto', 'user.proto'),
'package com.example;\nservice UserService { rpc GetUser (R) returns (R); }',
);
await fsp.writeFile(
path.join(tmpDir, 'client.py'),
`import grpc
channel = grpc.insecure_channel('localhost:50051')
stub = UserServiceStub(channel)`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const pyConsumer = contracts.find((c) => c.meta.source === 'python_stub');
expect(pyConsumer).toBeDefined();
expect(pyConsumer!.contractId).toBe('grpc::com.example.UserService/*');
expect(pyConsumer!.confidence).toBe(0.75);
});
it('test_ts_provider_with_proto_adds_package', async () => {
await fsp.mkdir(path.join(tmpDir, 'proto'), { recursive: true });
await fsp.writeFile(
path.join(tmpDir, 'proto', 'user.proto'),
'package com.example;\nservice UserService { rpc GetUser (R) returns (R); }',
);
await fsp.mkdir(path.join(tmpDir, 'src'), { recursive: true });
await fsp.writeFile(
path.join(tmpDir, 'src', 'controller.ts'),
"@GrpcMethod('UserService', 'GetUser')\nasync getUser() {}",
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const tsProvider = contracts.find((c) => c.meta.source === 'ts_grpc_method');
expect(tsProvider).toBeDefined();
expect(tsProvider!.contractId).toBe('grpc::com.example.UserService/GetUser');
expect(tsProvider!.confidence).toBe(0.8);
});
it('test_proto_provider_inherits_package_from_imported_definition', async () => {
await fsp.mkdir(path.join(tmpDir, 'proto', 'shared'), { recursive: true });
await fsp.mkdir(path.join(tmpDir, 'proto', 'services'), { recursive: true });
await fsp.writeFile(
path.join(tmpDir, 'proto', 'shared', 'package.proto'),
'package auth.v1;\nmessage LoginRequest {}',
);
await fsp.writeFile(
path.join(tmpDir, 'proto', 'services', 'auth.proto'),
`syntax = "proto3";
import "../shared/package.proto";
service AuthService {
rpc Login (LoginRequest) returns (LoginRequest);
}`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const protoProvider = contracts.find(
(c) => c.symbolRef.filePath === 'proto/services/auth.proto',
);
expect(protoProvider).toBeDefined();
expect(protoProvider!.contractId).toBe('grpc::auth.v1.AuthService/Login');
});
});

240
gitnexus/test/unit/group/http-route-extractor.test.ts
View file

@ -157,6 +157,89 @@ export default router;
providers.find((c) => c.contractId === 'http::DELETE::/api/users/{param}'),
).toBeDefined();
});
it('extracts Go Gin and Echo route registrations', async () => {
const dir = path.join(tmpDir, 'go-frameworks');
fs.mkdirSync(path.join(dir, 'cmd'), { recursive: true });
fs.writeFileSync(
path.join(dir, 'cmd', 'server.go'),
`
package main
func createOrder(c *gin.Context) {}
func listOrders(c echo.Context) error { return nil }
func main() {
r := gin.Default()
r.POST("/api/orders/:id", createOrder)
e := echo.New()
e.GET("/api/orders", listOrders)
}
`,
);
const contracts = await extractor.extract(null, dir, makeRepo(dir));
const providers = contracts.filter((c) => c.role === 'provider');
const ginRoute = providers.find((c) => c.contractId === 'http::POST::/api/orders/{param}');
expect(ginRoute).toBeDefined();
expect(ginRoute?.symbolName).toBe('createOrder');
const echoRoute = providers.find((c) => c.contractId === 'http::GET::/api/orders');
expect(echoRoute).toBeDefined();
expect(echoRoute?.symbolName).toBe('listOrders');
});
it('extracts stdlib HandleFunc providers', async () => {
const dir = path.join(tmpDir, 'go-stdlib-provider');
fs.mkdirSync(path.join(dir, 'cmd'), { recursive: true });
fs.writeFileSync(
path.join(dir, 'cmd', 'server.go'),
`
package main
func healthHandler(w http.ResponseWriter, r *http.Request) {}
func main() {
http.HandleFunc("/api/health", healthHandler)
}
`,
);
const contracts = await extractor.extract(null, dir, makeRepo(dir));
const providers = contracts.filter((c) => c.role === 'provider');
const healthRoute = providers.find((c) => c.contractId === 'http::GET::/api/health');
expect(healthRoute).toBeDefined();
expect(healthRoute?.symbolName).toBe('healthHandler');
});
it('extracts NestJS controller decorators', async () => {
const dir = path.join(tmpDir, 'nestjs');
fs.mkdirSync(path.join(dir, 'src'), { recursive: true });
fs.writeFileSync(
path.join(dir, 'src', 'orders.controller.ts'),
`
import { Controller, Patch } from '@nestjs/common';
@Controller('orders')
export class OrdersController {
@Patch(':id')
updateOrder() {
return {};
}
}
`,
);
const contracts = await extractor.extract(null, dir, makeRepo(dir));
const providers = contracts.filter((c) => c.role === 'provider');
const patchRoute = providers.find((c) => c.contractId === 'http::PATCH::/orders/{param}');
expect(patchRoute).toBeDefined();
expect(patchRoute?.symbolName).toBe('updateOrder');
});
});
describe('consumer extraction — fetch patterns', () => {
@ -206,6 +289,91 @@ export const deleteUser = (id: string) => axios.delete(\`/api/users/\${id}\`);
consumers.find((c) => c.contractId === 'http::DELETE::/api/users/{param}'),
).toBeDefined();
});
it('extracts Python requests calls', async () => {
const dir = path.join(tmpDir, 'python-consumer');
fs.mkdirSync(path.join(dir, 'src'), { recursive: true });
fs.writeFileSync(
path.join(dir, 'src', 'client.py'),
`
import requests
def create_order():
return requests.post("https://svc.local/api/orders/42", json={"id": 42})
`,
);
const contracts = await extractor.extract(null, dir, makeRepo(dir));
const consumers = contracts.filter((c) => c.role === 'consumer');
expect(
consumers.find((c) => c.contractId === 'http::POST::/api/orders/{param}'),
).toBeDefined();
});
it('extracts Java RestTemplate, WebClient and OkHttp calls', async () => {
const dir = path.join(tmpDir, 'java-consumer');
fs.mkdirSync(path.join(dir, 'src'), { recursive: true });
fs.writeFileSync(
path.join(dir, 'src', 'ApiClient.java'),
`
import org.springframework.http.HttpMethod;
import org.springframework.web.client.RestTemplate;
import org.springframework.web.reactive.function.client.WebClient;
import okhttp3.Request;
class ApiClient {
void run(RestTemplate restTemplate, WebClient webClient) {
restTemplate.getForObject("/api/users/{id}", String.class, 42);
webClient.method(HttpMethod.PATCH, "/api/users/42");
new Request.Builder().url("/api/orders/42").build();
}
}
`,
);
const contracts = await extractor.extract(null, dir, makeRepo(dir));
const consumers = contracts.filter((c) => c.role === 'consumer');
expect(consumers.find((c) => c.contractId === 'http::GET::/api/users/{param}')).toBeDefined();
expect(
consumers.find((c) => c.contractId === 'http::PATCH::/api/users/{param}'),
).toBeDefined();
expect(
consumers.find((c) => c.contractId === 'http::GET::/api/orders/{param}'),
).toBeDefined();
});
it('extracts Go stdlib and resty calls', async () => {
const dir = path.join(tmpDir, 'go-consumer');
fs.mkdirSync(path.join(dir, 'cmd'), { recursive: true });
fs.writeFileSync(
path.join(dir, 'cmd', 'client.go'),
`
package main
import (
"net/http"
"github.com/go-resty/resty/v2"
)
func main() {
http.Get("/api/health")
client := resty.New()
client.R().Delete("/api/orders/42")
}
`,
);
const contracts = await extractor.extract(null, dir, makeRepo(dir));
const consumers = contracts.filter((c) => c.role === 'consumer');
expect(consumers.find((c) => c.contractId === 'http::GET::/api/health')).toBeDefined();
expect(
consumers.find((c) => c.contractId === 'http::DELETE::/api/orders/{param}'),
).toBeDefined();
});
});
describe('provider extraction — Laravel', () => {
@ -326,78 +494,6 @@ async def create_user(user: UserCreate):
});
});
describe('interface regex anchoring', () => {
it('skips Feign client interfaces (no @Controller)', async () => {
const dir = path.join(tmpDir, 'feign-skip');
fs.mkdirSync(path.join(dir, 'src'), { recursive: true });
fs.writeFileSync(
path.join(dir, 'src/UserClient.java'),
`
package com.example;
@FeignClient(name = "user-service")
public interface UserClient {
@GetMapping("/users")
List<User> getUsers();
}
`,
);
const contracts = await extractor.extract(null, dir, makeRepo(dir));
expect(contracts.filter((c) => c.role === 'provider')).toHaveLength(0);
});
it('does NOT skip when @RestController is present', async () => {
const dir = path.join(tmpDir, 'ctrl-iface');
fs.mkdirSync(path.join(dir, 'src'), { recursive: true });
fs.writeFileSync(
path.join(dir, 'src/UserController.java'),
`
@RestController
@RequestMapping("/api")
public class UserController {
@GetMapping("/users")
public List<User> list() { return null; }
}
`,
);
const contracts = await extractor.extract(null, dir, makeRepo(dir));
expect(contracts.filter((c) => c.role === 'provider').length).toBeGreaterThanOrEqual(1);
});
it('does NOT false-positive on interface in comments', async () => {
const dir = path.join(tmpDir, 'iface-comment');
fs.mkdirSync(path.join(dir, 'src'), { recursive: true });
fs.writeFileSync(
path.join(dir, 'src/Api.java'),
`
// implements the interface UserApi
public class Api {
@GetMapping("/health")
public String health() { return "ok"; }
}
`,
);
const contracts = await extractor.extract(null, dir, makeRepo(dir));
expect(contracts.filter((c) => c.role === 'provider').length).toBeGreaterThanOrEqual(1);
});
it('does NOT false-positive on interface in a string', async () => {
const dir = path.join(tmpDir, 'iface-str');
fs.mkdirSync(path.join(dir, 'src'), { recursive: true });
fs.writeFileSync(
path.join(dir, 'src/Svc.java'),
`
public class Svc {
String desc = "implements interface Foo";
@GetMapping("/status")
public String status() { return desc; }
}
`,
);
const contracts = await extractor.extract(null, dir, makeRepo(dir));
expect(contracts.filter((c) => c.role === 'provider').length).toBeGreaterThanOrEqual(1);
});
});
describe('path normalization', () => {
it('strips trailing slash', async () => {
const dir = path.join(tmpDir, 'trailing');

308
gitnexus/test/unit/group/manifest-extractor.test.ts Normal file
View file

@ -0,0 +1,308 @@
import { describe, it, expect } from 'vitest';
import { ManifestExtractor } from '../../../src/core/group/extractors/manifest-extractor.js';
import type { GroupManifestLink } from '../../../src/core/group/types.js';
describe('ManifestExtractor', () => {
const extractor = new ManifestExtractor();
it('creates provider + consumer contracts and a cross-link for each manifest link', async () => {
const links: GroupManifestLink[] = [
{
from: 'hr/payroll/backend',
to: 'hr/hiring/backend',
type: 'topic',
contract: 'employee.hired',
role: 'provider',
},
];
const result = await extractor.extractFromManifest(links);
expect(result.contracts).toHaveLength(2);
const provider = result.contracts.find((c) => c.role === 'provider');
expect(provider).toBeDefined();
expect(provider!.contractId).toBe('topic::employee.hired');
expect(provider!.type).toBe('topic');
expect(provider!.confidence).toBe(1.0);
const consumer = result.contracts.find((c) => c.role === 'consumer');
expect(consumer).toBeDefined();
expect(consumer!.contractId).toBe('topic::employee.hired');
expect(result.crossLinks).toHaveLength(1);
expect(result.crossLinks[0].matchType).toBe('manifest');
expect(result.crossLinks[0].confidence).toBe(1.0);
expect(result.crossLinks[0].from.repo).toBe('hr/hiring/backend');
expect(result.crossLinks[0].to.repo).toBe('hr/payroll/backend');
});
it('handles role: consumer (from-repo is consumer)', async () => {
const links: GroupManifestLink[] = [
{
from: 'sales/admin/bff',
to: 'sales/crm/backend',
type: 'http',
contract: '/api/v2/leads/*',
role: 'consumer',
},
];
const result = await extractor.extractFromManifest(links);
const provider = result.contracts.find((c) => c.role === 'provider');
const consumer = result.contracts.find((c) => c.role === 'consumer');
expect(consumer!.contractId).toBe('http::*::/api/v2/leads/*');
expect(provider!.contractId).toBe('http::*::/api/v2/leads/*');
expect(result.crossLinks[0].from.repo).toBe('sales/admin/bff');
expect(result.crossLinks[0].to.repo).toBe('sales/crm/backend');
});
it('resolves grpc manifest provider by exact method name (no .proto fallback)', async () => {
const links: GroupManifestLink[] = [
{
from: 'platform/orders',
to: 'platform/auth',
type: 'grpc',
contract: 'auth.AuthService/Login',
role: 'consumer',
},
];
const dbExecutors = new Map<
string,
(cypher: string, params?: Record<string, unknown>) => Promise<Record<string, unknown>[]>
>([
[
'platform/auth',
async (_cypher, params) => {
// Exact match on method name.
if (params?.methodName === 'Login') {
return [
{
uid: 'uid-auth-login',
name: 'Login',
filePath: 'src/auth.proto',
},
];
}
return [];
},
],
[
'platform/orders',
async (_cypher, params) => {
// No symbol with the exact method name — resolve returns null and
// the consumer contract gets an empty symbolUid, falling back to
// name-based hint at cross-impact time.
if (params?.methodName === 'Login') return [];
return [];
},
],
]);
const result = await extractor.extractFromManifest(links, dbExecutors);
const provider = result.contracts.find((c) => c.role === 'provider');
const consumer = result.contracts.find((c) => c.role === 'consumer');
// Provider resolved to the concrete proto symbol.
expect(provider?.symbolUid).toBe('uid-auth-login');
expect(provider?.symbolRef.filePath).toBe('src/auth.proto');
// Consumer falls back to a deterministic synthetic uid + name-based ref.
// The synthetic uid lets the bridge cross-impact query anchor on it
// even when the indexer doesn't expose a matching symbol.
expect(consumer?.symbolUid).toBe('manifest::platform/orders::grpc::auth.AuthService/Login');
expect(consumer?.symbolRef.name).toBe('auth.AuthService/Login');
expect(result.crossLinks[0].to.symbolRef.filePath).toBe('src/auth.proto');
expect(result.crossLinks[0].from.symbolUid).toBe(
'manifest::platform/orders::grpc::auth.AuthService/Login',
);
});
it('does NOT resolve grpc manifest to an arbitrary .proto file', async () => {
// Regression test for a previous bug: the extractor had an unconditional
// `OR n.filePath ENDS WITH '.proto'` fallback that returned the first
// proto symbol in the repo, regardless of whether it matched the contract.
const links: GroupManifestLink[] = [
{
from: 'platform/orders',
to: 'platform/auth',
type: 'grpc',
contract: 'auth.AuthService/Login',
role: 'consumer',
},
];
const dbExecutors = new Map<
string,
(cypher: string, params?: Record<string, unknown>) => Promise<Record<string, unknown>[]>
>([
[
'platform/auth',
// Executor returns matches for ANY query (simulates the old buggy
// fallback that returned a random .proto file). The new code must
// only accept a hit when the method/service name matches exactly.
async (_cypher, params) => {
if (params?.methodName === 'Login' || params?.serviceName === 'auth.AuthService') {
return [
{
uid: 'uid-correct-login',
name: 'Login',
filePath: 'src/auth.proto',
},
];
}
return [];
},
],
['platform/orders', async () => []],
]);
const result = await extractor.extractFromManifest(links, dbExecutors);
const provider = result.contracts.find((c) => c.role === 'provider');
// Must resolve to the correct symbol (not a random proto one).
expect(provider?.symbolUid).toBe('uid-correct-login');
});
it('resolves lib manifest links by exact name only', async () => {
const links: GroupManifestLink[] = [
{
from: 'platform/web',
to: 'platform/shared-lib',
type: 'lib',
contract: '@platform/contracts',
role: 'consumer',
},
];
const dbExecutors = new Map<
string,
(cypher: string, params?: Record<string, unknown>) => Promise<Record<string, unknown>[]>
>([
[
'platform/shared-lib',
async (_cypher, params) => {
if (params?.contract !== '@platform/contracts') return [];
return [
{
uid: 'uid-lib',
name: '@platform/contracts',
filePath: 'src/index.ts',
},
];
},
],
[
'platform/web',
async (_cypher, params) => {
if (params?.contract !== '@platform/contracts') return [];
return [];
},
],
]);
const result = await extractor.extractFromManifest(links, dbExecutors);
const provider = result.contracts.find((c) => c.role === 'provider');
const consumer = result.contracts.find((c) => c.role === 'consumer');
expect(provider?.symbolUid).toBe('uid-lib');
// Consumer doesn't have a symbol named exactly '@platform/contracts' —
// exact matching returns null, falling back to the synthetic manifest uid.
expect(consumer?.symbolUid).toBe('manifest::platform/web::lib::@platform/contracts');
});
it('does NOT resolve lib manifest via CONTAINS on name', async () => {
// Regression test: previous CONTAINS fallback would match "react" to
// "react-native" or "@types/react". Exact matching must reject both.
const links: GroupManifestLink[] = [
{
from: 'web',
to: 'packages/ui',
type: 'lib',
contract: 'react',
role: 'consumer',
},
];
const dbExecutors = new Map<
string,
(cypher: string, params?: Record<string, unknown>) => Promise<Record<string, unknown>[]>
>([
[
'packages/ui',
async (_cypher, params) => {
// Executor is called with contract='react'. Only exact matches
// should come back; return only wrong candidates to verify the
// Cypher uses `=` not `CONTAINS`.
if (params?.contract === 'react') {
// Simulated DB returns nothing because it has only "react-native"
// and "@types/react" — neither is an exact match for "react".
return [];
}
return [];
},
],
['web', async () => []],
]);
const result = await extractor.extractFromManifest(links, dbExecutors);
const provider = result.contracts.find((c) => c.role === 'provider');
// No exact match → synthetic manifest uid, not a wrong real one.
expect(provider?.symbolUid).toBe('manifest::packages/ui::lib::react');
});
it('normalizes http contract path for exact Route.name match', async () => {
// Manifest may be written as "/api/orders/" or "api/orders"; both should
// match the canonical "/api/orders" stored in the graph.
const variants = ['/api/orders', '/api/orders/', 'api/orders', '//api//orders'];
for (const raw of variants) {
const links: GroupManifestLink[] = [
{
from: 'gateway',
to: 'orders-svc',
type: 'http',
contract: raw,
role: 'consumer',
},
];
let seenParam: string | undefined;
const dbExecutors = new Map<
string,
(cypher: string, params?: Record<string, unknown>) => Promise<Record<string, unknown>[]>
>([
[
'orders-svc',
async (_cypher, params) => {
seenParam = params?.normalized as string;
return [
{
uid: 'uid-orders-list',
name: 'listOrders',
filePath: 'src/orders.ts',
},
];
},
],
['gateway', async () => []],
]);
const result = await extractor.extractFromManifest(links, dbExecutors);
expect(seenParam).toBe('/api/orders');
const provider = result.contracts.find((c) => c.role === 'provider');
expect(provider?.symbolUid).toBe('uid-orders-list');
}
});
it('returns empty for no links', async () => {
const result = await extractor.extractFromManifest([]);
expect(result.contracts).toHaveLength(0);
expect(result.crossLinks).toHaveLength(0);
});
});

177
gitnexus/test/unit/group/topic-extractor.test.ts
View file

@ -75,8 +75,7 @@ public void handleUserCreated(ConsumerRecord<String, String> record) {
it('test_extract_kafkajs_subscribe_returns_consumer', async () => {
writeFile(
'src/consumer.ts',
`await consumer.subscribe({ topic: 'order.placed', fromBeginning: true });
await consumer.run({ eachMessage: async ({ message }) => {} });`,
`await consumer.subscribe({ topic: 'order.placed', fromBeginning: true });`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
@ -101,6 +100,23 @@ await consumer.run({ eachMessage: async ({ message }) => {} });`,
});
});
describe('KafkaJS consumer run', () => {
it('test_extract_kafkajs_consumer_run_eachmessage_returns_consumer', async () => {
writeFile(
'src/consumer.ts',
`await consumer.subscribe({ topic: 'user.logged-in' });
await consumer.run({ eachMessage: async () => {} });`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const consumers = contracts.filter((c) => c.role === 'consumer');
expect(consumers).toHaveLength(1);
expect(consumers[0].contractId).toBe('topic::user.logged-in');
expect(consumers[0].meta.broker).toBe('kafka');
});
});
describe('RabbitMQ — Java', () => {
it('test_extract_rabbit_listener_returns_consumer', async () => {
writeFile(
@ -174,6 +190,62 @@ public void processOrder(OrderMessage msg) {}`,
});
});
describe('JetStream', () => {
it('test_extract_jetstream_publish_returns_provider', async () => {
writeFile('src/stream.go', `js.Publish("orders.created", payload)`);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const producers = contracts.filter((c) => c.role === 'provider');
expect(producers).toHaveLength(1);
expect(producers[0].contractId).toBe('topic::orders.created');
expect(producers[0].meta.broker).toBe('nats');
});
it('test_extract_jetstream_subscribe_returns_consumer', async () => {
writeFile('src/stream.go', `js.Subscribe("orders.created", handler)`);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const consumers = contracts.filter((c) => c.role === 'consumer');
expect(consumers).toHaveLength(1);
expect(consumers[0].contractId).toBe('topic::orders.created');
expect(consumers[0].meta.broker).toBe('nats');
});
});
describe('Python NATS', () => {
it('test_extract_python_nats_subscribe_returns_consumer', async () => {
writeFile(
'src/subscriber.py',
`nc = await nats.connect()
await nc.subscribe("orders.created", cb=handler)`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const consumers = contracts.filter((c) => c.role === 'consumer');
expect(consumers).toHaveLength(1);
expect(consumers[0].contractId).toBe('topic::orders.created');
expect(consumers[0].meta.broker).toBe('nats');
});
it('test_extract_python_nats_publish_returns_provider', async () => {
writeFile(
'src/publisher.py',
`nc = await nats.connect()
await nc.publish("orders.created", payload)`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const producers = contracts.filter((c) => c.role === 'provider');
expect(producers).toHaveLength(1);
expect(producers[0].contractId).toBe('topic::orders.created');
expect(producers[0].meta.broker).toBe('nats');
});
});
describe('NATS', () => {
it('test_extract_nats_subscribe_go_returns_consumer', async () => {
writeFile(
@ -248,6 +320,96 @@ partConsumer, _ := consumer.ConsumePartition("inventory.update", 0, sarama.Offse
expect(consumers[0].contractId).toBe('topic::inventory.update');
expect(consumers[0].meta.broker).toBe('kafka');
});
it('test_extract_sarama_sync_producer_returns_provider', async () => {
writeFile(
'internal/publisher.go',
`package publisher
producer, _ := sarama.NewSyncProducer(brokers, cfg)
producer.SendMessage(&sarama.ProducerMessage{Topic: "inventory.update"})`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const producers = contracts.filter((c) => c.role === 'provider');
expect(producers).toHaveLength(1);
expect(producers[0].contractId).toBe('topic::inventory.update');
expect(producers[0].meta.broker).toBe('kafka');
});
it('test_extract_sarama_async_producer_returns_provider', async () => {
writeFile(
'internal/publisher.go',
`package publisher
producer, _ := sarama.NewAsyncProducer(brokers, cfg)
producer.Input() <- &sarama.ProducerMessage{Topic: "inventory.update"}`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const producers = contracts.filter((c) => c.role === 'provider');
expect(producers).toHaveLength(1);
expect(producers[0].contractId).toBe('topic::inventory.update');
expect(producers[0].meta.broker).toBe('kafka');
});
it('test_extract_sarama_producer_in_loop_captures_all_topics', async () => {
// Regression: a for loop that constructs multiple ProducerMessage
// literals inside a single NewSyncProducer scope. The previous
// regex anchored on NewSyncProducer and captured only the first
// Topic within 300 chars, silently dropping the rest.
writeFile(
'internal/multi-publisher.go',
`package publisher
func publishAll(producer sarama.SyncProducer, items []Item) error {
_, _ = sarama.NewSyncProducer(brokers, cfg)
for _, item := range items {
msg1 := &sarama.ProducerMessage{Topic: "order.created"}
msg2 := &sarama.ProducerMessage{Topic: "order.shipped"}
_ = msg1
_ = msg2
}
return nil
}`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const producers = contracts.filter((c) => c.role === 'provider');
const topics = producers.map((c) => c.contractId).sort();
// Both topics must appear (exact set match to catch any duplicates).
expect(topics).toEqual(['topic::order.created', 'topic::order.shipped']);
});
it('test_extract_kafka_go_writer_returns_provider', async () => {
writeFile(
'internal/writer.go',
`package publisher
writer := &kafka.Writer{Topic: "inventory.update"}`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const producers = contracts.filter((c) => c.role === 'provider');
expect(producers).toHaveLength(1);
expect(producers[0].contractId).toBe('topic::inventory.update');
expect(producers[0].meta.broker).toBe('kafka');
});
it('test_extract_kafka_go_reader_returns_consumer', async () => {
writeFile(
'internal/reader.go',
`package consumer
reader := kafka.NewReader(kafka.ReaderConfig{Topic: "inventory.update"})`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
const consumers = contracts.filter((c) => c.role === 'consumer');
expect(consumers).toHaveLength(1);
expect(consumers[0].contractId).toBe('topic::inventory.update');
expect(consumers[0].meta.broker).toBe('kafka');
});
});
describe('Kafka — Python', () => {
@ -309,5 +471,16 @@ await consumer.subscribe({ topic: 'order.placed' });`,
expect(producers).toHaveLength(2);
expect(consumers).toHaveLength(1);
});
it('test_extract_ignores_go_test_files', async () => {
writeFile(
'src/orders_test.go',
`consumer.ConsumePartition("fake-topic", 0, sarama.OffsetNewest)`,
);
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
expect(contracts).toEqual([]);
});
});
});

3
gitnexus/vendor/tree-sitter-proto/.gitignore vendored Normal file
View file

@ -0,0 +1,3 @@
build/
node_modules/
package-lock.json

30
gitnexus/vendor/tree-sitter-proto/binding.gyp vendored Normal file
View file

@ -0,0 +1,30 @@
{
"targets": [
{
"target_name": "tree_sitter_proto_binding",
"dependencies": [
"<!(node -p \"require('node-addon-api').targets\"):node_addon_api_except",
],
"include_dirs": [
"src",
],
"sources": [
"bindings/node/binding.cc",
"src/parser.c",
# NOTE: if your language has an external scanner, add it here.
],
"conditions": [
["OS!='win'", {
"cflags_c": [
"-std=c11",
],
}, { # OS == "win"
"cflags_c": [
"/std:c11",
"/utf-8",
],
}],
],
}
]
}

20
gitnexus/vendor/tree-sitter-proto/bindings/node/binding.cc vendored Normal file
View file

@ -0,0 +1,20 @@
#include <napi.h>
typedef struct TSLanguage TSLanguage;
extern "C" TSLanguage *tree_sitter_proto();
// "tree-sitter", "language" hashed with BLAKE2
const napi_type_tag LANGUAGE_TYPE_TAG = {
0x8AF2E5212AD58ABF, 0xD5006CAD83ABBA16
};
Napi::Object Init(Napi::Env env, Napi::Object exports) {
exports["name"] = Napi::String::New(env, "proto");
auto language = Napi::External<TSLanguage>::New(env, tree_sitter_proto());
language.TypeTag(&LANGUAGE_TYPE_TAG);
exports["language"] = language;
return exports;
}
NODE_API_MODULE(tree_sitter_proto_binding, Init)

28
gitnexus/vendor/tree-sitter-proto/bindings/node/index.d.ts vendored Normal file
View file

@ -0,0 +1,28 @@
type BaseNode = {
type: string;
named: boolean;
};
type ChildNode = {
multiple: boolean;
required: boolean;
types: BaseNode[];
};
type NodeInfo =
| (BaseNode & {
subtypes: BaseNode[];
})
| (BaseNode & {
fields: { [name: string]: ChildNode };
children: ChildNode[];
});
type Language = {
name: string;
language: unknown;
nodeTypeInfo: NodeInfo[];
};
declare const language: Language;
export = language;

7
gitnexus/vendor/tree-sitter-proto/bindings/node/index.js vendored Normal file
View file

@ -0,0 +1,7 @@
const root = require("path").join(__dirname, "..", "..");
module.exports = require("node-gyp-build")(root);
try {
module.exports.nodeTypeInfo = require("../../src/node-types.json");
} catch (_) {}

18
gitnexus/vendor/tree-sitter-proto/package.json vendored Normal file
View file

@ -0,0 +1,18 @@
{
"name": "tree-sitter-proto",
"version": "0.4.1",
"description": "tree-sitter grammar for protobuf — ABI 14 build from coder3101/tree-sitter-proto latest grammar.js, compatible with tree-sitter 0.25",
"repository": "https://github.com/coder3101/tree-sitter-proto",
"license": "MIT",
"main": "bindings/node",
"scripts": {
"install": "node-gyp-build"
},
"peerDependencies": {
"tree-sitter": ">=0.21.0"
},
"dependencies": {
"node-addon-api": "^8.0.0",
"node-gyp-build": "^4.8.0"
}
}

1145
gitnexus/vendor/tree-sitter-proto/src/node-types.json vendored Normal file
View file

File diff suppressed because it is too large Load diff

10149
gitnexus/vendor/tree-sitter-proto/src/parser.c vendored Normal file
View file

File diff suppressed because it is too large Load diff

54
gitnexus/vendor/tree-sitter-proto/src/tree_sitter/alloc.h vendored Normal file
View file

@ -0,0 +1,54 @@
#ifndef TREE_SITTER_ALLOC_H_
#define TREE_SITTER_ALLOC_H_
#ifdef __cplusplus
extern "C" {
#endif
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
// Allow clients to override allocation functions
#ifdef TREE_SITTER_REUSE_ALLOCATOR
extern void *(*ts_current_malloc)(size_t size);
extern void *(*ts_current_calloc)(size_t count, size_t size);
extern void *(*ts_current_realloc)(void *ptr, size_t size);
extern void (*ts_current_free)(void *ptr);
#ifndef ts_malloc
#define ts_malloc ts_current_malloc
#endif
#ifndef ts_calloc
#define ts_calloc ts_current_calloc
#endif
#ifndef ts_realloc
#define ts_realloc ts_current_realloc
#endif
#ifndef ts_free
#define ts_free ts_current_free
#endif
#else
#ifndef ts_malloc
#define ts_malloc malloc
#endif
#ifndef ts_calloc
#define ts_calloc calloc
#endif
#ifndef ts_realloc
#define ts_realloc realloc
#endif
#ifndef ts_free
#define ts_free free
#endif
#endif
#ifdef __cplusplus
}
#endif
#endif // TREE_SITTER_ALLOC_H_

291
gitnexus/vendor/tree-sitter-proto/src/tree_sitter/array.h vendored Normal file
View file

@ -0,0 +1,291 @@
#ifndef TREE_SITTER_ARRAY_H_
#define TREE_SITTER_ARRAY_H_
#ifdef __cplusplus
extern "C" {
#endif
#include "./alloc.h"
#include <assert.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#ifdef _MSC_VER
#pragma warning(push)
#pragma warning(disable : 4101)
#elif defined(__GNUC__) || defined(__clang__)
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wunused-variable"
#endif
#define Array(T) \
struct { \
T *contents; \
uint32_t size; \
uint32_t capacity; \
}
/// Initialize an array.
#define array_init(self) \
((self)->size = 0, (self)->capacity = 0, (self)->contents = NULL)
/// Create an empty array.
#define array_new() \
{ NULL, 0, 0 }
/// Get a pointer to the element at a given `index` in the array.
#define array_get(self, _index) \
(assert((uint32_t)(_index) < (self)->size), &(self)->contents[_index])
/// Get a pointer to the first element in the array.
#define array_front(self) array_get(self, 0)
/// Get a pointer to the last element in the array.
#define array_back(self) array_get(self, (self)->size - 1)
/// Clear the array, setting its size to zero. Note that this does not free any
/// memory allocated for the array's contents.
#define array_clear(self) ((self)->size = 0)
/// Reserve `new_capacity` elements of space in the array. If `new_capacity` is
/// less than the array's current capacity, this function has no effect.
#define array_reserve(self, new_capacity) \
_array__reserve((Array *)(self), array_elem_size(self), new_capacity)
/// Free any memory allocated for this array. Note that this does not free any
/// memory allocated for the array's contents.
#define array_delete(self) _array__delete((Array *)(self))
/// Push a new `element` onto the end of the array.
#define array_push(self, element) \
(_array__grow((Array *)(self), 1, array_elem_size(self)), \
(self)->contents[(self)->size++] = (element))
/// Increase the array's size by `count` elements.
/// New elements are zero-initialized.
#define array_grow_by(self, count) \
do { \
if ((count) == 0) break; \
_array__grow((Array *)(self), count, array_elem_size(self)); \
memset((self)->contents + (self)->size, 0, (count) * array_elem_size(self)); \
(self)->size += (count); \
} while (0)
/// Append all elements from one array to the end of another.
#define array_push_all(self, other) \
array_extend((self), (other)->size, (other)->contents)
/// Append `count` elements to the end of the array, reading their values from the
/// `contents` pointer.
#define array_extend(self, count, contents) \
_array__splice( \
(Array *)(self), array_elem_size(self), (self)->size, \
0, count, contents \
)
/// Remove `old_count` elements from the array starting at the given `index`. At
/// the same index, insert `new_count` new elements, reading their values from the
/// `new_contents` pointer.
#define array_splice(self, _index, old_count, new_count, new_contents) \
_array__splice( \
(Array *)(self), array_elem_size(self), _index, \
old_count, new_count, new_contents \
)
/// Insert one `element` into the array at the given `index`.
#define array_insert(self, _index, element) \
_array__splice((Array *)(self), array_elem_size(self), _index, 0, 1, &(element))
/// Remove one element from the array at the given `index`.
#define array_erase(self, _index) \
_array__erase((Array *)(self), array_elem_size(self), _index)
/// Pop the last element off the array, returning the element by value.
#define array_pop(self) ((self)->contents[--(self)->size])
/// Assign the contents of one array to another, reallocating if necessary.
#define array_assign(self, other) \
_array__assign((Array *)(self), (const Array *)(other), array_elem_size(self))
/// Swap one array with another
#define array_swap(self, other) \
_array__swap((Array *)(self), (Array *)(other))
/// Get the size of the array contents
#define array_elem_size(self) (sizeof *(self)->contents)
/// Search a sorted array for a given `needle` value, using the given `compare`
/// callback to determine the order.
///
/// If an existing element is found to be equal to `needle`, then the `index`
/// out-parameter is set to the existing value's index, and the `exists`
/// out-parameter is set to true. Otherwise, `index` is set to an index where
/// `needle` should be inserted in order to preserve the sorting, and `exists`
/// is set to false.
#define array_search_sorted_with(self, compare, needle, _index, _exists) \
_array__search_sorted(self, 0, compare, , needle, _index, _exists)
/// Search a sorted array for a given `needle` value, using integer comparisons
/// of a given struct field (specified with a leading dot) to determine the order.
///
/// See also `array_search_sorted_with`.
#define array_search_sorted_by(self, field, needle, _index, _exists) \
_array__search_sorted(self, 0, _compare_int, field, needle, _index, _exists)
/// Insert a given `value` into a sorted array, using the given `compare`
/// callback to determine the order.
#define array_insert_sorted_with(self, compare, value) \
do { \
unsigned _index, _exists; \
array_search_sorted_with(self, compare, &(value), &_index, &_exists); \
if (!_exists) array_insert(self, _index, value); \
} while (0)
/// Insert a given `value` into a sorted array, using integer comparisons of
/// a given struct field (specified with a leading dot) to determine the order.
///
/// See also `array_search_sorted_by`.
#define array_insert_sorted_by(self, field, value) \
do { \
unsigned _index, _exists; \
array_search_sorted_by(self, field, (value) field, &_index, &_exists); \
if (!_exists) array_insert(self, _index, value); \
} while (0)
// Private
typedef Array(void) Array;
/// This is not what you're looking for, see `array_delete`.
static inline void _array__delete(Array *self) {
if (self->contents) {
ts_free(self->contents);
self->contents = NULL;
self->size = 0;
self->capacity = 0;
}
}
/// This is not what you're looking for, see `array_erase`.
static inline void _array__erase(Array *self, size_t element_size,
uint32_t index) {
assert(index < self->size);
char *contents = (char *)self->contents;
memmove(contents + index * element_size, contents + (index + 1) * element_size,
(self->size - index - 1) * element_size);
self->size--;
}
/// This is not what you're looking for, see `array_reserve`.
static inline void _array__reserve(Array *self, size_t element_size, uint32_t new_capacity) {
if (new_capacity > self->capacity) {
if (self->contents) {
self->contents = ts_realloc(self->contents, new_capacity * element_size);
} else {
self->contents = ts_malloc(new_capacity * element_size);
}
self->capacity = new_capacity;
}
}
/// This is not what you're looking for, see `array_assign`.
static inline void _array__assign(Array *self, const Array *other, size_t element_size) {
_array__reserve(self, element_size, other->size);
self->size = other->size;
memcpy(self->contents, other->contents, self->size * element_size);
}
/// This is not what you're looking for, see `array_swap`.
static inline void _array__swap(Array *self, Array *other) {
Array swap = *other;
*other = *self;
*self = swap;
}
/// This is not what you're looking for, see `array_push` or `array_grow_by`.
static inline void _array__grow(Array *self, uint32_t count, size_t element_size) {
uint32_t new_size = self->size + count;
if (new_size > self->capacity) {
uint32_t new_capacity = self->capacity * 2;
if (new_capacity < 8) new_capacity = 8;
if (new_capacity < new_size) new_capacity = new_size;
_array__reserve(self, element_size, new_capacity);
}
}
/// This is not what you're looking for, see `array_splice`.
static inline void _array__splice(Array *self, size_t element_size,
uint32_t index, uint32_t old_count,
uint32_t new_count, const void *elements) {
uint32_t new_size = self->size + new_count - old_count;
uint32_t old_end = index + old_count;
uint32_t new_end = index + new_count;
assert(old_end <= self->size);
_array__reserve(self, element_size, new_size);
char *contents = (char *)self->contents;
if (self->size > old_end) {
memmove(
contents + new_end * element_size,
contents + old_end * element_size,
(self->size - old_end) * element_size
);
}
if (new_count > 0) {
if (elements) {
memcpy(
(contents + index * element_size),
elements,
new_count * element_size
);
} else {
memset(
(contents + index * element_size),
0,
new_count * element_size
);
}
}
self->size += new_count - old_count;
}
/// A binary search routine, based on Rust's `std::slice::binary_search_by`.
/// This is not what you're looking for, see `array_search_sorted_with` or `array_search_sorted_by`.
#define _array__search_sorted(self, start, compare, suffix, needle, _index, _exists) \
do { \
*(_index) = start; \
*(_exists) = false; \
uint32_t size = (self)->size - *(_index); \
if (size == 0) break; \
int comparison; \
while (size > 1) { \
uint32_t half_size = size / 2; \
uint32_t mid_index = *(_index) + half_size; \
comparison = compare(&((self)->contents[mid_index] suffix), (needle)); \
if (comparison <= 0) *(_index) = mid_index; \
size -= half_size; \
} \
comparison = compare(&((self)->contents[*(_index)] suffix), (needle)); \
if (comparison == 0) *(_exists) = true; \
else if (comparison < 0) *(_index) += 1; \
} while (0)
/// Helper macro for the `_sorted_by` routines below. This takes the left (existing)
/// parameter by reference in order to work with the generic sorting function above.
#define _compare_int(a, b) ((int)*(a) - (int)(b))
#ifdef _MSC_VER
#pragma warning(pop)
#elif defined(__GNUC__) || defined(__clang__)
#pragma GCC diagnostic pop
#endif
#ifdef __cplusplus
}
#endif
#endif // TREE_SITTER_ARRAY_H_

266
gitnexus/vendor/tree-sitter-proto/src/tree_sitter/parser.h vendored Normal file
View file

@ -0,0 +1,266 @@
#ifndef TREE_SITTER_PARSER_H_
#define TREE_SITTER_PARSER_H_
#ifdef __cplusplus
extern "C" {
#endif
#include <stdbool.h>
#include <stdint.h>
#include <stdlib.h>
#define ts_builtin_sym_error ((TSSymbol)-1)
#define ts_builtin_sym_end 0
#define TREE_SITTER_SERIALIZATION_BUFFER_SIZE 1024
#ifndef TREE_SITTER_API_H_
typedef uint16_t TSStateId;
typedef uint16_t TSSymbol;
typedef uint16_t TSFieldId;
typedef struct TSLanguage TSLanguage;
#endif
typedef struct {
TSFieldId field_id;
uint8_t child_index;
bool inherited;
} TSFieldMapEntry;
typedef struct {
uint16_t index;
uint16_t length;
} TSFieldMapSlice;
typedef struct {
bool visible;
bool named;
bool supertype;
} TSSymbolMetadata;
typedef struct TSLexer TSLexer;
struct TSLexer {
int32_t lookahead;
TSSymbol result_symbol;
void (*advance)(TSLexer *, bool);
void (*mark_end)(TSLexer *);
uint32_t (*get_column)(TSLexer *);
bool (*is_at_included_range_start)(const TSLexer *);
bool (*eof)(const TSLexer *);
void (*log)(const TSLexer *, const char *, ...);
};
typedef enum {
TSParseActionTypeShift,
TSParseActionTypeReduce,
TSParseActionTypeAccept,
TSParseActionTypeRecover,
} TSParseActionType;
typedef union {
struct {
uint8_t type;
TSStateId state;
bool extra;
bool repetition;
} shift;
struct {
uint8_t type;
uint8_t child_count;
TSSymbol symbol;
int16_t dynamic_precedence;
uint16_t production_id;
} reduce;
uint8_t type;
} TSParseAction;
typedef struct {
uint16_t lex_state;
uint16_t external_lex_state;
} TSLexMode;
typedef union {
TSParseAction action;
struct {
uint8_t count;
bool reusable;
} entry;
} TSParseActionEntry;
typedef struct {
int32_t start;
int32_t end;
} TSCharacterRange;
struct TSLanguage {
uint32_t version;
uint32_t symbol_count;
uint32_t alias_count;
uint32_t token_count;
uint32_t external_token_count;
uint32_t state_count;
uint32_t large_state_count;
uint32_t production_id_count;
uint32_t field_count;
uint16_t max_alias_sequence_length;
const uint16_t *parse_table;
const uint16_t *small_parse_table;
const uint32_t *small_parse_table_map;
const TSParseActionEntry *parse_actions;
const char * const *symbol_names;
const char * const *field_names;
const TSFieldMapSlice *field_map_slices;
const TSFieldMapEntry *field_map_entries;
const TSSymbolMetadata *symbol_metadata;
const TSSymbol *public_symbol_map;
const uint16_t *alias_map;
const TSSymbol *alias_sequences;
const TSLexMode *lex_modes;
bool (*lex_fn)(TSLexer *, TSStateId);
bool (*keyword_lex_fn)(TSLexer *, TSStateId);
TSSymbol keyword_capture_token;
struct {
const bool *states;
const TSSymbol *symbol_map;
void *(*create)(void);
void (*destroy)(void *);
bool (*scan)(void *, TSLexer *, const bool *symbol_whitelist);
unsigned (*serialize)(void *, char *);
void (*deserialize)(void *, const char *, unsigned);
} external_scanner;
const TSStateId *primary_state_ids;
};
static inline bool set_contains(TSCharacterRange *ranges, uint32_t len, int32_t lookahead) {
uint32_t index = 0;
uint32_t size = len - index;
while (size > 1) {
uint32_t half_size = size / 2;
uint32_t mid_index = index + half_size;
TSCharacterRange *range = &ranges[mid_index];
if (lookahead >= range->start && lookahead <= range->end) {
return true;
} else if (lookahead > range->end) {
index = mid_index;
}
size -= half_size;
}
TSCharacterRange *range = &ranges[index];
return (lookahead >= range->start && lookahead <= range->end);
}
/*
* Lexer Macros
*/
#ifdef _MSC_VER
#define UNUSED __pragma(warning(suppress : 4101))
#else
#define UNUSED __attribute__((unused))
#endif
#define START_LEXER() \
bool result = false; \
bool skip = false; \
UNUSED \
bool eof = false; \
int32_t lookahead; \
goto start; \
next_state: \
lexer->advance(lexer, skip); \
start: \
skip = false; \
lookahead = lexer->lookahead;
#define ADVANCE(state_value) \
{ \
state = state_value; \
goto next_state; \
}
#define ADVANCE_MAP(...) \
{ \
static const uint16_t map[] = { __VA_ARGS__ }; \
for (uint32_t i = 0; i < sizeof(map) / sizeof(map[0]); i += 2) { \
if (map[i] == lookahead) { \
state = map[i + 1]; \
goto next_state; \
} \
} \
}
#define SKIP(state_value) \
{ \
skip = true; \
state = state_value; \
goto next_state; \
}
#define ACCEPT_TOKEN(symbol_value) \
result = true; \
lexer->result_symbol = symbol_value; \
lexer->mark_end(lexer);
#define END_STATE() return result;
/*
* Parse Table Macros
*/
#define SMALL_STATE(id) ((id) - LARGE_STATE_COUNT)
#define STATE(id) id
#define ACTIONS(id) id
#define SHIFT(state_value) \
{{ \
.shift = { \
.type = TSParseActionTypeShift, \
.state = (state_value) \
} \
}}
#define SHIFT_REPEAT(state_value) \
{{ \
.shift = { \
.type = TSParseActionTypeShift, \
.state = (state_value), \
.repetition = true \
} \
}}
#define SHIFT_EXTRA() \
{{ \
.shift = { \
.type = TSParseActionTypeShift, \
.extra = true \
} \
}}
#define REDUCE(symbol_name, children, precedence, prod_id) \
{{ \
.reduce = { \
.type = TSParseActionTypeReduce, \
.symbol = symbol_name, \
.child_count = children, \
.dynamic_precedence = precedence, \
.production_id = prod_id \
}, \
}}
#define RECOVER() \
{{ \
.type = TSParseActionTypeRecover \
}}
#define ACCEPT_INPUT() \
{{ \
.type = TSParseActionTypeAccept \
}}
#ifdef __cplusplus
}
#endif
#endif // TREE_SITTER_PARSER_H_