populateRQualifiedCalls applies the fallback a second time, to decide which
files define a local package's symbols. Nothing exercised that site, so
reverting it left every unit test green. The new test fails without it.
rQualifierLocality treated a qualifier as a local package when any parsed
file sat under a directory of that name followed by R/, even when discovery
had attributed the directory to a package with a different Package: name.
The directory name is now evidence only for files no discovered package owns.
refineRExportStatus matched a file against the package directory, so
functions in tests/, vignettes/ and inst/ were marked not exported although
R never loads them into the namespace, and a nested package without a
NAMESPACE inherited the enclosing package's. Ownership now comes from the
nearest discovered package whose R/ directory holds the file
(rOwningPackageDir), over every directory discovery read a DESCRIPTION in.
library() of a root-level package resolved to nothing because its directory
'' is falsy in resolveRImportInternal; it now resolves to the package's R/
files like a nested package.
Installed libraries and check output hold one DESCRIPTION per third-party
package. Discovery treated them as repository content: a renv project
library below the depth limit flagged discovery as truncated, which made
every pkg::fn() qualifier unknown, and a shallow copy became a local
package with no parsed files.
The main walk and the hidden-package scan now share one predicate that
skips a directory whose name is exactly renv, packrat or revdep, or ends in
.Rcheck after at least one character. Look-alike names are still searched.
Among same-depth directories declaring one package name, the winner was
decided by breadth-first walk order, which compares path segment by
segment (a/x before a-b/x). Compare the whole relative path by code unit
instead (a-b/x before a/x), so the winner no longer depends on how the
walk happens to meet the directories, and update the comments to say so.
Package discovery used the filesystem's directory listing order, so the
200-directory cap visited different directories, and a package name
declared in several directories resolved to a different directory, on
different filesystems. Read every directory in code-unit name order and
keep the first package seen for a name: the shallowest directory, then the
path that sorts first. A root-level package is no longer replaced by a
nested copy of the same name.
The linear-time exportPattern matcher refused anything over 1,024 states or
4,096 characters, so a legitimate `^(f1|...|f80)$` or `^[[:alpha:]]{1,2000}$`
compiled to nothing and every name in the package read as unexported, with no
diagnostic.
Raise the caps to 16,384 states and characters, and bound the hostile case by a
per-matcher work budget instead: a matcher counts the states it visits and, once
40 million are spent (about 0.4 s), answers false for names it has not decided
and reports itself exhausted. Alternations of literals are built as a trie so
alternatives sharing a prefix share states, the start closure is looked up by
code unit, closures are iterative, and verdicts are memoised per distinct name.
Every non-empty exportPattern that is dropped, and every matcher that runs out
of budget, now produces one warning per pattern per package naming the package
directory, the first 60 characters of the pattern and the reason. The dropped
patterns are also kept on RNamespaceInfo.droppedExportPatterns.
exportPattern() text comes from the analysed repository and was compiled
to a backtracking RegExp, so a pattern such as ^(a+)+$ took 1.6 s on a
27-character name and minutes on a slightly longer one, hanging analyze.
R itself evaluates these with the non-backtracking TRE.
Compile the pattern to a Thompson NFA and simulate it instead, so a match
costs O(name length x NFA size) for any pattern. Constructs with no
linear-time algorithm (back-references, look-around) and patterns that
expand beyond a fixed state cap are dropped like an uncompilable pattern.
Verdicts for ordinary patterns are unchanged, checked against the
previous RegExp on a generated corpus.
Pin the verdicts of the current exportPattern matcher for the patterns R
packages commonly use (POSIX classes, anchors, alternation, bracket
expressions, repetition, escapes) and for patterns it rejects, then
compare a seeded corpus of generated patterns against RegExp. This fixes
the behaviour a linear-time matcher must preserve.
Rewrite comment and describe-title text that pointed at labels defined
nowhere in the repository and history-narrative wording that referred to
earlier work, stating each rule directly. No behaviour change; test counts
are unchanged.
Rewrite references to planning notes (section numbers, item labels, review
rounds, scenario/unit numbering) as self-contained statements of the rule or
trade-off. Comment, fixture comment and test-description text only; no
behaviour change.
Replace issue-number references in comments, test names and descriptions with
self-contained wording, drop merge-coordination notes from the SCHEMA_BUMP
history, and rename a fixture file to qualified_use.R. Comment,
description and file-name changes only.
R is registered in SCOPE_RESOLVERS but had no arm in the import-target
benchmark, so `bench/import-target/measure.mjs --check` failed its
inventory arm twice: no LANG_REGISTRY row, and a five-parameter
resolveImportTarget with no context arm.
Add the R row and its five dispatcher branches (uniqueDir, collideDir,
uniqueTarget, collideTarget, resolveOne), driving the registered hook with
library() (wildcard), source() (side-effect) and importFrom() (named, reading
the parsed workspace). R joins CONTEXT_LANGS with a probe whose two answers
differ, and HEAP_BUDGETED. baselines.json carries the R fingerprints, shape,
heap reading, context pair and budgets, with an `_r_arm` note recording the
two O(files)-per-import legs the arm measures.
loadRPackageConfig returned null when discovery found no package, and
rQualifierLocality maps a null config to unknown, so in a repo with no
DESCRIPTION anywhere dplyr::filter() still bound to a local filter through
the name guess. Return an empty config instead: its truncated flag now
distinguishes a complete scan (external) from one that may have missed a
package (unknown, name guess kept).
Discovery set RPackageConfig.truncated whenever it skipped any directory
(depth-3 limit) or hit the 200-directory cap, so nearly every real repo
(tests/testthat/fixtures, inst/extdata, ...) counted as truncated. Qualified
calls to outside packages were then classified unknown and kept, leaving the
dplyr::filter() bound to a local filter() case unfixed.
Skipped directories and the cap leftover queue are now collected and scanned
by hidesUndiscoveredPackage (iterative DFS, global 5000-directory budget,
skipping node_modules, .git, .Rproj.user). truncated is set only when that
scan finds a DESCRIPTION whose Package: name is not already discovered,
exceeds the budget, or hits an unreadable path; an unreadable directory in
the main walk still truncates.
The scope query now captures the qualifier of `pkg::fn()` / `pkg:::fn()`; use it.
- qualified-call.ts (new): parseRQualifier, a tri-state rQualifierLocality
(local / external / unknown) and populateRQualifiedCalls, wired after the
NAMESPACE import synthesis in R's populateWorkspaceReferences.
* external package (discovery complete): the site is dropped, so no tier
can guess an edge for it;
* local package with exactly one top-level definition of the name: bound
precisely (0.85) through resolveQualifiedFreeCall, keyed by file + start
position;
* local package with two or more definitions: the site is dropped;
* local package with none (re-export) or unknown locality: unchanged.
- package-config.ts: RPackageConfig.truncated is set when the directory cap,
the depth limit or an unreadable directory left discovery incomplete, so a
package is only called external when the scan was complete.
- isRGlobalNameFallbackPlausible: a candidate inside the named local package
is allowed, one elsewhere is vetoed when the package defines the name.
- Flip the tests that pinned the pre-fix name-only results and drop the
temporary pins and it.fails markers of the step-1 tests.
No shared-core changes.
Capture the whole namespace_operator of pkg::fn() and pkg:::fn() calls as
@reference.qualified-name so the qualifier reaches site.rawQualifiedName and
survives the parse cache. On its own this changes no graph output: no R
consumer reads the qualifier yet.
SCHEMA_BUMP is incremented because warm ParsedFiles lack the new capture.
Add the r-qualified-calls and r-qualified-calls-truncated fixtures and failing-first
tests for qualifier-aware calls: the qualifier of pkg::fn() / pkg:::fn() decides the edge
(precise binding into a local package, no edge for duplicate definitions or certainly
external packages, sites kept when locality is unknown).
New expectations that encode not-yet-true behaviour are it.fails; TEMPORARY-PIN tests
assert today's edges for the same callers so an it.fails cannot pass for the wrong
reason. Existing tests are untouched; the seven that flip are listed in the header of
r-qualified-calls.test.ts.
The node-property ownerId spread in the parse worker was added with R
support and is never read for non-R languages; upstream's worker never
wrote it onto definition nodes. R never had it either (its owner is
attached post-parse). Symbol ownerId and HAS_METHOD/HAS_PROPERTY edges
are unchanged.
Move R's deferred-owner hint out of the shared parse worker into R's
definitionPropertiesExtractor (rDefinitionProperties), and remove the
worker-side hint block, the node/symbol ownerNameHint spreads,
ParsedSymbol.ownerNameHint and the R extractor imports. Delete the dead
hasRoxygenExport branch from rExportChecker (both branches returned true).
Add an optional resolveMemberOwnerNode hook consulted by the worker's
enclosing-owner helper after the container walk and resolveFileTypeOwner.
R implements it via rResolveMemberOwnerNode, replacing the two
language === R branches in parse-worker.
Move r-post-parse.ts into languages/r/post-parse.ts, expose rPostParse and
wire it on the R provider only. The parse phase drops its R-specific block
and relies on the generic runPostParseHooks call, so a repo with no R files
no longer reads DESCRIPTION/NAMESPACE. Flip the gating test from it.fails to it.
Add an optional LanguageProvider.postParse(ctx: PostParseContext) hook, the
PostParseContext type and runPostParseHooks, a pure runner beside
prepareRouteConstantsByProvider that takes a provider-lookup callback (no
registry import). It runs hooks sequentially in language-id order, awaits each
and does not isolate throws, because the hooks mutate the shared graph/model.
runChunkedParseAndResolve computes the languages present among the parseable
scanned files and calls the runner immediately after the existing R block
(same statement position: after every chunk is merged and the export-map
enrichment, before scope resolution). No provider defines the hook yet, so
this is a no-op; the R block is removed in a later step.
Pin the behaviour that the following commits move behind LanguageProvider hooks,
before any production change:
- r-post-parse-owners (unit): attachDeferredROwners on a synthetic graph and
semantic model, including the already-owned early return (no re-attach, no
double registration).
- r.test.ts: r-packages HAS_METHOD 11 / HAS_PROPERTY 12, the measured edge
triple set, 23 owned R nodes, no leftover ownerNameHint.
- r-worker-owner-absence: the compiled worker never emits ownerId for R
(six r-* fixtures; 39 members carry ownerNameHint).
- r-post-parse-gating: a non-R repo with DESCRIPTION/NAMESPACE decoys must
not read a manifest. Written as it.fails (the shared parse phase still
loads the R package config for every repo) so the commit stays green; the
R control passes today.
The shared simple-name indexes key every def by the text after the last '.'
of its qualifiedName, so an S3 method `print.foo` is indexed under `foo`
next to a bare `foo`. Two measured defects followed:
- `library(prov)` bound `foo` to `print.foo` (a wrong 0.85 import-resolved
edge) when the S3 method was declared before the bare def;
- a same-package cross-file bare call `foo(v)` saw two candidates under
`foo` and got no CALLS edge at all.
R-only fix, no shared-core edit, no SCHEMA_BUMP (both hooks run at resolve
time over cached ParsedFiles):
- rScopeResolver.isCallableVisibleFromCaller refuses dotted candidates: a
bare R call can never invoke `print.foo`.
- expandRWildcardNames drops a bare name whose file also defines a dotted
top-level def with that tail (order-blind, like the importFrom guard).
Characterisation tests deliberately updated to the new behaviour (r.test.ts,
R NAMESPACE importFrom fixture):
- print.tidy before tidy / tally before print.tally: was no edge, now a
0.5 global-name-fallback edge to the bare def;
- bare call to a name that only an R6 method carries: was a 0.5 guess to
the method, now no edge (a bare call cannot reach an R6 member).
Residual: dotted names bind only through qualified-name lookup (0.35),
never import-resolved; a bare name sharing a tail with an S3 method in the
same file resolves at 0.5 (fallback), not 0.85; backticked names unchanged.
Since the shared scanner landed, a garbled or unbalanced
directive stopped parsing the rest of the NAMESPACE file, silently dropping
every later export/importFrom directive (the old line-based export parser
resumed at the next line).
The scanner now discards the malformed directive and resumes at the next line
that begins, at column 0, with `identifier(`. To keep that safe:
- quoted strings no longer span lines, so a stray quote cannot pair with a
later one and swallow the directives in between;
- a column-0 `identifier(` line ends any argument list still open, which also
bounds each failed scan to the next directive rather than the end of file;
- the resume point is always strictly after the failing directive's line, so
scanning always makes forward progress.
An indented directive following a malformed one is dropped with it. Both
parseRNamespaceExports and parseRNamespaceImportFrom benefit. No existing test
pinned stop-on-error behaviour; all pass unchanged.
refineRExportStatus never refined a package whose DESCRIPTION/NAMESPACE sit
at the repository root (package dir ''). Two defects combined:
- the containment test used startsWith('' + '/'), i.e. '/', which no
repo-relative path matches;
- the result of pkgDirs.find() was guarded with `!pkgDir`, which treats the
valid root key '' as "no package".
Treat '' as containing every path (it sorts last, so nested packages still
win) and compare the find() result against undefined. Adds a synthetic
root-package fixture, an integration test and unit tests covering export(),
exportPattern(), S3method(), unexported helpers, no-NAMESPACE, and the
unchanged nested-package behaviour.
loadRPackageConfig read export(), exportClasses(), exportMethods(),
S3method() and exportPattern() line by line, so any wrapped directive was
silently skipped. Read them from the whole NAMESPACE text with the tokeniser
already used for importFrom(): the tokeniser is split into a shared
scanRNamespaceDirectives, and parseRNamespaceExports is a second consumer.
Wrapped, quoted, backticked, commented and CRLF directives now give the same
result as their single-line form; S3method() accepts its optional third
argument and exportPattern() accepts several patterns.
exportPattern() strings are now R-unescaped before compilation, so the R
string "\\." becomes the regex \. rather than \\. (unescapeRString handles
\\, \", \', \n, \t, \r, \a, \b, \f, \v and \x/\u/\U code points).
loadRPackageConfig changes only at the call site; no parse-cache output
changes.
exportPattern() arguments are POSIX (TRE) regexes, but loadRPackageConfig
compiled them with new RegExp(), so classes such as [[:alpha:]] were read as
a JS character set and never matched. The RStudio default
exportPattern("^[[:alpha:]]+") therefore exported nothing, and named-import
export checks judged every name non-exported.
Add compileRExportPattern (languages/r/export-pattern.ts): rewrites
[:alpha:], [:alnum:], [:digit:], [:upper:], [:lower:], [:punct:],
[:space:] (plus blank, cntrl, print, graph, xdigit) inside bracket
expressions to ASCII JS equivalents, handles negation and mixed ranges, and
returns null for unknown classes or unterminated brackets. Patterns without
"[:" compile verbatim, so existing behaviour is unchanged. Compilation still
happens once per pattern in loadRPackageConfig.
Deliberate test change: the "POSIX exportPattern characterisation" in
r-namespace-imports.test.ts pinned the wrong behaviour (a JS-compiled POSIX
pattern refuses every name). It now asserts the corrected behaviour, plus a
non-matching control.
Add sample-code/simple.r and an R block in tree-sitter-languages covering
function, S4, and R6 definitions, library()/source() imports, and free,
member, namespaced, and native-pipe calls. List simple.r in the parsing
fixture-existence check and add R extension detection to ingestion-utils.
Add R to the language lists and feature matrices in README.md and
gitnexus/README.md, with an honest capability note (experimental,
name-based fallback for same-package bare calls, no export() handling).
Correct the ARCHITECTURE.md language and provider counts from 18 to 19.
The nested node_modules/tree-sitter-cpp/node_modules/node-addon-api 8.7.0
entry is unrelated to the R grammar. The hoisted 8.9.2 already satisfies
tree-sitter-cpp's ^8.2.1 range; npm regenerates the lockfile unchanged.
Refuse a unique-name global fallback edge when the caller package NAMESPACE
imports the name (last importFrom entry wins) from a package other than the
one the guess landed in and other than the caller's own package. Callers and
candidates outside a package R/ directory, local providers that re-export the
name, self-imports and undecidable configs keep the baseline edge. The
pkg::name() and import() limitations are pinned by characterisation tests.
Synthesise a `named` import for every NAMESPACE importFrom(pkg, name) whose
pkg is a local package, into each R/ file of the importing package
(populateRNamespaceImports, wired via populateWorkspaceReferences with
postExtractSourceTextPolicy 'uncached-files'), and resolve it in
resolveRImportTarget's named branch.
Rules: last entry wins per name over all entries; a self-import, an
external last entry, and a name the importing package defines at top level
(own namespace masks) synthesise nothing. The named branch builds no
workspace file index, returns null for external or unknown packages, dotted
names (finalize keys defs by the text after the last dot), providers whose
NAMESPACE does not export the name (backticks stripped), names with no
top-level definition, and a defining file that also holds a dotted def
whose tail equals the name. Top-level definitions are read from the Module
scope's hoisted bindings, not ownedDefs/localDefs.
Known limitations, pinned by tests: pkg::name() qualifiers are discarded
at this point, a nested same-named function can still win the binding, and
POSIX-class exportPattern() never matches in a JS RegExp.
Set namedImportsBindTopLevelOnly on rScopeResolver so a named import can
only bind a module-level definition of the target file, never an R6/R5
member with the same simple name. Groundwork for NAMESPACE importFrom()
binding; r.test.ts (127) is unchanged by the flag alone.
Every R call site's CALLS source is now the callable that lexically contains
it (named function, nested named function, R6/R5 method), or the File node when
no named callable encloses it; a call in a class body outside any method (an
R6 field default, a setClass validity body) is sourced from the Class node via
the unchanged resolveCallerGraphId class-anchor fallback.
Cause: R's @scope.function anchored only on the function_definition, while
@declaration.function / @declaration.method anchor on the enclosing assignment /
named argument. The def was therefore owned by the module / outer-function /
Class scope, and resolveCallerGraphId credited every call in a body to the FIRST
callable owned by that scope (false inner -> inner self-loops; module-level
calls credited to a function instead of File).
Fix (languages/r/query.ts only; no shared pipeline change, no @declaration.*
change so def ids and resolveDefGraphId reconciliation are untouched):
- (binary_operator lhs: (identifier) rhs: (function_definition)) @scope.function,
directly beside the @declaration.function pattern it must track.
- (argument name: (identifier) value: (function_definition)) @scope.function,
directly beside the @declaration.method patterns; generic, so it cannot drift
from the three R6/R5 nesting patterns, and inert for non-method arguments.
- Both carry a "must stay in lockstep" comment; the blanket function_definition
scope stays (anonymous functions); the query header rationale is rewritten.
Cache: SCHEMA_BUMP is incremented (ledger comment added, pin test updated). Warm
ParsedFiles and durable stores hold the pre-fix scope tree.
Tests: removes every it.fails wrapper from the tests added in the previous commit
(18 unit, 36 integration) and deletes the pins of the old, defective behaviour
(5 unit, 4 integration);
they now pass as ordinary tests. No other assertion was changed.
Verification (built dist): scope-function-anchors 42/42, r.test.ts 97/97
(39 existing + 58 new), incremental-parse-cache, call-arguments-r,
external-import-conformance, import-target-index-reuse.contract, the three
grammar-literal gate files, tsc --noEmit clean.
Adds the tests for the R caller-attribution fix BEFORE the fix, so the fix commit only flips them to normal tests.
Defect: R's @scope.function anchors on the function_definition while
@declaration.function / @declaration.method anchor on the enclosing assignment
or method argument. The def is therefore owned by the Module / outer-function /
Class scope, and resolveCallerGraphId credits every call in a body to the FIRST
callable that scope owns (false inner -> inner self-loops, R6 calls credited to
the class's first method, module-level calls credited to a function not File).
Contents:
- test/unit/scope-resolution/r/scope-function-anchors.test.ts: scope-tree proofs
via extractParsedFile + populateClassOwnedMembers (named forms, nesting,
anonymous lambdas, R6/R5 methods, nested helper in a method, class-body field
default, single-function file, member-assigned functions, def node id pins).
- test/fixtures/lang-resolution/r-call-attribution/: synthetic fixture, a decoy
first function per file, unique leaf callees.
- test/integration/resolvers/r.test.ts: new 'R caller attribution' describe with
exact per-source edge sets and named-type invariance counts.
Assertions that fail at HEAD because of the misattribution are wrapped in
it.fails (one assertion per it, value-comparing, no non-null derefs) and are
unwrapped by the fix commit; assertions that already hold stay plain; each
wrapped case has a plain twin (the target edge exists from any source). The
step-1-only HEAD pins assert the defect itself and are deleted by the fix
commit. Verified: every wrapped test fails without the wrapper at HEAD with an
assertion mismatch (no throws) and passes when the query is patched with the
two scope patterns; only the HEAD pins flip.
R was missing from BASENAME_LANGS/DIR_LANG (literal-collectors.ts) and
GRAMMAR_PACKAGES (grammar-introspection.ts), so literals in field-
extractors/r.ts, method-extractors/r.ts, and type-extractors/r.ts were
tagged with every OTHER language's grammar instead of R's, and the new
languages/r/ resolution-layer files (scope-resolver, query.ts, etc.)
were skipped entirely by the gate's Mode 3/4 scans. `namespace_operator`
and `extract_operator` are valid R node types, so this was a mistagging
bug in the test helpers, not a dead grammar literal.
