mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-10-02 02:11:29 +00:00
Index Sage and Pyodide kernels, keep later cells when one cell is broken, and link %run of a local module without executing a kernel.
3.4 KiB
3.4 KiB
Jupyter notebook (.ipynb) indexing
Status: implemented (Python code cells)
GitNexus indexes Jupyter notebooks by extracting Python code cells and parsing them with the existing Python language provider. Notebooks are not executed.
Goal
After analyze, functions, classes, and imports defined in Python code cells are queryable like ordinary .py files.
Compatibility
.ipynbis detected as Python (gitnexus-sharedEXTENSION_MAPandpythonProvider.extensions).- Extraction lives in
gitnexus/src/core/ingestion/ipynb-extractor.ts. Shared ingestion modules do not name nbformat AST types. - Notebooks are not Python import targets. A notebook may import
.pymodules;import some_notebookdoes not resolve to an.ipynb. - Files over the walker size cap (default 512KB,
GITNEXUS_MAX_FILE_SIZE) are skipped like any other oversized file. Output-heavy notebooks may need a higher cap. Outputs are not stripped in this slice. - Group-layer FastAPI/Flask/Django scanners that require a
.pysuffix still ignore notebooks.
Kernel and magics
- Skip the file when
kernelspec.languageorlanguage_info.nameis present and is not a Python-family name (python,python2,python3,ipython,python 3,ipython3).pythonandpython3together still index. - If
kernelspec.languageis missing,kernelspec.nameis used only when it is an obvious language id (python3,ir,julia-1.8). Conda env names are ignored. - If
language_info.nameis missing,language_info.file_extension(.pyvs.r/.jl) is used the same way. - If those fields are absent, code cells are treated as Python unless a cell's own
language,metadata.language, ormetadata.vscode.languageIdsays otherwise. - A cell whose first non-empty line is a foreign cell magic (
%%bash,%%html,%%sql,%%R) is skipped. Python-body cell magics (%%time,%%timeit,%%capture,%%prun,%%debug,%%px,%%python) stay, with the magic line commented. %run,%load, and%loadpyof a local.pypath becomeimport moduleon that same line so the notebook links to the file. URLs,..paths, and.ipynbtargets stay comments. The notebook is not executed.- Sage, SageMath, MicroPython, Pyodide, PyPy, and PySpark kernels are indexed as Python. SQL, R, and Julia cells are not.
- A cell with an unclosed string or bracket is commented out so it cannot hide later cells. Those cells are concatenated in order; one syntax error no longer drops the rest of the notebook.
- Line magics (
%), shell (!), and IPython help (train?,?train) are commented in place so JSON line mapping stays affine. - nbformat v4
cells/sourceis the normal path. nbformat v3worksheets[].cellsand code-cellinputare accepted. A leading UTF-8 BOM is accepted. Markdown and raw cells are not code.
Line numbers
Graph startLine / endLine are 0-based coordinates in the on-disk .ipynb JSON. FTS/MCP symbol snippets reconstruct cell Python; they do not slice raw JSON.
Concatenating cells in document order is notebook semantics. An earlier cell with a syntax error may cause later definitions to be missed; that is a documented limitation, not a per-cell fallback parse.
Tests
gitnexus/test/unit/ipynb-extractor.test.tsgitnexus/test/unit/ingestion-utils.test.ts(.ipynbdetection)gitnexus/test/integration/resolvers/ipynb-python-pipeline.test.ts
No Jupyter, nbconvert, or nbformat runtime dependency.