build: bake in the spaCy model unstructured installs at runtime

`unstructured` classifies narrative text with `sent_tokenize`/`pos_tag`, which
install `en_core_web_sm` into site-packages on first use. site-packages is
root-owned, so under a non-root UID the install fails and takes the upload with
it:

    Failed to install en_core_web_sm to /usr/local/lib/python3.11/site-packages:
    [Errno 13] Permission denied: '.../site-packages/en_core_web_sm'.
    Ensure the site-packages directory is writable, or pre-install the model
    with: python -m spacy download en_core_web_sm

unstructured's own error prescribes that command, so run it at build time. It
also removes a runtime fetch from github.com, which no air-gapped deployment can
satisfy however it is run -- root included.

Reproduced on OpenShift (arbitrary UID) with v0.11.0: .xml, .rst and .xlsx
uploads end at `status: failed` with the error above. Running the real loaders
from `retrieval/loaders/main.py` with `_get_nlp` stubbed to raise, the affected
set is xls/xlsx, pptx, epub, msg and rst/xml. doc, odt and ppt reach unstructured
but die earlier on a missing python-docx or libreoffice, so this does not fix
them; docx and html use Docx2txtLoader/BSHTMLLoader and never reach spaCy.

The .xlsx case needs no exotic input: any sheet with a title row above the table
has a cell outside the detected subtable, and that cell goes through
`_create_element` -> `is_possible_narrative_text` -> spaCy.

I know 5b8975b7d deliberately left this out over image size. What I think is
worth revisiting: uncommenting requires building your own image, so it is not
available on a published tag, and the "read-only site-packages" case named there
is any container not running as root. Line 22 says non-root is untested, which
is fair -- this is one of the things that does not survive contact with it, and
it fails as a hard error on upload rather than a degradation.

Cost is 15 MB of model; the layer grows 22.6 MB once the .pyc files the CLI
writes across spacy/thinc/click are counted, against a 7.16 GB image (1.83 GB
compressed). `--no-cache-dir` avoids a further 13 MB of pip cache, which would
otherwise be the first pip cache in the image. main, cuda, cuda126 and ollama
all grow; slim is unaffected. If you would rather not move the default I am glad
to put it behind an ARG instead -- say the word.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Sebastian Danielsson 2026-08-20 12:49:42 +02:00
parent dd8786f414
commit 3e3281faa6
No known key found for this signature in database

View file

@ -152,6 +152,9 @@ RUN set -e; \
# Without an explicit dir this lands in /root/nltk_data, which is mode 0700
# and so unreadable when the container does not run as root.
python -c "import nltk; nltk.download('punkt_tab', download_dir='/usr/local/share/nltk_data')"; \
# unstructured installs this into site-packages on first use, which fails
# when the container is not root. See the note below this RUN.
python -m spacy download en_core_web_sm --no-cache-dir; \
else \
pip3 install 'torch<=2.9.1' torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu --no-cache-dir; \
uv pip install --system -r requirements.txt --no-cache-dir; \
@ -163,16 +166,18 @@ RUN set -e; \
# Without an explicit dir this lands in /root/nltk_data, which is mode 0700
# and so unreadable when the container does not run as root.
python -c "import nltk; nltk.download('punkt_tab', download_dir='/usr/local/share/nltk_data')"; \
# unstructured installs this into site-packages on first use, which fails
# when the container is not root. See the note below this RUN.
python -m spacy download en_core_web_sm --no-cache-dir; \
fi; \
fi; \
mkdir -p /app/backend/data; chown -R $UID:$GID /app/backend/data/; \
if [ -d /app/backend/data/cache ]; then chmod -R a+rX /app/backend/data/cache; fi; \
rm -rf /var/lib/apt/lists/*;
# Optional: PPTX parsing through unstructured may need spaCy's English model.
# Keep this out of the default image to avoid the extra image bloat; deployments
# with read-only site-packages can uncomment it and bake the model in.
# RUN python -m spacy download en_core_web_sm
# The spaCy model above is installed by default rather than left for downstreams
# to uncomment here (5b8975b7d): uncommenting requires building your own image,
# so it is not available on a published tag. The slim tag still skips it.
# Install Ollama if requested
RUN if [ "$USE_OLLAMA" = "true" ]; then \