From 3e3281faa61a813bc799be8fd02d01cb1857cb57 Mon Sep 17 00:00:00 2001 From: Sebastian Danielsson Date: Thu, 20 Aug 2026 12:49:42 +0200 Subject: [PATCH] build: bake in the spaCy model unstructured installs at runtime `unstructured` classifies narrative text with `sent_tokenize`/`pos_tag`, which install `en_core_web_sm` into site-packages on first use. site-packages is root-owned, so under a non-root UID the install fails and takes the upload with it: Failed to install en_core_web_sm to /usr/local/lib/python3.11/site-packages: [Errno 13] Permission denied: '.../site-packages/en_core_web_sm'. Ensure the site-packages directory is writable, or pre-install the model with: python -m spacy download en_core_web_sm unstructured's own error prescribes that command, so run it at build time. It also removes a runtime fetch from github.com, which no air-gapped deployment can satisfy however it is run -- root included. Reproduced on OpenShift (arbitrary UID) with v0.11.0: .xml, .rst and .xlsx uploads end at `status: failed` with the error above. Running the real loaders from `retrieval/loaders/main.py` with `_get_nlp` stubbed to raise, the affected set is xls/xlsx, pptx, epub, msg and rst/xml. doc, odt and ppt reach unstructured but die earlier on a missing python-docx or libreoffice, so this does not fix them; docx and html use Docx2txtLoader/BSHTMLLoader and never reach spaCy. The .xlsx case needs no exotic input: any sheet with a title row above the table has a cell outside the detected subtable, and that cell goes through `_create_element` -> `is_possible_narrative_text` -> spaCy. I know 5b8975b7d deliberately left this out over image size. What I think is worth revisiting: uncommenting requires building your own image, so it is not available on a published tag, and the "read-only site-packages" case named there is any container not running as root. Line 22 says non-root is untested, which is fair -- this is one of the things that does not survive contact with it, and it fails as a hard error on upload rather than a degradation. Cost is 15 MB of model; the layer grows 22.6 MB once the .pyc files the CLI writes across spacy/thinc/click are counted, against a 7.16 GB image (1.83 GB compressed). `--no-cache-dir` avoids a further 13 MB of pip cache, which would otherwise be the first pip cache in the image. main, cuda, cuda126 and ollama all grow; slim is unaffected. If you would rather not move the default I am glad to put it behind an ARG instead -- say the word. Co-Authored-By: Claude Opus 5 (1M context) --- Dockerfile | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/Dockerfile b/Dockerfile index 8336a3e6e6..63a263712a 100644 --- a/Dockerfile +++ b/Dockerfile @@ -152,6 +152,9 @@ RUN set -e; \ # Without an explicit dir this lands in /root/nltk_data, which is mode 0700 # and so unreadable when the container does not run as root. python -c "import nltk; nltk.download('punkt_tab', download_dir='/usr/local/share/nltk_data')"; \ + # unstructured installs this into site-packages on first use, which fails + # when the container is not root. See the note below this RUN. + python -m spacy download en_core_web_sm --no-cache-dir; \ else \ pip3 install 'torch<=2.9.1' torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu --no-cache-dir; \ uv pip install --system -r requirements.txt --no-cache-dir; \ @@ -163,16 +166,18 @@ RUN set -e; \ # Without an explicit dir this lands in /root/nltk_data, which is mode 0700 # and so unreadable when the container does not run as root. python -c "import nltk; nltk.download('punkt_tab', download_dir='/usr/local/share/nltk_data')"; \ + # unstructured installs this into site-packages on first use, which fails + # when the container is not root. See the note below this RUN. + python -m spacy download en_core_web_sm --no-cache-dir; \ fi; \ fi; \ mkdir -p /app/backend/data; chown -R $UID:$GID /app/backend/data/; \ if [ -d /app/backend/data/cache ]; then chmod -R a+rX /app/backend/data/cache; fi; \ rm -rf /var/lib/apt/lists/*; -# Optional: PPTX parsing through unstructured may need spaCy's English model. -# Keep this out of the default image to avoid the extra image bloat; deployments -# with read-only site-packages can uncomment it and bake the model in. -# RUN python -m spacy download en_core_web_sm +# The spaCy model above is installed by default rather than left for downstreams +# to uncomment here (5b8975b7d): uncommenting requires building your own image, +# so it is not available on a published tag. The slim tag still skips it. # Install Ollama if requested RUN if [ "$USE_OLLAMA" = "true" ]; then \