mirror of
https://github.com/open-webui/open-webui.git
synced 2026-10-09 03:18:18 +00:00
build: bake in the spaCy model unstructured installs at runtime
`unstructured` classifies narrative text with `sent_tokenize`/`pos_tag`, which
install `en_core_web_sm` into site-packages on first use. site-packages is
root-owned, so under a non-root UID the install fails and takes the upload with
it:
Failed to install en_core_web_sm to /usr/local/lib/python3.11/site-packages:
[Errno 13] Permission denied: '.../site-packages/en_core_web_sm'.
Ensure the site-packages directory is writable, or pre-install the model
with: python -m spacy download en_core_web_sm
unstructured's own error prescribes that command, so run it at build time. It
also removes a runtime fetch from github.com, which no air-gapped deployment can
satisfy however it is run -- root included.
Reproduced on OpenShift (arbitrary UID) with v0.11.0: .xml, .rst and .xlsx
uploads end at `status: failed` with the error above. Running the real loaders
from `retrieval/loaders/main.py` with `_get_nlp` stubbed to raise, the affected
set is xls/xlsx, pptx, epub, msg and rst/xml. doc, odt and ppt reach unstructured
but die earlier on a missing python-docx or libreoffice, so this does not fix
them; docx and html use Docx2txtLoader/BSHTMLLoader and never reach spaCy.
The .xlsx case needs no exotic input: any sheet with a title row above the table
has a cell outside the detected subtable, and that cell goes through
`_create_element` -> `is_possible_narrative_text` -> spaCy.
I know 5b8975b7d deliberately left this out over image size. What I think is
worth revisiting: uncommenting requires building your own image, so it is not
available on a published tag, and the "read-only site-packages" case named there
is any container not running as root. Line 22 says non-root is untested, which
is fair -- this is one of the things that does not survive contact with it, and
it fails as a hard error on upload rather than a degradation.
Cost is 15 MB of model; the layer grows 22.6 MB once the .pyc files the CLI
writes across spacy/thinc/click are counted, against a 7.16 GB image (1.83 GB
compressed). `--no-cache-dir` avoids a further 13 MB of pip cache, which would
otherwise be the first pip cache in the image. main, cuda, cuda126 and ollama
all grow; slim is unaffected. If you would rather not move the default I am glad
to put it behind an ARG instead -- say the word.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
dd8786f414
commit
3e3281faa6
1 changed files with 9 additions and 4 deletions
13
Dockerfile
13
Dockerfile
|
|
@ -152,6 +152,9 @@ RUN set -e; \
|
|||
# Without an explicit dir this lands in /root/nltk_data, which is mode 0700
|
||||
# and so unreadable when the container does not run as root.
|
||||
python -c "import nltk; nltk.download('punkt_tab', download_dir='/usr/local/share/nltk_data')"; \
|
||||
# unstructured installs this into site-packages on first use, which fails
|
||||
# when the container is not root. See the note below this RUN.
|
||||
python -m spacy download en_core_web_sm --no-cache-dir; \
|
||||
else \
|
||||
pip3 install 'torch<=2.9.1' torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu --no-cache-dir; \
|
||||
uv pip install --system -r requirements.txt --no-cache-dir; \
|
||||
|
|
@ -163,16 +166,18 @@ RUN set -e; \
|
|||
# Without an explicit dir this lands in /root/nltk_data, which is mode 0700
|
||||
# and so unreadable when the container does not run as root.
|
||||
python -c "import nltk; nltk.download('punkt_tab', download_dir='/usr/local/share/nltk_data')"; \
|
||||
# unstructured installs this into site-packages on first use, which fails
|
||||
# when the container is not root. See the note below this RUN.
|
||||
python -m spacy download en_core_web_sm --no-cache-dir; \
|
||||
fi; \
|
||||
fi; \
|
||||
mkdir -p /app/backend/data; chown -R $UID:$GID /app/backend/data/; \
|
||||
if [ -d /app/backend/data/cache ]; then chmod -R a+rX /app/backend/data/cache; fi; \
|
||||
rm -rf /var/lib/apt/lists/*;
|
||||
|
||||
# Optional: PPTX parsing through unstructured may need spaCy's English model.
|
||||
# Keep this out of the default image to avoid the extra image bloat; deployments
|
||||
# with read-only site-packages can uncomment it and bake the model in.
|
||||
# RUN python -m spacy download en_core_web_sm
|
||||
# The spaCy model above is installed by default rather than left for downstreams
|
||||
# to uncomment here (5b8975b7d): uncommenting requires building your own image,
|
||||
# so it is not available on a published tag. The slim tag still skips it.
|
||||
|
||||
# Install Ollama if requested
|
||||
RUN if [ "$USE_OLLAMA" = "true" ]; then \
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue