mirror of
https://github.com/open-webui/open-webui.git
synced 2026-09-19 00:03:21 +00:00
8c7428122b
Documents (especially AI/LLM documentation) legitimately contain special token strings like <|endoftext|> as literal text. The tiktoken encoder raises a ValueError when encountering these during chunk size measurement in merge_docs_to_target_size(), preventing the entire file from being indexed. Pass disallowed_special=() to encoding.encode() to treat all text as normal content. |
||
|---|---|---|
| .. | ||
| data | ||
| open_webui | ||
| .dockerignore | ||
| .gitignore | ||
| dev.sh | ||
| requirements-min.txt | ||
| requirements.txt | ||
| start.sh | ||
| start_windows.bat | ||