mirror of
https://github.com/open-webui/open-webui.git
synced 2026-09-16 23:43:03 +00:00
1bfa59acbd
Uploading a file whose text contains literal HTML entities stored a rewritten copy of it: ` ` became a non-breaking space, `>` became `>`, and `&nbsp;` was decoded twice down to a bare non-breaking space. That stored text is what gets indexed and what the model reads, so notes, specs and source files reached the model differing from the file that was uploaded. Every loaded document goes through `ftfy.fix_text`, which is there to repair mojibake left by the encoding-detection fallback. Its default configuration also decodes HTML entities, per line and sticky forward: entities are decoded on every line up to the first line holding a literal `<`, then left alone for the rest of the document. The same escape therefore survives or vanishes depending on where it sits in the file. This disables that one behaviour and leaves every other ftfy repair in place. Text from a third-party extraction engine that returns escaped output now keeps those escapes. Guessing whether an escape is markup or content is the bug being fixed. Fixes #29732 |
||
|---|---|---|
| .. | ||
| datalab_marker.py | ||
| external_document.py | ||
| external_web.py | ||
| main.py | ||
| microsoft_web_iq.py | ||
| mineru.py | ||
| mistral.py | ||
| paddleocr_vl.py | ||
| tavily.py | ||
| youtube.py | ||