fix: uploaded files lose Chinese punctuation and quotes (#31655)

Text extracted from any uploaded file had full-width punctuation like :(),!? turned into ASCII :(),!? and curly quotes like “ ” turned into straight quotes, so both the file preview and the model saw altered text. Extracted text now keeps these characters as written, while garbled text from wrong encodings (like café becoming café) is still repaired. Files uploaded before this change keep the altered text until they are uploaded again.

Fixes #17087
This commit is contained in:
Classic298 2026-10-01 05:22:08 +02:00 • committed by GitHub
parent 3ef0d15433
commit c3fbf36384
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

View file

@ -342,7 +342,12 @@ class Loader:
docs = loader.load()
# ftfy's auto mode unescapes entities on every line before the first literal '<', rewriting the document.
return [
Document(page_content=ftfy.fix_text(doc.page_content, unescape_html=False), metadata=doc.metadata)
Document(
page_content=ftfy.fix_text(
doc.page_content, unescape_html=False, fix_character_width=False, uncurl_quotes=False
),
metadata=doc.metadata,
)
for doc in docs
]