scriptoriumadmin/ReMe
1
0
Fork 0
mirror of https://github.com/agentscope-ai/ReMe.git synced 2026-08-28 05:25:04 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 11

Modify budget calculation for text limit safety margin

Browse source 
Adjust budget calculation to use 92% margin for token estimation.
This commit is contained in:
jinliyl 2026-07-14 11:05:08 +08:00 • committed by GitHub
parent 07d4527a0d
commit b5e0ec2d8d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
reme/components/embedding_store/base_embedding_store.py
View file

@ -46,7 +46,7 @@ class BaseEmbeddingStore(BaseComponent):
return text[:limit]
# Reserve a 5% safety margin for token estimation.
budget = limit * 2 * 95 // 100
budget = limit * 2 * 92 // 100
used = 0
for index, char in enumerate(text):
used += 3 if unicodedata.east_asian_width(char) in {"W", "F"} else 2