scriptoriumadmin/zotero
1
0
Fork 0
mirror of https://github.com/zotero/zotero.git synced 2026-10-07 02:58:09 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2
Mirror of https://github.com/zotero/zotero.git
16734 commits 12 branches 404 tags 325 MiB
Find a file
Bogdan Abaev fbb0495f05 approximate chunk token sizes instead of tokenizer 
Using tokenizer to determine exact token size of a chunk
puts a lot of work on the main thread during indexing
only to find if a chunk is over the min limit and is
under the max. There is not much value in precision
so we estimate the token value per chunk with a language
table (e.g. english ~4 chars/token, CJK ~1 char/token, etc.)

If needed precision ends up being necessary, we can
run a tokenizer on a small subset of the text in document
and derive chars/token ration from that instead of
actually tokenizing the entire document.
2026-09-02 12:10:01 -07:00
.github/workflows CI: Don't run duplicate jobs for same-repo pull requests 2026-07-08 10:50:32 -04:00
app Restore dxcompiler.dll on Windows for WebGPU and leave WebGPU pref on 2026-08-05 14:10:04 -04:00
chrome approximate chunk token sizes instead of tokenizer 2026-09-02 12:10:01 -07:00
defaults/preferences temp: pref to select hybrid/semantic/lexical mode 2026-08-18 18:01:04 -07:00
document-worker@6d0c0ce45d Update document-worker submodule 2026-07-15 12:52:00 -04:00
js-build Add SDT support 2026-06-12 13:47:08 +03:00
note-editor@acec74d09b Improve undo/redo in the note editor (#3398) 2026-07-31 14:22:53 +02:00
reader@c6edbfff75 Update reader submodule 2026-07-20 10:50:56 -04:00
resource Generate embeddings with Firefox's inference runtime 2026-08-05 15:41:41 -04:00
scripts Add script for pulling Fluent translations 2026-01-25 22:15:29 -05:00
scss do not try to render previews dynamically 2026-08-28 13:17:35 -07:00
styles@dff7452b24 Update translators, styles, CSL locales, and renamed-styles.json 2026-04-02 15:51:43 -04:00
test approximate chunk token sizes instead of tokenizer 2026-09-02 12:10:01 -07:00
translators@60f2d542ff Update translators submodule 2026-07-15 12:52:54 -04:00
types/gecko fx140: Update type definitions 2025-07-30 22:30:37 -04:00
.babelrc Use production builds of react libraries (#4482) 2024-08-02 03:43:02 -04:00
.gitattributes local semantic search on abstracts 2026-08-05 15:26:55 -04:00
.gitignore Use user-provided codesign script for Windows signing 2024-04-08 10:12:58 +01:00
.gitmodules Rename pdf-worker submodule to document-worker 2026-04-22 15:50:02 -04:00
chrome.manifest Remove old citation dialogs (#5177) 2025-04-02 05:26:06 -04:00
CLAUDE.md Rename pdf-worker submodule to document-worker 2026-04-22 15:50:02 -04:00
CONTRIBUTING.md Add "support questions" to CONTRIBUTING.md 2022-12-01 05:10:42 -05:00
COPYING Update COPYING 2018-03-26 11:27:48 +02:00
eslint.config.mjs Convert ItemPane{SectionElement,Container}Base to ES modules 2026-05-18 15:48:31 -04:00
package-lock.json Update package-lock.json 2026-01-29 13:22:13 -05:00
package.json Upgrade ftl-tx. Fix #5671. 2025-12-20 02:22:23 +01:00
README.md "Zotero wiki" → "Zotero documentation" in app build README 2023-04-26 04:40:22 -04:00
update.rdf Update Fx minVersion to 45 2016-12-13 09:12:55 -05:00
version Update version to 11.0 2026-08-03 14:14:05 -04:00

README.md

Zotero

CI

Zotero is a free, easy-to-use tool to help you collect, organize, cite, and share your research sources.

Please post feature requests or bug reports to the Zotero Forums. If you're having trouble with Zotero, see Getting Help.

For more information on how to use this source code, see the Zotero documentation.