Commit graph

3 commits

Author SHA1 Message Date
Yujong Lee
661da87c91 fix(rust): validate tokenizer ranks and cover backend features 2026-09-20 15:13:59 -07:00
Yujong Lee
56c5d31e73 test(rust): run fast token counter parity tests by default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-20 21:52:48 +00:00
Yujong Lee
84c26978a3 refactor(rust): split token counter backends 2026-09-20 14:10:31 -07:00