GitHub
p/github-77BERT, Tokenizer, Python, WordPiece, pybind11,C++,Flash,Trie
ā€¢0 reviewsā€¢4 followers
Start new thread
trending
rowenā€¢

10mo ago

GitHub - BERT, Tokenizer, Python, WordPiece, pybind11,C++,Flash,Trie

šŸš€ **FlashTokenizer: World's Fastest CPU Tokenizer!** āš” 8~15x faster than `BertTokenizerFast` šŸ› ļø High-performance C++ šŸ”„ Parallel with OpenMP šŸ“¦ Easy pip install šŸ’» Cross-platform (Win/Mac/Linux) ā–¶ļø Demo: https://youtu.be/a_sTiAXeSE0