GitHub

BERT, Tokenizer, Python, WordPiece, pybind11,C++,Flash,Trie

4 followers

BERT, Tokenizer, Python, WordPiece, pybind11,C++,Flash,Trie

4 followers

🚀 **FlashTokenizer: World's Fastest CPU Tokenizer!** ⚡ 8~15x faster than `BertTokenizerFast` 🛠️ High-performance C++ 🔄 Parallel with OpenMP 📦 Easy pip install 💻 Cross-platform (Win/Mac/Linux) ▶️ Demo: https://youtu.be/a_sTiAXeSE0