GitHub
BERT, Tokenizer, Python, WordPiece, pybind11,C++,Flash,Trie
4 followers
BERT, Tokenizer, Python, WordPiece, pybind11,C++,Flash,Trie
4 followers
š **FlashTokenizer: World's Fastest CPU Tokenizer!** ā” 8~15x faster than `BertTokenizerFast` š ļø High-performance C++ š Parallel with OpenMP š¦ Easy pip install š» Cross-platform (Win/Mac/Linux) ā¶ļø Demo: https://youtu.be/a_sTiAXeSE0
