kitoken

Fast and versatile tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization