overview-js-tokenizer

Splits text into tokens in as many languages as possible, using libicu

Browse on unpkg