wink-tokenizer

Multilingual tokenizer that automatically tags each token with its type

v5.3.0 URL: https://unpkg.com/wink-tokenizer@5.3.0/src/wink-tokenizer.js

Open Browse Files

Tokenizer Tokenize Tags Tagging NLP email twitter URL Emoji Emoticon Multilingual French German Spanish Icelandic wink

lucene-tokenizers

Port from Apache Lucene

v5.3.3 URL: https://unpkg.com/lucene-tokenizers@5.3.3/lucene-tokenizers.babel.js

Open Browse Files

Lucene Tokenizer JFlex

bleuscore-js

A fast bleu score calculator

v0.1.4 URL: https://unpkg.com/bleuscore-js@0.1.4/bleuscore_js.js

Open Browse Files

NLP Tokenizer BLEU DeepLearning

@dmitrysoshnikov/lex-js

Lexer generator from RegExp spec

v0.1.10 URL: https://unpkg.com/@dmitrysoshnikov/lex-js@0.1.10/index.js

Open Browse Files

Lexer Tokenizer RegExp Parser Token Scanner JavaScript

rwkv-tokenizer-node

RWKV / gpt-NeoX / Pythia, 0-dep tokenizer library, for nodejs

v1.0.5 URL: https://unpkg.com/rwkv-tokenizer-node@1.0.5/src/tokenizer.js

Open Browse Files

RWKV Tokenizer neox gpt-neox pythia