wink-tokenizer

Multilingual tokenizer that automatically tags each token with its type

v5.3.0 URL: https://unpkg.com/wink-tokenizer@5.3.0/src/wink-tokenizer.js

Open Browse Files

Tokenizer Tokenize Tags Tagging NLP email twitter URL Emoji Emoticon Multilingual French German Spanish Icelandic wink

kingchop

A text tokenizing library that handles strings, by tokenizing them into arrays, depending on intended format, like sentences, sub sentences, paragraphs, words...

v2.0.2 URL: https://unpkg.com/kingchop@2.0.2/index.js

Open Browse Files

NLP tokenizer Tokenizer Sentence tokenizer Text tokenizer Node.js tokenizer Flexible text tokenizer CommonJS text tokenizer Paragraph tokenizer Sub sentence tokenizer Stable tokenizer Easy tokenizer Light text tokenizer

@coder-seb/tokenizer

Lexical tokenizer with grammar classes ready to go.

v1.0.10 URL: https://unpkg.com/@coder-seb/tokenizer@1.0.10/src/app.js

Open Browse Files

Tokenizer coder-seb

@dmitrysoshnikov/lex-js

Lexer generator from RegExp spec

v0.1.10 URL: https://unpkg.com/@dmitrysoshnikov/lex-js@0.1.10/index.js

Open Browse Files

Lexer Tokenizer RegExp Parser Token Scanner JavaScript

lucene-tokenizers

Port from Apache Lucene

v5.3.3 URL: https://unpkg.com/lucene-tokenizers@5.3.3/lucene-tokenizers.babel.js

Open Browse Files

Lucene Tokenizer JFlex