wink-tokenizer

Multilingual tokenizer that automatically tags each token with its type

v5.3.0 URL: https://unpkg.com/wink-tokenizer@5.3.0/src/wink-tokenizer.js

Open Browse Files

Tokenizer Tokenize Tags Tagging NLP email twitter URL Emoji Emoticon Multilingual French German Spanish Icelandic wink

lucene-tokenizers

Port from Apache Lucene

v5.3.3 URL: https://unpkg.com/lucene-tokenizers@5.3.3/lucene-tokenizers.babel.js

Open Browse Files

Lucene Tokenizer JFlex

libcore-tokenizer

NFA Tokenizer

v0.1.8 URL: https://unpkg.com/libcore-tokenizer@0.1.8/dist/libcore-tokenizer.js

Open Browse Files

libcore Tokenizer Automata Automaton Finite Non-Deterministic ES5

@dmitrysoshnikov/lex-js

Lexer generator from RegExp spec

v0.1.10 URL: https://unpkg.com/@dmitrysoshnikov/lex-js@0.1.10/index.js

Open Browse Files

Lexer Tokenizer RegExp Parser Token Scanner JavaScript

@ranchonyx/nanoparse

A super simple basic parser / tokenizer for easier processing of various configuration files

v1.1.0 URL: https://unpkg.com/@ranchonyx/nanoparse@1.1.0/dist/BasicParser.js

Open Browse Files

Tokenizer Text processing Parsing Lightweight