printable-characters

A little helper for handling strings containing zero width control characters, ANSI styling, whitespaces, newlines, 💩, etc.
OpenBrowse Files
string widthstring lengthreal string widthreal string lengthoptical string widthprinted widthunicodecodepointscode pointsstrlenzero widthzero width symbolszero width charactersvisible charactersvisible symbolsvisibleinvisibleinvisible symbolsinvisible charactersprintableprintable lengthprintable symbolsprintable charactersnon-printable charactersnonprintable characterscharacterssymbolsstring lengthreal string lengthstring trimmingtrimmingescapesescape codescodesansi escapestokenizingansiwhitespaces

happynodetokenizer

A simple, Twitter-aware tokenizer.
OpenBrowse Files
tokenisetokenizetokenisingtokenizingtokenisertokenizertokentwitterNLPlanguagetextstringsstanforddlatkwwbp

printable-characters-plus-directional

A little helper for handling strings containing zero width control characters, ANSI styling, whitespaces, newlines, etc.
OpenBrowse Files
string widthstring lengthreal string widthreal string lengthoptical string widthprinted widthunicodecodepointscode pointsstrlenzero widthzero width symbolszero width charactersvisible charactersvisible symbolsvisibleinvisibleinvisible symbolsinvisible charactersprintableprintable lengthprintable symbolsprintable charactersnon-printable charactersnonprintable characterscharacterssymbolsstring lengthreal string lengthstring trimmingtrimmingescapesescape codescodesansi escapestokenizingansiwhitespaces

sentence-tools

Tools for manipulating sentences
OpenBrowse Files
sentenceparsingtokenizationtokenizing

ksana-analyzer

convert text stream to tokens
OpenBrowse Files
tokenizingsegmentation