ksana-analyzer

convert text stream to tokens