nai-js-tokenizer

BPE tokenizer used for the NovelAI frontend.

v1.1.0 URL: https://unpkg.com/nai-js-tokenizer@1.1.0/lib/index.js

Open Browse Files

gpt gpt2 pile sentencepiece nerdstash tokenizer novelai