gpt-tokenizer

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

v3.4.0 URL: https://unpkg.com/gpt-tokenizer@3.4.0/dist/cl100k_base.js

Open Browse Files

BPE encoder decoder tokenizer GPT GPT-2 GPT-3 GPT-3.5 GPT-4 GPT-4o NLP Natural Language Processing Text Generation OpenAI Machine Learning ml

wink-tokenizer

Multilingual tokenizer that automatically tags each token with its type

v5.3.0 URL: https://unpkg.com/wink-tokenizer@5.3.0/src/wink-tokenizer.js

Open Browse Files

Tokenizer Tokenize Tags Tagging NLP email twitter URL Emoji Emoticon Multilingual French German Spanish Icelandic wink

@meistrari/gpt-tokenizer

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 / Claude Instant / Claude 2

v0.3.2 URL: https://unpkg.com/@meistrari/gpt-tokenizer@0.3.2/dist/cl100k_base.js

Open Browse Files

BPE encoder decoder tokenizer GPT GPT-2 GPT-3 GPT-3.5 GPT-4 NLP Natural Language Processing Text Generation OpenAI Machine Learning ml

happynodetokenizer

A simple, Twitter-aware tokenizer.

v7.1.0 URL: https://unpkg.com/happynodetokenizer@7.1.0/dist/index.js

Open Browse Files

tokenise tokenize tokenising tokenizing tokeniser tokenizer token twitter NLP language text strings stanford dlatk wwbp

gpt-tok

Lightweight trimmed down encoder/decoder/tokenizer/token counter for gpt3 that is compatible with both node and browser environments

v0.0.1 URL: https://unpkg.com/gpt-tok@0.0.1/dist/index.js

Open Browse Files

BPE encoder decoder tokenizer GPT GPT-2 GPT-3 GPT-3.5 GPT-4 NLP Natural Language Processing Text Generation OpenAI Machine Learning ml