gpt-tokenizer

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

v2.6.2 URL: https://unpkg.com/gpt-tokenizer@2.6.2/dist/cl100k_base.js

Open Browse Files

BPE encoder decoder tokenizer GPT GPT-2 GPT-3 GPT-3.5 GPT-4 GPT-4o NLP Natural Language Processing Text Generation OpenAI Machine Learning ml

bpe-tokenizer

Build your own vocabulary from application-specific corpus using Byte pair encoding (BPE) algorithm.

v2.2.0 URL: https://unpkg.com/bpe-tokenizer@2.2.0/core.js

Open Browse Files

BPE byte pair encoding algorithm sqlite browser cross-platform isomorphic natural language processing NLP GPT tokenizer typescript

@meistrari/gpt-tokenizer

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 / Claude Instant / Claude 2

v0.3.2 URL: https://unpkg.com/@meistrari/gpt-tokenizer@0.3.2/dist/cl100k_base.js

Open Browse Files

BPE encoder decoder tokenizer GPT GPT-2 GPT-3 GPT-3.5 GPT-4 NLP Natural Language Processing Text Generation OpenAI Machine Learning ml

@jill64/cf-tiktoken

⏳ js-tiktoken on Cloudflare Pages

v1.0.6 URL: https://unpkg.com/@jill64/cf-tiktoken@1.0.6/index.js

Open Browse Files

bpe cloudflare gpt llm pages tiktoken

@syonfox/gpt-3-encoder

Javascript BPE Encoder Decoder for GPT-2 / GPT-3. The "gpt-3-encoder" module provides functions for encoding and decoding text using the Byte Pair Encoding (BPE) algorithm. It can be used to process text data for input into machine learning models, or to

v1.4.0-rc5 URL: https://unpkg.com/@syonfox/gpt-3-encoder@1.4.0-rc5/index.js

Open Browse Files

JavaScript BPE Encoder Decoder GPT-2 GPT-3 Natural Language Processing (NLP)Text Generation Machine Learning