@orama/tokenizers

Additional tokenizers for Orama

v3.1.4 URL: https://unpkg.com/@orama/tokenizers@3.1.4

Open Browse Files

full-text search search fuzzy search typo-tolerant search full-text vector search stemming tokenizers mandarin chinese

@dvdagames/pgn-tokenizer

TypeScript version of PGN Tokenizer, a Byte Pair Encoding (BPE) tokenizer for Chess Portable Game Notiation (PGN).

v0.1.6 URL: https://unpkg.com/@dvdagames/pgn-tokenizer@0.1.6/dist/index.js

Open Browse Files

chess tokenizer pgn byte pair encoding bpe llm tokenizers

fib-tokenizers

cpp tokenizer module for fibjs.

v1.2.1 URL: https://unpkg.com/fib-tokenizers@1.2.1/index.js

Open Browse Files

tokenizers fib-tokenizers

openvino-tokenizers-node

OpenVINO™ Tokenizers adds text processing operations to openvino-node package

v2025.0.0 URL: https://unpkg.com/openvino-tokenizers-node@2025.0.0/openvino-tokenizers.js

Open Browse Files

OpenVINO openvino tokenizers OpenVINO tokenizers openvino tokenizers openvino-tokenizers openvino-tokenizers-node tokenization text processing string tensors text tensors

tokengeex

This repository holds the code for the TokenGeeX Rust crate and Python package. TokenGeeX is a tokenizer for [CodeGeeX](https://github.com/THUDM/Codegeex2) aimed at code and Chinese. It is based on [UnigramLM (Taku Kudo 2018)](https://arxiv.org/abs/1804.1

v0.6.2 URL: https://unpkg.com/tokengeex@0.6.2/index.node

Open Browse Files