@orama/tokenizers

Additional tokenizers for Orama

v3.1.16 URL: https://unpkg.com/@orama/tokenizers@3.1.16

Open Browse Files

full-text search search fuzzy search typo-tolerant search full-text vector search stemming tokenizers mandarin chinese

openvino-tokenizers-node

OpenVINO™ Tokenizers adds text processing operations to openvino-node package

v2025.3.0 URL: https://unpkg.com/openvino-tokenizers-node@2025.3.0/openvino-tokenizers.js

Open Browse Files

OpenVINO openvino tokenizers OpenVINO tokenizers openvino tokenizers openvino-tokenizers openvino-tokenizers-node tokenization text processing string tensors text tensors

fib-tokenizers

cpp tokenizer module for fibjs.

v1.2.1 URL: https://unpkg.com/fib-tokenizers@1.2.1/index.js

Open Browse Files

tokenizers fib-tokenizers

@dvdagames/pgn-tokenizer

TypeScript version of PGN Tokenizer, a Byte Pair Encoding (BPE) tokenizer for Chess Portable Game Notiation (PGN).

v0.1.6 URL: https://unpkg.com/@dvdagames/pgn-tokenizer@0.1.6/dist/index.js

Open Browse Files

chess tokenizer pgn byte pair encoding bpe llm tokenizers

@naveen521kk/expo-hf-tokenizers

Port of HuggingFace's tokenizers using Expo Modules for React Native Apps

v0.1.0 URL: https://unpkg.com/@naveen521kk/expo-hf-tokenizers@0.1.0/build/index.js

Open Browse Files

react-native expo expo-hf-tokenizers ExpoHfTokenizers tokenizers HuggingFace