Search

@orama/tokenizers

Additional tokenizers for Orama

v3.0.2 URL: https://unpkg.com/@orama/tokenizers@3.0.2/index.js

OpenBrowse Files

full-text searchsearchfuzzy searchtypo-tolerant searchfull-textvector searchstemmingtokenizersmandarinchinese

fib-tokenizers

cpp tokenizer module for fibjs.

v1.2.1 URL: https://unpkg.com/fib-tokenizers@1.2.1/index.js

OpenBrowse Files

tokenizersfib-tokenizers

tokengeex

This repository holds the code for the TokenGeeX Rust crate and Python package. TokenGeeX is a tokenizer for [CodeGeeX](https://github.com/THUDM/Codegeex2) aimed at code and Chinese. It is based on [UnigramLM (Taku Kudo 2018)](https://arxiv.org/abs/1804.1

v0.6.2 URL: https://unpkg.com/tokengeex@0.6.2/index.node

OpenBrowse Files

NLPtokenizers

@naveen521kk/expo-hf-tokenizers

Port of HuggingFace's tokenizers using Expo Modules for React Native Apps

v0.1.0 URL: https://unpkg.com/@naveen521kk/expo-hf-tokenizers@0.1.0/build/index.js

OpenBrowse Files

react-nativeexpoexpo-hf-tokenizersExpoHfTokenizerstokenizersHuggingFace