Beta

Explore
Marketplace
Neural Labs
Chat
Wallet
Docs

Beta

Explore Marketplace Neural Labs Wallet Docs

Build

Manage

My assets Manage wallet Organization

Community

Discord X (Twitter)Telegram

Docs Privacy Policy Terms and Conditions

Proud member of

Proud member of

Docs Cookie Policy

Privacy Policy Terms and Conditions

© 2026 AlphaNeural AI. All rights reserved.

hf-fast-tokenizer-v22b2 – AI Model by llm-jp | AlphaNeural AI

You can deploy this model and start earning money today!

llm-jp/
hf-fast-tokenizer-v22b2

like

0

Views

No views yet

The Hugging Face fast tokenizer for LLM-jp ABCI challenge 2023.

The vocab size is 96,867.

Requirements:

transformers>=4.34.0
tokenizers>=0.14.0
torch

Usage:

Python

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("llm-jp/hf-fast-tokenizer-v22b2")