Beta

Explore
Marketplace
Neural Labs
Chat
Wallet
Docs

Beta

Explore Marketplace Neural Labs Wallet Docs

Build

Manage

My assets Manage wallet Organization

Community

Discord X (Twitter)Telegram

Docs Privacy Policy Terms and Conditions

Proud member of

Proud member of

Docs Cookie Policy

Privacy Policy Terms and Conditions

© 2026 AlphaNeural AI. All rights reserved.

hindi-bpe-tokenizer – AI Model by Ankit91 | AlphaNeural AI

You can deploy this model and start earning money today!

Ankit91/
hindi-bpe-tokenizer

like

0

Views

No views yet

Hindi BPE Tokenizer

This tokenizer was trained on a mixed Hindi dataset using Byte-Pair Encoding (BPE).

Vocabulary size: 6000
Base tokens: UTF-8 bytes (256)
Trained in Python using a custom implementation
Original length: 7053637
Compressed length: 869484
Compression ratio: 8.11X

Files

vocab.json: Token IDs
merges.json: Merge rules
metadata.json: Tokenizer configuration