You can deploy this model and start earning money today!

tranhuyHoang/
mini_VN_decoder_tokenizer

Name: mini_VN_decoder_tokenizer
Availability: InStock

VTSNLP/vietnamese_curated_dataset

apache-2.0

Views

No views yet

Vietnamese Tokenizer

This repository contains a ByteLevel BPE tokenizer trained from scratch specifically for the Vietnamese language, designed for decoder-only language model pretraining.

🚀 Usage

Load tokenizer

python

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "tranhuyHoang/mini_VN_decoder_tokenizer",
    use_fast=True
)