Beta

Explore
Marketplace
Neural Labs
Chat
Wallet
Docs

Beta

Explore Marketplace Neural Labs Wallet Docs

Build

Manage

My assets Manage wallet Organization

Community

Discord X (Twitter)Telegram

Docs Privacy Policy Terms and Conditions

Proud member of

Proud member of

Docs Cookie Policy

Privacy Policy Terms and Conditions

© 2026 AlphaNeural AI. All rights reserved.

xlm-roberta-tokenizer-50k – AI Model by metythorn | AlphaNeural AI

You can deploy this model and start earning money today!

metythorn/
xlm-roberta-tokenizer-50k

like

0

custom-khmer-dataset

Views

No views yet

XLM-RoBERTa Khmer Custom Tokenizer

This is a custom tokenizer trained for Khmer language based on XLM-RoBERTa architecture.

Model Details

Model Type: XLM-RoBERTa Tokenizer
Language: Khmer (km)
Training Data: Custom Khmer dataset with ~84M examples
Tokenizer Type: SentencePiece-based

Usage

python

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("metythorn/xlm-roberta-tokenizer-50k")

# Example usage
text = "ខ្ញុំចង់រៀនភាសាខ្មែរ"
tokens = tokenizer.tokenize(text)
print(tokens)

Training Details

Base model: XLM-RoBERTa
Training examples: 84Million
Epochs: 3
Batch size: 8
Parameters: 278,295,186

Intended Use

This tokenizer is designed for:

Khmer text processing
Masked language modeling
Fine-tuning on Khmer NLP tasks
Research in Khmer language understanding

Limitations

Primarily trained on Khmer text
May not handle code-switching effectively
Performance on formal vs informal Khmer may vary