You can deploy this model and start earning money today!

LazarusNLP/
IndoNanoT5-base

Name: IndoNanoT5-base
Rating: 2.5 (77 reviews)

text-generation-inference

endpoints_compatible

Views

No views yet

IndoNanoT5 Base

IndoNanoT5 Base is an Indonesian sequence-to-sequence language model based on the T5 architecture. We conducted pre-training on an open-source Indonesian corpus of uonlp/CulturaX. On a held-out subset of the corpus, our model achieved an evaluation loss of 2.082 or a perplexity of about 8.02.

This model was trained using the nanoT5 PyTorch framework. All training was done on an NVIDIA H100 GPU. LazarusNLP/IndoNanoT5-base is released under Apache 2.0 license.

Model Detail

Developed by: LazarusNLP
Model type: Encoder-decoder T5 transformer language model
Language(s): Indonesian
License: Apache 2.0
Contact: Wilson Wongso

Use in 🤗Transformers

python

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_checkpoint = "LazarusNLP/IndoNanoT5-base"

tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint)

Training Datasets

Around 4B tokens from the following corpora were used during pre-training.

Cleaned, Enormous, and Public: The Multilingual Fuel to Democratize Large Language Models for 167 Languages

Training Hyperparameters

The following hyperparameters were used during training:

total_steps: 65536
input_length: 512
batch_size: 128
grad_acc: 1
base_lr: 5e-3
optimizer: AdamWScaled with betas=(0.9,0.999) and epsilon=1e-08
weight_decay: 0.0
lr_scheduler: cosine
warmup_steps: 10000
final_cosine: 1e-5
grad_clip: 1.0
precision: bf16

Acknowledgements

We would like to acknowledge nanoT5 for inspiring this project.

Credits

BhinnekaLM is developed with love by:

GitHub Profile