This model is a Continuous Pre-trained (CPT) model, adapted from lelapa/InkubaLM-0.4B on the custom dataset. In this new model we add different languages from Kenya, Uganda, South Africa, Western Africa, Somalia, Ethiopia; sub-saharan Africa.

Model description

Caracal GPT is a small causal model that can be used for fine-tuning tasks. It's goal is to be used by the represented language speaker for fine-tuning to a certain language task.

The Pre-trained LM was trained on 20+ African languages, introducing Kenyan lanugages and other languages not widely available in many datasets - Luo (luo), Kamba (kam) , Maasai (mas), Somalia (som), etc.

Intended uses & limitations

The model is to be used for fine-tuning on instruction set data for the given languages.

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0002
train_batch_size: 32
eval_batch_size: 8
seed: 42
gradient_accumulation_steps: 4
total_train_batch_size: 128
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 500
num_epochs: 4
loss: 1.8

Framework versions

Transformers 4.45.2
Pytorch 2.11.0+cu128
Datasets 2.21.0
Tokenizers 0.20.3

theophilusowiti/Caracal_GPT

Caracal_GPT

Model description

Intended uses & limitations

Training hyperparameters

Framework versions

theophilusowiti/
Caracal_GPT