Trained on a relatively small dataset
Loss is still relatively high (3.8027), meaning the model may generate inaccurate or repetitive text
Not recommended for production use without further fine-tuning

How to Use

python

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("EhabSuliman/my_awesome_eli5_clm-model")
model = AutoModelForCausalLM.from_pretrained("EhabSuliman/my_awesome_eli5_clm-model")

prompt = "Somatic hypermutation allows the immune system to"
inputs = tokenizer(prompt, return_tensors="pt").input_ids
outputs = model.generate(
    inputs,
    max_new_tokens=100,
    do_sample=True,
    top_k=50,
    top_p=0.95
)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

Training and Evaluation Data

Fine-tuned on an Arabic text dataset using the ELI5 (Explain Like I'm 5) format.

Training Procedure

Training Hyperparameters

The following hyperparameters were used during training:

Learning rate: 2e-05
Train batch size: 8
Eval batch size: 8
Seed: 42
Optimizer: AdamW (fused) with betas=(0.9, 0.999), epsilon=1e-08
LR scheduler: Linear
Epochs: 3

Training Results

Training Loss	Epoch	Step	Validation Loss
3.8556	1.0	1327	3.8101
3.7851	2.0	2654	3.8035
3.7514	3.0	3981	3.8027

The model shows steady improvement across epochs with validation loss decreasing from 3.8101 → 3.8027.

Framework Versions

Transformers 5.9.0
PyTorch 2.11.0+cu128
Datasets 4.0.0
Tokenizers 0.22.2

Author

Ehab Suliman — Machine Learning Engineer
🔗 HuggingFace Profile