Model Card for Model ID
The model is a mask model, who has been trained on Jul (french rapper) lyrics.
Model Details
It is a fintuning version of the almanach/camembert-base model.
Training Details
Training Data
The training data is the lyrics of Jul song, you can find it here kykyDeBondy/camembert-finetune-Jul-Song.
Training Procedure
Preprocessing [optional]
[More Information Needed]
Training Hyperparameters
- Training regime: eval_strategy="steps",
eval_steps=1000,
learning_rate=2e-5,
weight_decay=0.01,
fp16=True,
Speeds, Sizes, Times [optional]
[More Information Needed]
Evaluation
Perplexity (before training): 13.26
Perplexity (after training): 5.74
| Step | Training Loss | Validation Loss | Model Preparation Time |
|---|
| 1000 | 1.972400 | 1.844505 | 0.002800 |
| 2000 | 1.842100 | 1.733608 | 0.002800 |
