Views
No views yet
| Parameter | Value |
|---|---|
| LoRA rank | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| DoRA | False |
| Learning rate | 0.0002 |
| Batch size | 8 |
| Gradient accumulation | 4 |
| Epochs | 1 |
| Max seq length | 1024 |
| Warmup ratio | 0.1 |
| Dtype | bf16 |
{"loss": 1.6248851776123048, "grad_norm": 1.4257605075836182, "learning_rate": 0.00017058823529411766, "entropy": 1.579152311384678, "mean_token_accuracy": 0.6371947549283504, "num_tokens": 52078.0, "epoch": 0.26666666666666666, "step": 10}
{"loss": 0.4948576927185059, "grad_norm": 0.6731943488121033, "learning_rate": 0.00011176470588235294, "entropy": 0.5243682205677033, "mean_token_accuracy": 0.8807921424508095, "num_tokens": 104429.0, "epoch": 0.5333333333333333, "step": 20}
{"loss": 0.3364424228668213, "grad_norm": 0.7052034735679626, "learning_rate": 5.294117647058824e-05, "entropy": 0.34181889072060584, "mean_token_accuracy": 0.9150784552097321, "num_tokens": 156603.0, "epoch": 0.8, "step": 30}
{"train_runtime": 67.3283, "train_samples_per_second": 17.749, "train_steps_per_second": 0.564, "total_flos": 1992908272281216.0, "train_loss": 0.702773696497867, "entropy": 0.28197900851567587, "mean_token_accuracy": 0.9320357978343964, "num_tokens": 195384.0, "epoch": 1.0, "step": 38}| Package | Version |
|---|---|
| torch | 2.13.0 |
| transformers | 5.14.1 |
| trl | 1.9.2 |
| datasets | 5.0.1 |
| accelerate | 1.14.0 |
| python-dotenv | 1.2.2 |
| peft | 0.20.0 |
| bitsandbytes | 0.50.0 |
| huggingface-hub | ? |
| jinja2 | ? |
| torchvision | 0.28.0 |
| pillow | 12.3.0 |