Views
No views yet
1hyperparameters ={
2 'num_train_epochs': 3, # number of training epochs
3 'per_device_train_batch_size': 6, # batch size for training
4 'gradient_accumulation_steps': 2, # Number of updates steps to accumulate
5 'gradient_checkpointing': True, # save memory but slower backward pass
6 'bf16': True, # use bfloat16 precision
7 'tf32': True, # use tf32 precision
8 'learning_rate': 2e-4, # learning rate
9 'max_grad_norm': 0.3, # Maximum norm (for gradient clipping)
10 'warmup_ratio': 0.03, # warmup ratio
11 "lr_scheduler_type":"constant", # learning rate scheduler
12 'save_strategy': "epoch", # save strategy for checkpoints
13 "logging_steps": 10, # log every x steps
14 'merge_adapters': True, # wether to merge LoRA into the model (needs more memory)
15 'use_flash_attn': True, # Whether to use Flash Attention
16}
17| Metric | Value |
|---|---|
| Avg. | 34.04 |
| AI2 Reasoning Challenge (25-Shot) | 30.55 |
| HellaSwag (10-Shot) | 53.70 |
| MMLU (5-Shot) | 26.07 |
| TruthfulQA (0-shot) | 35.85 |
| Winogrande (5-shot) | 58.09 |
| GSM8k (5-shot) | 0.00 |