Views
No views yet
1accelerate launch $BASE_DIR/qlora/train.py \
2 --model_name_or_path $BASE_MODEL \
3 --working_dir $BASE_DIR/$OUTPUT_NAME-checkpoints \
4 --output_dir $BASE_DIR/$OUTPUT_NAME-peft \
5 --merged_output_dir $BASE_DIR/$OUTPUT_NAME \
6 --final_output_dir $BASE_DIR/$OUTPUT_NAME-final \
7 --num_train_epochs 1 \
8 --logging_steps 1 \
9 --save_strategy steps \
10 --save_steps 75 \
11 --save_total_limit 2 \
12 --data_seed 11422 \
13 --evaluation_strategy steps \
14 --per_device_eval_batch_size 4 \
15 --eval_dataset_size 0.01 \
16 --eval_steps 75 \
17 --max_new_tokens 1024 \
18 --dataloader_num_workers 3 \
19 --logging_strategy steps \
20 --do_train \
21 --do_eval \
22 --lora_r 64 \
23 --lora_alpha 16 \
24 --lora_modules all \
25 --bits 4 \
26 --double_quant \
27 --quant_type nf4 \
28 --lr_scheduler_type constant \
29 --dataset habanoz/airoboros-3.1-no-mathjson-max-1k \
30 --dataset_format airoboros_chat \
31 --model_max_len 1024 \
32 --per_device_train_batch_size 4 \
33 --gradient_accumulation_steps 4 \
34 --learning_rate 1e-5 \
35 --adam_beta2 0.999 \
36 --max_grad_norm 0.3 \
37 --lora_dropout 0.0 \
38 --weight_decay 0.0 \
39 --seed 11422 \
40 --gradient_checkpointing \
41 --use_flash_attention_2 \
42 --ddp_find_unused_parameters False| Metric | Value |
|---|---|
| Avg. | 34.98 |
| AI2 Reasoning Challenge (25-Shot) | 30.72 |
| HellaSwag (10-Shot) | 54.32 |
| MMLU (5-Shot) | 24.78 |
| TruthfulQA (0-shot) | 41.67 |
| Winogrande (5-shot) | 57.62 |
| GSM8k (5-shot) | 0.76 |