Views
No views yet
1accelerate launch finetune.py \
2 --model_name_or_path TinyLlama/TinyLlama-1.1B-intermediate-step-715k-1.5T \
3 --output_dir ./output/1_5T_FT_lr1e-5_ep5_top1_2023-08-25 \
4 --logging_steps 10 \
5 --save_strategy epoch \
6 --data_seed 42 \
7 --save_total_limit 2 \
8 --evaluation_strategy epoch \
9 --eval_dataset_size 512 \
10 --max_eval_samples 1000 \
11 --per_device_eval_batch_size 1 \
12 --max_new_tokens 32 \
13 --dataloader_num_workers 3 \
14 --group_by_length=False \
15 --logging_strategy steps \
16 --remove_unused_columns False \
17 --do_train \
18 --do_eval \
19 --warmup_ratio 0.05 \
20 --lr_scheduler_type constant \
21 --dataset OpenAssistant/oasst_top1_2023-08-25 \
22 --dataset_format oasst1 \
23 --source_max_len 1 \
24 --target_max_len 1023 \
25 --per_device_train_batch_size 2 \
26 --gradient_accumulation_steps 8 \
27 --max_steps 0 \
28 --num_train_epochs 5 \
29 --learning_rate 1e-5 \
30 --adam_beta2 0.999 \
31 --max_grad_norm 1.0 \
32 --weight_decay 0.0 \
33 --seed 0 \
34 --trust_remote_code| Metric | Value |
|---|---|
| Avg. | 35.58 |
| AI2 Reasoning Challenge (25-Shot) | 32.85 |
| HellaSwag (10-Shot) | 58.16 |
| MMLU (5-Shot) | 25.96 |
| TruthfulQA (0-shot) | 38.35 |
| Winogrande (5-shot) | 57.70 |
| GSM8k (5-shot) | 0.45 |