Views
No views yet
0.4.01base_model: openlm-research/open_llama_3b_v2
2model_type: LlamaForCausalLM
3tokenizer_type: LlamaTokenizer
4load_in_8bit: false
5load_in_4bit: true
6strict: false
7push_dataset_to_hub:
8datasets:
9 - path: mhenrichsen/alpaca_2k_test
10 type: alpaca
11dataset_prepared_path:
12val_set_size: 0.05
13adapter: qlora
14lora_model_dir:
15sequence_len: 1024
16sample_packing: true
17lora_r: 8
18lora_alpha: 32
19lora_dropout: 0.05
20lora_target_modules:
21lora_target_linear: true
22lora_fan_in_fan_out:
23wandb_project:
24wandb_entity:
25wandb_watch:
26wandb_name:
27wandb_log_model:
28output_dir: ./qlora-out
29gradient_accumulation_steps: 1
30micro_batch_size: 2
31num_epochs: 4
32optimizer: paged_adamw_32bit
33torchdistx_path:
34lr_scheduler: cosine
35learning_rate: 0.0002
36train_on_inputs: false
37group_by_length: false
38bf16: false
39fp16: true
40tf32: false
41gradient_checkpointing: true
42early_stopping_patience:
43resume_from_checkpoint:
44local_rank:
45logging_steps: 1
46xformers_attention:
47flash_attention: true
48gptq_groupsize:
49gptq_model_v1:
50warmup_steps: 20
51evals_per_epoch: 4
52saves_per_epoch: 1
53debug:
54deepspeed:
55weight_decay: 0.1
56fsdp:
57fsdp_config:
58special_tokens:
59 bos_token: "<s>"
60 eos_token: "</s>"
61 unk_token: "<unk>"
62| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.3026 | 0.01 | 1 | 1.3435 |
| 1.1146 | 0.25 | 50 | 1.1476 |
| 1.2387 | 0.5 | 100 | 1.1319 |
| 1.4159 | 0.75 | 150 | 1.1192 |
| 1.2807 | 1.01 | 200 | 1.1153 |
| 1.0465 | 1.24 | 250 | 1.1569 |
| 0.9577 | 1.49 | 300 | 1.1493 |
| 1.1257 | 1.74 | 350 | 1.1462 |
| 0.9404 | 1.99 | 400 | 1.1520 |
| 0.7161 | 2.22 | 450 | 1.2603 |
| 0.5897 | 2.47 | 500 | 1.2661 |
| 0.5271 | 2.72 | 550 | 1.2814 |
| 0.6239 | 2.97 | 600 | 1.2705 |
| 0.3486 | 3.21 | 650 | 1.3848 |
| 0.5591 | 3.46 | 700 | 1.4171 |
| 0.3804 | 3.71 | 750 | 1.4177 |