Views
No views yet
0.5.01base_model: openlm-research/open_llama_3b_v2
2model_type: LlamaForCausalLM
3tokenizer_type: LlamaTokenizer
4load_in_8bit: true
5load_in_4bit: false
6strict: false
7push_dataset_to_hub:
8datasets:
9 - path: vicgalle/alpaca-gpt4
10 type: alpaca
11dataset_prepared_path:
12val_set_size: 0.02
13adapter: lora
14lora_model_dir:
15sequence_len: 1024
16sample_packing: true
17lora_r: 8
18lora_alpha: 16
19lora_dropout: 0.0
20lora_target_modules:
21 - gate_proj
22 - down_proj
23 - up_proj
24 - q_proj
25 - v_proj
26 - k_proj
27 - o_proj
28lora_fan_in_fan_out:
29wandb_project:
30wandb_entity:
31wandb_watch:
32wandb_name:
33wandb_log_model:
34output_dir: ./outputs/lora-out
35gradient_accumulation_steps: 1
36micro_batch_size: 2
37num_epochs: 4
38optimizer: adamw_bnb_8bit
39torchdistx_path:
40lr_scheduler: cosine
41learning_rate: 0.0002
42train_on_inputs: false
43group_by_length: false
44bf16: false
45fp16: true
46tf32: false
47gradient_checkpointing: true
48early_stopping_patience:
49resume_from_checkpoint:
50local_rank:
51logging_steps: 1
52xformers_attention:
53flash_attention: true
54gptq_groupsize:
55s2_attention:
56gptq_model_v1:
57warmup_steps: 20
58evals_per_epoch: 4
59saves_per_epoch: 1
60debug:
61deepspeed:
62weight_decay: 0.1
63fsdp:
64fsdp_config:
65special_tokens:
66 bos_token: "<s>"
67 eos_token: "</s>"
68 unk_token: "<unk>"
69| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.1997 | 0.0002 | 1 | 1.3698 |
| 1.1468 | 0.25 | 1404 | 1.1159 |
| 1.2207 | 0.5 | 2808 | 1.1072 |
| 0.9448 | 0.75 | 4212 | 1.0982 |
| 1.0709 | 1.0 | 5616 | 1.0931 |
| 0.9592 | 1.2498 | 7020 | 1.1051 |
| 1.1133 | 1.4998 | 8424 | 1.1058 |
| 0.884 | 1.7498 | 9828 | 1.1018 |
| 0.9117 | 1.9998 | 11232 | 1.0963 |
| 0.9594 | 2.2496 | 12636 | 1.1336 |
| 0.9034 | 2.4996 | 14040 | 1.1338 |
| 0.6645 | 2.7496 | 15444 | 1.1326 |
| 0.8913 | 2.9996 | 16848 | 1.1309 |
| 0.9476 | 3.2495 | 18252 | 1.1752 |
| 0.9015 | 3.4995 | 19656 | 1.1762 |
| 0.6284 | 3.7495 | 21060 | 1.1768 |
| 0.7522 | 3.9995 | 22464 | 1.1770 |