Views
No views yet
0.4.01base_model: openlm-research/open_llama_3b_v2
2model_type: LlamaForCausalLM
3tokenizer_type: LlamaTokenizer
4load_in_8bit: true
5load_in_4bit: false
6strict: false
7push_dataset_to_hub:
8datasets:
9 - path: teknium/GPT4-LLM-Cleaned
10 type: alpaca
11dataset_prepared_path:
12val_set_size: 0.02
13adapter: lora
14lora_model_dir:
15sequence_len: 1024
16sample_packing: true
17lora_r: 8
18lora_alpha: 16
19lora_dropout: 0.0
20lora_target_modules:
21 - gate_proj
22 - down_proj
23 - up_proj
24 - q_proj
25 - v_proj
26 - k_proj
27 - o_proj
28lora_fan_in_fan_out:
29wandb_project:
30wandb_entity:
31wandb_watch:
32wandb_name:
33wandb_log_model:
34output_dir: ./outputs/lora-out
35gradient_accumulation_steps: 1
36micro_batch_size: 64
37num_epochs: 4
38optimizer: adamw_bnb_8bit
39torchdistx_path:
40lr_scheduler: cosine
41learning_rate: 0.0002
42train_on_inputs: false
43group_by_length: false
44bf16: false
45fp16: true
46tf32: false
47gradient_checkpointing: true
48early_stopping_patience:
49resume_from_checkpoint:
50local_rank:
51logging_steps: 1
52xformers_attention:
53flash_attention: true
54gptq_groupsize:
55s2_attention:
56gptq_model_v1:
57warmup_steps: 20
58evals_per_epoch: 4
59saves_per_epoch: 1
60debug:
61deepspeed:
62weight_decay: 0.1
63fsdp:
64fsdp_config:
65special_tokens:
66 bos_token: "<s>"
67 eos_token: "</s>"
68 unk_token: "<unk>"
69| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.3031 | 0.0064 | 1 | 1.5004 |
| 1.1084 | 0.2548 | 40 | 1.1224 |
| 1.0912 | 0.5096 | 80 | 1.0586 |
| 1.0727 | 0.7643 | 120 | 1.0301 |
| 1.0438 | 1.0191 | 160 | 1.0126 |
| 1.0126 | 1.2484 | 200 | 1.0035 |
| 1.048 | 1.5032 | 240 | 0.9938 |
| 1.0839 | 1.7580 | 280 | 0.9859 |
| 1.0817 | 2.0127 | 320 | 0.9801 |
| 1.0115 | 2.2420 | 360 | 0.9788 |
| 1.0356 | 2.4968 | 400 | 0.9730 |
| 0.992 | 2.7516 | 440 | 0.9725 |
| 1.0219 | 3.0064 | 480 | 0.9682 |
| 0.9637 | 3.2357 | 520 | 0.9707 |
| 1.0085 | 3.4904 | 560 | 0.9698 |
| 0.9547 | 3.7452 | 600 | 0.9697 |