Views
No views yet
0.4.01adapter: qlora
2base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
3bf16: false
4dataset_prepared_path: null
5datasets:
6- path: mhenrichsen/alpaca_2k_test
7 type: alpaca
8debug: null
9deepspeed: null
10early_stopping_patience: null
11evals_per_epoch: null
12flash_attention: false
13fp16: true
14fsdp: null
15fsdp_config: null
16gradient_accumulation_steps: 1
17gradient_checkpointing: true
18group_by_length: false
19is_llama_derived_model: true
20learning_rate: 0.0002
21load_in_4bit: true
22load_in_8bit: false
23local_rank: null
24logging_steps: 1
25lora_alpha: 16
26lora_dropout: 0.05
27lora_fan_in_fan_out: null
28lora_model_dir: null
29lora_r: 32
30lora_target_linear: true
31lora_target_modules: null
32lr_scheduler: cosine
33max_steps: 20
34micro_batch_size: 1
35mlflow_experiment_name: colab-example
36model_type: LlamaForCausalLM
37num_epochs: 4
38optimizer: paged_adamw_32bit
39output_dir: ./qlora-out
40pad_to_sequence_len: true
41resume_from_checkpoint: null
42sample_packing: true
43saves_per_epoch: null
44sequence_len: 1096
45special_tokens: null
46strict: false
47tf32: false
48tokenizer_type: LlamaTokenizer
49train_on_inputs: false
50val_set_size: 0.05
51wandb_entity: null
52wandb_log_model: null
53wandb_name: null
54wandb_project: null
55wandb_watch: null
56warmup_steps: 10
57weight_decay: 0.0
58xformers_attention: null
59| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 4.0306 | 0.05 | 20 | 3.5158 |