Views
No views yet
0.5.01adapter: qlora
2base_model: matrixportal/Metafor
3bf16: true
4dataset_prepared_path: last_run_prepared
5datasets:
6- path: matrixportal/aya_dataset_alpaca
7 type: alpaca
8debug: null
9deepspeed: null
10early_stopping_patience: null
11eval_sample_packing: true
12eval_table_size: null
13evals_per_epoch: 1
14flash_attention: false
15fp16: null
16fsdp: null
17fsdp_config: null
18gradient_accumulation_steps: 2
19gradient_checkpointing: true
20group_by_length: false
21learning_rate: 2e-5
22load_in_4bit: true
23load_in_8bit: false
24logging_steps: 1
25lora_alpha: 16
26lora_dropout: 0.05
27lora_fan_in_fan_out: null
28lora_model_dir: null
29lora_modules_to_save:
30- embed_tokens
31- lm_head
32lora_r: 32
33lora_target_linear: true
34lr_scheduler: cosine
35max_steps: 25
36micro_batch_size: 1
37num_epochs: 1
38optimizer: paged_adamw_8bit
39output_dir: lora-out
40pad_to_sequence_len: true
41resume_from_checkpoint: null
42sample_packing: true
43saves_per_epoch: 1
44sdp_attention: true
45sequence_len: 2048
46special_tokens:
47 pad_token: <|end_of_text|>
48strict: false
49tf32: false
50train_on_inputs: false
51val_set_size: 0.05
52wandb_entity: null
53wandb_log_model: null
54wandb_name: null
55wandb_project: null
56wandb_watch: null
57warmup_steps: 1
58weight_decay: 0.0
59xformers_attention: null
60| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 5.0384 | 0.0015 | 25 | 3.4000 |