Views
No views yet
1xH100 using axolotl.0.4.01base_model: mistralai/Mistral-7B-v0.1
2model_type: MistralForCausalLM
3tokenizer_type: LlamaTokenizer
4
5load_in_8bit: false
6load_in_4bit: true
7strict: false
8
9datasets:
10 - path: Open-Orca/SlimOrca
11 type: sharegpt
12dataset_prepared_path: last_run_prepared
13val_set_size: 0.1
14output_dir: ./qlora-out_2
15
16adapter: qlora
17lora_model_dir:
18
19sequence_len: 8192
20sample_packing: true
21pad_to_sequence_len: true
22
23lora_r: 128
24lora_alpha: 32
25lora_dropout: 0.05
26lora_target_linear: true
27lora_fan_in_fan_out:
28lora_target_modules:
29 - gate_proj
30 - down_proj
31 - up_proj
32 - q_proj
33 - v_proj
34 - k_proj
35 - o_proj
36
37wandb_project: slim_orca
38wandb_entity:
39wandb_watch:
40wandb_name:
41wandb_log_model:
42
43gradient_accumulation_steps: 4
44micro_batch_size: 2
45num_epochs: 1
46optimizer: adamw_bnb_8bit
47lr_scheduler: cosine
48learning_rate: 0.0002
49
50train_on_inputs: false
51group_by_length: false
52bf16: auto
53fp16:
54tf32: false
55
56gradient_checkpointing: true
57early_stopping_patience:
58resume_from_checkpoint:
59local_rank:
60logging_steps: 1
61xformers_attention:
62flash_attention: true
63
64loss_watchdog_threshold: 5.0
65loss_watchdog_patience: 3
66
67warmup_steps: 10
68evals_per_epoch: 4
69eval_table_size:
70eval_max_new_tokens: 128
71saves_per_epoch: 1
72debug:
73deepspeed:
74weight_decay: 0.0
75fsdp:
76fsdp_config:
77special_tokens:
78 bos_token: "<s>"
79 eos_token: "</s>"
80 unk_token: "<unk>"
81| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | |
|---|---|---|---|---|---|---|---|
| winogrande | 1 | none | None | acc | 0.7498 | ± | 0.0122 |
| piqa | 1 | none | None | acc | 0.8172 | ± | 0.0090 |
| none | None | acc_norm | 0.8286 | ± | 0.0088 | ||
| openbookqa | 1 | none | None | acc | 0.3380 | ± | 0.0212 |
| none | None | acc_norm | 0.4420 | ± | 0.0222 | ||
| hellaswag | 1 | none | None | acc | 0.6254 | ± | 0.0048 |
| none | None | acc_norm | 0.8061 | ± | 0.0039 | ||
| boolq | 2 | none | None | acc | 0.8740 | ± | 0.0058 |
| arc_easy | 1 | none | None | acc | 0.8199 | ± | 0.0079 |
| none | None | acc_norm | 0.7891 | ± | 0.0084 | ||
| arc_challenge | 1 | none | None | acc | 0.5145 | ± | 0.0146 |
| none | None | acc_norm | 0.5461 | ± | 0.0145 | ||
| Average: 71.93 |
