Views
No views yet
<|im_start|>system
You are Dolphin, a helpful AI assistant.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
| Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
|-----------------------------------------------------------|-------|------|-----:|-----------------------|---|-----:|---|------|
|leaderboard |N/A |none | 0|acc |↑ |0.3437|± |0.0043|
| | |none | 0|acc_norm |↑ |0.5076|± |0.0053|
| | |none | 0|exact_match |↑ |0.0536|± |0.0061|
| | |none | 0|inst_level_loose_acc |↑ |0.4388|± |N/A |
| | |none | 0|inst_level_strict_acc |↑ |0.3741|± |N/A |
| | |none | 0|prompt_level_loose_acc |↑ |0.3105|± |0.0199|
| | |none | 0|prompt_level_strict_acc|↑ |0.2477|± |0.0186|
| - leaderboard_bbh |N/A |none | 3|acc_norm |↑ |0.5549|± |0.0061|
| - leaderboard_bbh_boolean_expressions | 0|none | 3|acc_norm |↑ |0.8640|± |0.0217|
| - leaderboard_bbh_causal_judgement | 0|none | 3|acc_norm |↑ |0.6417|± |0.0352|
| - leaderboard_bbh_date_understanding | 0|none | 3|acc_norm |↑ |0.6080|± |0.0309|
| - leaderboard_bbh_disambiguation_qa | 0|none | 3|acc_norm |↑ |0.6480|± |0.0303|
| - leaderboard_bbh_formal_fallacies | 0|none | 3|acc_norm |↑ |0.5360|± |0.0316|
| - leaderboard_bbh_geometric_shapes | 0|none | 3|acc_norm |↑ |0.5240|± |0.0316|
| - leaderboard_bbh_hyperbaton | 0|none | 3|acc_norm |↑ |0.6440|± |0.0303|
| - leaderboard_bbh_logical_deduction_five_objects | 0|none | 3|acc_norm |↑ |0.4600|± |0.0316|
| - leaderboard_bbh_logical_deduction_seven_objects | 0|none | 3|acc_norm |↑ |0.4680|± |0.0316|
| - leaderboard_bbh_logical_deduction_three_objects | 0|none | 3|acc_norm |↑ |0.7000|± |0.0290|
| - leaderboard_bbh_movie_recommendation | 0|none | 3|acc_norm |↑ |0.8160|± |0.0246|
| - leaderboard_bbh_navigate | 0|none | 3|acc_norm |↑ |0.6040|± |0.0310|
| - leaderboard_bbh_object_counting | 0|none | 3|acc_norm |↑ |0.3680|± |0.0306|
| - leaderboard_bbh_penguins_in_a_table | 0|none | 3|acc_norm |↑ |0.5548|± |0.0413|
| - leaderboard_bbh_reasoning_about_colored_objects | 0|none | 3|acc_norm |↑ |0.6320|± |0.0306|
| - leaderboard_bbh_ruin_names | 0|none | 3|acc_norm |↑ |0.7440|± |0.0277|
| - leaderboard_bbh_salient_translation_error_detection | 0|none | 3|acc_norm |↑ |0.5280|± |0.0316|
| - leaderboard_bbh_snarks | 0|none | 3|acc_norm |↑ |0.6292|± |0.0363|
| - leaderboard_bbh_sports_understanding | 0|none | 3|acc_norm |↑ |0.8040|± |0.0252|
| - leaderboard_bbh_temporal_sequences | 0|none | 3|acc_norm |↑ |0.4680|± |0.0316|
| - leaderboard_bbh_tracking_shuffled_objects_five_objects | 0|none | 3|acc_norm |↑ |0.2160|± |0.0261|
| - leaderboard_bbh_tracking_shuffled_objects_seven_objects| 0|none | 3|acc_norm |↑ |0.1160|± |0.0203|
| - leaderboard_bbh_tracking_shuffled_objects_three_objects| 0|none | 3|acc_norm |↑ |0.3000|± |0.0290|
| - leaderboard_bbh_web_of_lies | 0|none | 3|acc_norm |↑ |0.4880|± |0.0317|
| - leaderboard_gpqa |N/A |none | 0|acc_norm |↑ |0.3146|± |0.0135|
| - leaderboard_gpqa_diamond | 1|none | 0|acc_norm |↑ |0.3182|± |0.0332|
| - leaderboard_gpqa_extended | 1|none | 0|acc_norm |↑ |0.3187|± |0.0200|
| - leaderboard_gpqa_main | 1|none | 0|acc_norm |↑ |0.3080|± |0.0218|
| - leaderboard_ifeval | 2|none | 0|inst_level_loose_acc |↑ |0.4388|± |N/A |
| | |none | 0|inst_level_strict_acc |↑ |0.3741|± |N/A |
| | |none | 0|prompt_level_loose_acc |↑ |0.3105|± |0.0199|
| | |none | 0|prompt_level_strict_acc|↑ |0.2477|± |0.0186|
| - leaderboard_math_algebra_hard | 1|none | 4|exact_match |↑ |0.0749|± |0.0150|
| - leaderboard_math_counting_and_prob_hard | 1|none | 4|exact_match |↑ |0.0244|± |0.0140|
| - leaderboard_math_geometry_hard | 1|none | 4|exact_match |↑ |0.0227|± |0.0130|
| - leaderboard_math_hard |N/A |none | 4|exact_match |↑ |0.0536|± |0.0061|
| - leaderboard_math_intermediate_algebra_hard | 1|none | 4|exact_match |↑ |0.0250|± |0.0093|
| - leaderboard_math_num_theory_hard | 1|none | 4|exact_match |↑ |0.0390|± |0.0156|
| - leaderboard_math_prealgebra_hard | 1|none | 4|exact_match |↑ |0.1295|± |0.0242|
| - leaderboard_math_precalculus_hard | 1|none | 4|exact_match |↑ |0.0296|± |0.0146|
| - leaderboard_mmlu_pro | 0.1|none | 5|acc |↑ |0.3437|± |0.0043|
| - leaderboard_musr |N/A |none | 0|acc_norm |↑ |0.4511|± |0.0178|
| - leaderboard_musr_murder_mysteries | 1|none | 0|acc_norm |↑ |0.5880|± |0.0312|
| - leaderboard_musr_object_placements | 1|none | 0|acc_norm |↑ |0.3438|± |0.0297|
| - leaderboard_musr_team_allocation | 1|none | 0|acc_norm |↑ |0.4240|± |0.0313|
| Groups |Version|Filter|n-shot| Metric | |Value | |Stderr|
|------------------------|-------|------|-----:|-----------------------|---|-----:|---|------|
|leaderboard |N/A |none | 0|acc |↑ |0.3437|± |0.0043|
| | |none | 0|acc_norm |↑ |0.5076|± |0.0053|
| | |none | 0|exact_match |↑ |0.0536|± |0.0061|
| | |none | 0|inst_level_loose_acc |↑ |0.4388|± |N/A |
| | |none | 0|inst_level_strict_acc |↑ |0.3741|± |N/A |
| | |none | 0|prompt_level_loose_acc |↑ |0.3105|± |0.0199|
| | |none | 0|prompt_level_strict_acc|↑ |0.2477|± |0.0186|
| - leaderboard_bbh |N/A |none | 3|acc_norm |↑ |0.5549|± |0.0061|
| - leaderboard_gpqa |N/A |none | 0|acc_norm |↑ |0.3146|± |0.0135|
| - leaderboard_math_hard|N/A |none | 4|exact_match |↑ |0.0536|± |0.0061|
| - leaderboard_musr |N/A |none | 0|acc_norm |↑ |0.4511|± |0.0178|
0.4.11base_model: /workspace/models/Mistral-Nemo-Base-2407
2model_type: AutoModelForCausalLM
3tokenizer_type: AutoTokenizer
4
5load_in_8bit: false
6# load_in_4bit: true
7strict: false
8
9datasets:
10 - path: /workspace/datasets/dolphin-2.9.3/dolphin201-sharegpt2.jsonl
11 type: sharegpt
12 conversation: chatml
13 - path: /workspace/datasets/dolphin-2.9.3/SystemChat_filtered_sharegpt.jsonl
14 type: sharegpt
15 conversation: chatml
16 - path: /workspace/datasets/dolphin-2.9.3/SystemChat_multilingual_sharegpt.jsonl
17 type: sharegpt
18 conversation: chatml
19 - path: /workspace/datasets/dolphin-2.9.3/dolphin-coder-translate-sharegpt2.jsonl
20 type: sharegpt
21 conversation: chatml
22 - path: /workspace/datasets/dolphin-2.9.3/dolphin-coder-codegen-sharegpt2.jsonl
23 type: sharegpt
24 conversation: chatml
25 - path: /workspace/datasets/dolphin-2.9.3/m-a-p_Code-Feedback-sharegpt-unfiltered.jsonl
26 type: sharegpt
27 conversation: chatml
28 - path: /workspace/datasets/dolphin-2.9.3/m-a-p_CodeFeedback-Filtered-Instruction-sharegpt-unfiltered.jsonl
29 type: sharegpt
30 conversation: chatml
31 - path: /workspace/datasets/dolphin-2.9.3/not_samantha_norefusals.jsonl
32 type: sharegpt
33 conversation: chatml
34 - path: /workspace/datasets/dolphin-2.9.3/Orca-Math-resort-unfiltered.jsonl
35 type: sharegpt
36 conversation: chatml
37 - path: /workspace/datasets/dolphin-2.9.3/agent_instruct_react_unfiltered.jsonl
38 type: sharegpt
39 conversation: chatml
40 - path: /workspace/datasets/dolphin-2.9.3/toolbench_instruct_j1s1_3k_unfiltered.jsonl
41 type: sharegpt
42 conversation: chatml
43 - path: /workspace/datasets/dolphin-2.9.3/toolbench_negative_unfiltered.jsonl
44 type: sharegpt
45 conversation: chatml
46 - path: /workspace/datasets/dolphin-2.9.3/toolbench_react_10p_unfiltered.jsonl
47 type: sharegpt
48 conversation: chatml
49 - path: /workspace/datasets/dolphin-2.9.3/toolbench_tflan_cot_30p_unfiltered.jsonl
50 type: sharegpt
51 conversation: chatml
52 - path: /workspace/datasets/dolphin-2.9.3/openhermes200k_unfiltered.jsonl
53 type: sharegpt
54 conversation: chatml
55
56chat_template: chatml
57# adapter: qlora
58# lora_r: 128
59# lora_alpha: 16
60# lora_modules_to_save: [embed_tokens, lm_head]
61# lora_dropout: 0.05
62# lora_target_linear: true
63
64
65unfrozen_parameters:
66- ^lm_head.weight$
67- ^model.embed_tokens.weight$
68- input_layernorm
69- model.norm
70- post_attention_layernorm
71- self_attn.rotary_emb
72# mlp.down_proj layers
73- model.layers.0.mlp.down_proj
74- model.layers.1.mlp.down_proj
75- model.layers.4.mlp.down_proj
76- model.layers.37.mlp.down_proj
77- model.layers.24.mlp.down_proj
78- model.layers.2.mlp.down_proj
79- model.layers.38.mlp.down_proj
80- model.layers.35.mlp.down_proj
81- model.layers.25.mlp.down_proj
82- model.layers.6.mlp.down_proj
83- model.layers.22.mlp.down_proj
84- model.layers.23.mlp.down_proj
85- model.layers.3.mlp.down_proj
86- model.layers.21.mlp.down_proj
87- model.layers.5.mlp.down_proj
88- model.layers.28.mlp.down_proj
89- model.layers.20.mlp.down_proj
90- model.layers.26.mlp.down_proj
91- model.layers.19.mlp.down_proj
92- model.layers.34.mlp.down_proj
93# mlp.gate_proj layers
94- model.layers.2.mlp.gate_proj
95- model.layers.1.mlp.gate_proj
96- model.layers.3.mlp.gate_proj
97- model.layers.5.mlp.gate_proj
98- model.layers.4.mlp.gate_proj
99- model.layers.35.mlp.gate_proj
100- model.layers.36.mlp.gate_proj
101- model.layers.37.mlp.gate_proj
102- model.layers.38.mlp.gate_proj
103- model.layers.34.mlp.gate_proj
104- model.layers.33.mlp.gate_proj
105- model.layers.8.mlp.gate_proj
106- model.layers.32.mlp.gate_proj
107- model.layers.6.mlp.gate_proj
108- model.layers.28.mlp.gate_proj
109- model.layers.26.mlp.gate_proj
110- model.layers.30.mlp.gate_proj
111- model.layers.23.mlp.gate_proj
112- model.layers.29.mlp.gate_proj
113- model.layers.27.mlp.gate_proj
114# mlp.up_proj layers
115- model.layers.3.mlp.up_proj
116- model.layers.4.mlp.up_proj
117- model.layers.6.mlp.up_proj
118- model.layers.2.mlp.up_proj
119- model.layers.5.mlp.up_proj
120- model.layers.8.mlp.up_proj
121- model.layers.10.mlp.up_proj
122- model.layers.9.mlp.up_proj
123- model.layers.7.mlp.up_proj
124- model.layers.0.mlp.up_proj
125- model.layers.17.mlp.up_proj
126- model.layers.15.mlp.up_proj
127- model.layers.22.mlp.up_proj
128- model.layers.18.mlp.up_proj
129- model.layers.16.mlp.up_proj
130- model.layers.11.mlp.up_proj
131- model.layers.21.mlp.up_proj
132- model.layers.23.mlp.up_proj
133- model.layers.20.mlp.up_proj
134- model.layers.27.mlp.up_proj
135# self_attn.k_proj layers
136- model.layers.30.self_attn.k_proj
137- model.layers.27.self_attn.k_proj
138- model.layers.25.self_attn.k_proj
139- model.layers.33.self_attn.k_proj
140- model.layers.26.self_attn.k_proj
141- model.layers.31.self_attn.k_proj
142- model.layers.35.self_attn.k_proj
143- model.layers.39.self_attn.k_proj
144- model.layers.22.self_attn.k_proj
145- model.layers.24.self_attn.k_proj
146- model.layers.21.self_attn.k_proj
147- model.layers.28.self_attn.k_proj
148- model.layers.23.self_attn.k_proj
149- model.layers.36.self_attn.k_proj
150- model.layers.20.self_attn.k_proj
151- model.layers.37.self_attn.k_proj
152- model.layers.29.self_attn.k_proj
153- model.layers.32.self_attn.k_proj
154- model.layers.16.self_attn.k_proj
155- model.layers.18.self_attn.k_proj
156# self_attn.o_proj layers
157- model.layers.7.self_attn.o_proj
158- model.layers.6.self_attn.o_proj
159- model.layers.9.self_attn.o_proj
160- model.layers.5.self_attn.o_proj
161- model.layers.27.self_attn.o_proj
162- model.layers.26.self_attn.o_proj
163- model.layers.4.self_attn.o_proj
164- model.layers.31.self_attn.o_proj
165- model.layers.8.self_attn.o_proj
166- model.layers.16.self_attn.o_proj
167- model.layers.3.self_attn.o_proj
168- model.layers.10.self_attn.o_proj
169- model.layers.18.self_attn.o_proj
170- model.layers.33.self_attn.o_proj
171- model.layers.17.self_attn.o_proj
172- model.layers.32.self_attn.o_proj
173- model.layers.30.self_attn.o_proj
174- model.layers.2.self_attn.o_proj
175- model.layers.15.self_attn.o_proj
176- model.layers.11.self_attn.o_proj
177# self_attn.q_proj layers
178- model.layers.14.self_attn.q_proj
179- model.layers.11.self_attn.q_proj
180- model.layers.15.self_attn.q_proj
181- model.layers.9.self_attn.q_proj
182- model.layers.8.self_attn.q_proj
183- model.layers.18.self_attn.q_proj
184- model.layers.12.self_attn.q_proj
185- model.layers.13.self_attn.q_proj
186- model.layers.19.self_attn.q_proj
187- model.layers.16.self_attn.q_proj
188- model.layers.10.self_attn.q_proj
189- model.layers.17.self_attn.q_proj
190- model.layers.7.self_attn.q_proj
191- model.layers.5.self_attn.q_proj
192- model.layers.20.self_attn.q_proj
193- model.layers.3.self_attn.q_proj
194- model.layers.26.self_attn.q_proj
195- model.layers.27.self_attn.q_proj
196- model.layers.28.self_attn.q_proj
197- model.layers.33.self_attn.q_proj
198# self_attn.v_proj layers
199- model.layers.27.self_attn.v_proj
200- model.layers.20.self_attn.v_proj
201- model.layers.24.self_attn.v_proj
202- model.layers.25.self_attn.v_proj
203- model.layers.30.self_attn.v_proj
204- model.layers.2.self_attn.v_proj
205- model.layers.23.self_attn.v_proj
206- model.layers.22.self_attn.v_proj
207- model.layers.26.self_attn.v_proj
208- model.layers.33.self_attn.v_proj
209- model.layers.37.self_attn.v_proj
210- model.layers.7.self_attn.v_proj
211- model.layers.4.self_attn.v_proj
212- model.layers.18.self_attn.v_proj
213- model.layers.31.self_attn.v_proj
214- model.layers.17.self_attn.v_proj
215- model.layers.35.self_attn.v_proj
216- model.layers.32.self_attn.v_proj
217- model.layers.21.self_attn.v_proj
218- model.layers.3.self_attn.v_proj
219
220
221
222dataset_prepared_path: /workspace/axolotl/dolph-2.9.3-nemo-prepared
223val_set_size: 0.01
224output_dir: /workspace/axolotl/dolphin-2.9.3-mistral-nemo
225
226sequence_len: 8192
227sample_packing: true
228pad_to_sequence_len: true
229
230wandb_project: dolphin-2.9.3-Mistral-nemo
231wandb_watch:
232wandb_run_id:
233wandb_log_model:
234
235gradient_accumulation_steps: 16
236micro_batch_size: 1
237num_epochs: 3
238optimizer: adamw_torch
239lr_scheduler: cosine
240learning_rate: 5e-6
241train_on_inputs: false
242group_by_length: false
243bf16: auto
244fp16:
245tf32:
246
247gradient_checkpointing: true
248gradient_checkpointing_kwargs:
249 use_reentrant: false
250early_stopping_patience:
251resume_from_checkpoint:
252logging_steps: 1
253xformers_attention:
254flash_attention: true
255
256warmup_steps: 100
257# evals_per_epoch: 4
258eval_table_size:
259saves_per_epoch: 1
260save_total_limit: 2
261save_steps:
262debug:
263deepspeed: deepspeed_configs/zero3_bf16.json
264weight_decay: 0.1
265special_tokens:
266 eos_token: "<|im_end|>"
267 pad_token: "<pad>"
268 bos_token: "<s>"
269 unk_token: "<unk>"
270tokens:
271 - "<|im_start|>"
272
273
274# fsdp:
275# - full_shard
276# - auto_wrap
277# fsdp_config:
278# fsdp_limit_all_gathers: true
279# fsdp_sync_module_states: true
280# fsdp_offload_params: true
281# fsdp_use_orig_params: false
282# fsdp_cpu_ram_efficient_loading: true
283# fsdp_transformer_layer_cls_to_wrap: MixtralSparseMoeBlock
284# fsdp_state_dict_type: FULL_STATE_DICT
285# fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
286# fsdp_sharding_strategy: FULL_SHARD
287# fsdp_forward_prefetch: false
288# fsdp_backward_prefetch: BACKWARD_PRE| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 0.5691 | 1.0162 | 983 | 0.5734 |
| 0.5335 | 2.0174 | 1968 | 0.5609 |
| 0.5297 | 2.9639 | 2901 | 0.5605 |
