image/png ❗⚠️ WARNING ⚠️❗ ❗ This model has been deprecated due to a sliding window error in the base model's configuration. This issue has been resolved with the
following commit in the base model , and upcoming versions of the Einstein series will utilize the correct configuration in the base model.
🔬 Einstein-v5-v0.2-7B
This model is a full fine-tuned version of
alpindale/Mistral-7B-v0.2-hf on diverse datasets.
This model is finetuned using
8xRTX3090 +
1xRTXA6000 using
axolotl .
This model's training was sponsored by
sablo.ai .
See axolotl config
axolotl version: 0.4.0
1 base_model : alpindale/Mistral - 7B - v0.2 - hf
2 model_type : MistralForCausalLM
3 tokenizer_type : LlamaTokenizer
4 is_mistral_derived_model : true
5
6 load_in_8bit : false
7 load_in_4bit : false
8 strict : false
9
10 chat_template : chatml
11 datasets :
12 - path : data/merged_all.json
13 ds_type : json
14 type : alpaca
15 conversation : chatml
16
17 - path : data/gpteacher - instruct - special - alpaca.json
18 ds_type : json
19 type : gpteacher
20 conversation : chatml
21
22 - path : data/capybara_sharegpt.json
23 ds_type : json
24 type : sharegpt
25 conversation : chatml
26
27 - path : data/synthia - v1.3_sharegpt_12500.json
28 ds_type : json
29 type : sharegpt
30 conversation : chatml
31
32 - path : data/cot_alpaca_gpt4_extracted_openhermes_2.5_sharegpt.json
33 ds_type : json
34 type : sharegpt
35 conversation : chatml
36
37 - path : data/slimorca_dedup_filtered_95k_sharegpt.json
38 ds_type : json
39 type : sharegpt
40 conversation : chatml
41
42 - path : data/airoboros_3.2_without_contextual_slimorca_orca_sharegpt.json
43 ds_type : json
44 type : sharegpt
45 conversation : chatml
46
47 - path : data/allenai_wild_chat_gpt4_english_toxic_random_half_4k_sharegpt.json
48 ds_type : json
49 type : sharegpt
50 strict : false
51 conversation : chatml
52
53 - path : data/pippa_bagel_repo_3k_sharegpt.json
54 ds_type : json
55 type : sharegpt
56 conversation : chatml
57
58 - path : data/gpt4_data_lmys_1m_sharegpt.json
59 ds_type : json
60 type : sharegpt
61 conversation : chatml
62
63 - path : data/sharegpt_gpt4_english.json
64 ds_type : json
65 type : sharegpt
66 conversation : chatml
67
68 dataset_prepared_path : last_run_prepared
69 # val_set_size: 0.005
70 val_set_size : 0.0
71
72 do_bench_eval : true
73
74 output_dir : ./Einstein - v5 - Mistral - v0.2 - beta - model
75
76 sequence_len : 8192
77 sample_packing : true
78 pad_to_sequence_len : true
79 eval_sample_packing : false
80
81 wandb_project : Einstein
82 wandb_entity :
83 wandb_watch :
84 wandb_name :
85 wandb_log_model :
86 hub_model_id : Weyaxi/Einstein - v5 - Mistral - v0.2 - beta
87
88 save_safetensors : true
89
90 gradient_accumulation_steps : 4
91 micro_batch_size : 1
92 num_epochs : 2
93 optimizer : adamw_bnb_8bit
94 lr_scheduler : cosine
95 learning_rate : 0.000005
96
97 train_on_inputs : false
98 group_by_length : false
99 bf16 : true
100 fp16 : false
101 tf32 : false
102
103 gradient_checkpointing : true
104 early_stopping_patience :
105 resume_from_checkpoint :
106 local_rank :
107 logging_steps : 1
108 xformers_attention :
109 flash_attention : true
110
111 warmup_steps : 10
112 evals_per_epoch : 3 # changed
113 eval_table_size :
114 eval_table_max_new_tokens : 128
115 saves_per_epoch : 3 # changed
116 debug :
117
118 deepspeed : zero3_bf16.json
119 weight_decay : 0.0
120 fsdp :
121 fsdp_config :
122 special_tokens :
123 bos_token : "<s>"
124 eos_token : "<|im_end|>"
125 unk_token : "<unk>"
126 tokens :
127 - "<|im_start|>"
💬 Prompt Template
You can use this prompt template while using the model:
ChatML
<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{asistant}<|im_end|>This prompt template is available as a
chat template , which means you can format messages using the
tokenizer.apply_chat_template() method:
1 messages = [
2 { "role" : "system" , "content" : "You are helpful AI asistant." } ,
3 { "role" : "user" , "content" : "Hello!" }
4 ]
5 gen_input = tokenizer . apply_chat_template ( message , return_tensors = "pt" )
6 model . generate ( ** gen_input )
🔄 Quantizationed versions
Quantizationed versions of this model is available.
Detailed results can be found
here Metric Value Avg. 65.65 AI2 Reasoning Challenge (25-Shot) 60.92 HellaSwag (10-Shot) 80.99 MMLU (5-Shot) 61.02 TruthfulQA (0-shot) 52.59 Winogrande (5-shot) 78.69 GSM8k (5-shot) 59.67
🤖 Additional information about training
This model is full fine-tuned for 1 epoch.
Total number of steps was 1124.
Loss graph
image/png 🤝 Acknowledgments
Thanks to
sablo.ai for sponsoring this model.
Thanks to all the dataset authors mentioned in the datasets section.
Thanks to
axolotl for making the repository I used to make this model.
Thanks to all open source AI community.
If you would like to support me:
