See axolotl config

axolotl version: 0.4.1

yaml

1base_model: meta-llama/Llama-3.2-1B 2model_type: LlamaForCausalLM 3tokenizer_type: AutoTokenizer 4 5load_in_8bit: false 6load_in_4bit: false 7strict: false 8 9chat_template: chatml 10datasets: 11 - path: datasets/airoboros_3.2_without_contextual_slimorca_orca_sharegpt.json 12 ds_type: json 13 type: sharegpt 14 conversation: chatml 15 16 - path: datasets/allenai_wild_chat_gpt4_english_toxic_random_half_4k_sharegpt.json 17 ds_type: json 18 type: sharegpt 19 strict: false 20 conversation: chatml 21 22 - path: datasets/buzz_unstacked_chosen_math_removed_filtered.json 23 ds_type: json 24 type: alpaca 25 conversation: chatml 26 27 - path: datasets/capybara_sharegpt.json 28 ds_type: json 29 type: sharegpt 30 conversation: chatml 31 32 - path: datasets/cot_alpaca_gpt4_extracted_openhermes_2.5_sharegpt.json 33 ds_type: json 34 type: sharegpt 35 conversation: chatml 36 37 - path: datasets/everythinglm-data-v3_sharegpt.json 38 ds_type: json 39 type: sharegpt 40 strict: false 41 conversation: chatml 42 43 - path: datasets/gpt4_data_lmys_1m_sharegpt.json 44 ds_type: json 45 type: sharegpt 46 conversation: chatml 47 48 - path: datasets/gpteacher-instruct-special-alpaca.json 49 ds_type: json 50 type: gpteacher 51 conversation: chatml 52 53 - path: datasets/merged_all.json 54 ds_type: json 55 type: alpaca 56 conversation: chatml 57 58 - path: datasets/no_robots_sharegpt.json 59 ds_type: json 60 type: sharegpt 61 strict: false 62 conversation: chatml 63 64 - path: datasets/oasst_top1_from_fusechatmixture_sharegpt.json 65 ds_type: json 66 type: sharegpt 67 strict: false 68 conversation: chatml 69 70 - path: datasets/pippa_bagel_repo_3k_sharegpt.json 71 ds_type: json 72 type: sharegpt 73 conversation: chatml 74 75 - path: datasets/rpguild_quarter_alignment_lab_sharegpt.json 76 ds_type: json 77 type: sharegpt 78 conversation: chatml 79 80 - path: datasets/sharegpt_gpt4_english.json 81 ds_type: json 82 type: sharegpt 83 conversation: chatml 84 85 - path: datasets/slimorca_dedup_filtered_95k_sharegpt.json 86 ds_type: json 87 type: sharegpt 88 conversation: chatml 89 90 - path: datasets/soda_diaolog_longest_tenth_buzz_sharegpt.json 91 ds_type: json 92 type: sharegpt 93 conversation: chatml 94 95 - path: datasets/synthia-v1.3_sharegpt_12500.json 96 ds_type: json 97 type: sharegpt 98 conversation: chatml 99 100 - path: datasets/system_conversations_dolphin_sharegpt.json 101 ds_type: json 102 type: sharegpt 103 conversation: chatml 104 105 - path: datasets/NuminaMath-CoT-olympiads-40k_alpaca.json 106 ds_type: json 107 type: alpaca 108 conversation: chatml 109 110 - path: datasets/math-gpt-4o-40k_alpaca.json 111 ds_type: json 112 type: alpaca 113 conversation: chatml 114 115 - path: datasets/sonnet3.5_science_conversations_sharegpt.json 116 ds_type: json 117 type: sharegpt 118 conversation: chatml 119 120 - path: datasets/reasoning-0.01_sharegpt.jsonl 121 ds_type: json 122 type: sharegpt 123 conversation: chatml 124 125dataset_prepared_path: last_run_prepared 126val_set_size: 0.002 127 128output_dir: ./Einstein-v8-Llama3.2-1B-model 129 130sequence_len: 8192 131sample_packing: true 132pad_to_sequence_len: true 133eval_sample_packing: false 134 135wandb_project: Einstein 136wandb_entity: 137wandb_watch: 138wandb_name: Einstein-v8-Llama3.2-1B-2-epoch 139wandb_log_model: 140hub_model_id: Weyaxi/Einstein-v8-Llama3.2-1B 141 142save_safetensors: true 143 144gradient_accumulation_steps: 4 145micro_batch_size: 4 146num_epochs: 2 147optimizer: adamw_bnb_8bit # look 148lr_scheduler: cosine 149learning_rate: 0.000005 # look 150 151train_on_inputs: false 152group_by_length: false 153bf16: true 154fp16: false 155tf32: false 156 157gradient_checkpointing: true 158early_stopping_patience: 159resume_from_checkpoint: 160local_rank: 161logging_steps: 1 162xformers_attention: 163flash_attention: true 164 165warmup_steps: 10 166evals_per_epoch: 4 167eval_table_size: 168eval_table_max_new_tokens: 128 169saves_per_epoch: 1 170debug: 171 172deepspeed: axolotl/deepspeed_configs/zero3_bf16.json 173weight_decay: 0.0 174fsdp: 175fsdp_config: 176special_tokens: 177 bos_token: "<s>" 178 eos_token: "<|im_end|>" 179 unk_token: "<unk>" 180 pad_token: <|end_of_text|> # changed 181tokens: 182 - "<|im_start|>" 183

Einstein-v8-Llama3.2-1B

• Loss: 0.9292

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 5e-06
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 2

Training results

Framework versions

• Transformers 4.45.0
• Pytorch 2.3.1+cu121
• Datasets 2.21.0
• Tokenizers 0.20.0

Open LLM Leaderboard Evaluation Results