See axolotl config

axolotl version: 0.12.0.dev0

yaml

1base_model: minpeter/tiny-ko-187m-base-250725 2 3hub_model_id: minpeter/tiny-ko-187m-sft-250725 4output_dir: ./outputs/tiny-ko-187m-sft-250725 5wandb_project: "axolotl" 6wandb_entity: "kasfiekfs-e" 7 8model_type: LlamaForCausalLM 9tokenizer_type: AutoTokenizer 10 11strict: false 12 13chat_template: chatml 14datasets: 15 - path: HuggingFaceTB/smol-smoltalk 16 type: chat_template 17 split: train 18 field_messages: messages 19 message_property_mappings: 20 role: role 21 content: content 22 23 - path: trillionlabs/multisystem-curated 24 type: chat_template 25 split: train 26 field_messages: messages 27 message_property_mappings: 28 role: role 29 content: content 30 31 - path: allenai/tulu-3-sft-personas-instruction-following 32 type: chat_template 33 split: train 34 field_messages: messages 35 message_property_mappings: 36 role: role 37 content: content 38 39 - path: lemon-mint/smol-koreantalk 40 type: chat_template 41 split: train 42 field_messages: messages 43 message_property_mappings: 44 role: role 45 content: content 46 47 - path: lemon-mint/Korean-FineTome-100k 48 type: chat_template 49 split: train 50 field_messages: messages 51 message_property_mappings: 52 role: role 53 content: content 54 55 - path: heegyu/open-korean-instructions-v20231020 56 type: chat_template 57 split: train 58 field_messages: conversations 59 message_property_mappings: 60 role: from 61 content: value 62 roles: 63 user: ["human", "user"] 64 assistant: ["gpt", "assistant", "bot"] 65 system: ["system", "input"] 66 67 - path: coastral/korean-writing-style-instruct 68 type: chat_template 69 split: train 70 field_messages: conversations 71 message_property_mappings: 72 role: from 73 content: value 74 75 - path: devngho/korean-instruction-mix 76 type: chat_template 77 split: train 78 field_messages: messages 79 message_property_mappings: 80 role: from 81 content: value 82 83dataset_prepared_path: last_run_prepared 84val_set_size: 0.001 85save_safetensors: true 86sequence_len: 8192 87sample_packing: false 88pad_to_sequence_len: false 89use_pose: true 90pose_max_context_len: 65536 91 92overrides_of_model_config: 93 rope_theta: 1000000.0 94 max_position_embeddings: 65536 95 96gradient_accumulation_steps: 8 97micro_batch_size: 16 98num_epochs: 1 99optimizer: muon 100lr_scheduler: cosine 101learning_rate: 3e-4 102 103train_on_inputs: false 104group_by_length: false 105bf16: true 106fp16: 107tf32: true 108 109gradient_checkpointing: false 110gradient_checkpointing_kwargs: 111 use_reentrant: true 112early_stopping_patience: 113resume_from_checkpoint: 114local_rank: 115logging_steps: 1 116xformers_attention: 117flash_attention: true 118sdp_attention: 119s2_attention: 120 121save_steps: 200 122warmup_steps: 20 123eval_steps: 200 124debug: 125deepspeed: 126weight_decay: 0.0 127fsdp: 128fsdp_config: 129 130special_tokens: 131 eos_token: '<|im_end|>' 132 133plugins: 134 - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin 135 - axolotl.integrations.liger.LigerPlugin 136# - axolotl.integrations.lm_eval.LMEvalPlugin 137 138# lm_eval_tasks: 139# - gsm8k 140# - hellaswag 141# - arc_easy 142# - arc_challenge 143# - piqa 144# - winogrande 145# - openbookqa 146# - wsc 147# - boolq 148 149liger_rope: true 150liger_rms_norm: true 151liger_glu_activation: true 152liger_layer_norm: true 153liger_fused_linear_cross_entropy: true 154

tiny-ko-187m-sft-250725

• Loss: 1.6810

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 0.0003
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 128
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 20
• training_steps: 13880

Training results

Framework versions

• Transformers 4.53.2
• Pytorch 2.7.1+cu126
• Datasets 4.0.0
• Tokenizers 0.21.2