You can deploy this model and start earning money today!

See axolotl config

axolotl version: 0.12.0.dev0

yaml

base_model: minpeter/tiny-ko-187m-base-250718

hub_model_id: minpeter/tiny-ko-187m-sft-250718
output_dir: ./outputs/tiny-ko-187m-sft-250718
wandb_project: "axolotl"
wandb_entity: "kasfiekfs-e"

model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer

strict: false

chat_template: chatml
datasets:
  - path: HuggingFaceTB/smol-smoltalk
    type: chat_template
    split: train
    field_messages: messages
    message_property_mappings:
      role: role
      content: content

  - path: trillionlabs/multisystem-curated
    type: chat_template
    split: train
    field_messages: messages
    message_property_mappings:
      role: role
      content: content

  - path: allenai/tulu-3-sft-personas-instruction-following
    type: chat_template
    split: train
    field_messages: messages
    message_property_mappings:
      role: role
      content: content

  - path: lemon-mint/smol-koreantalk
    type: chat_template
    split: train
    field_messages: messages
    message_property_mappings:
      role: role
      content: content

  - path: lemon-mint/Korean-FineTome-100k
    type: chat_template
    split: train
    field_messages: messages
    message_property_mappings:
      role: role
      content: content

  - path: heegyu/open-korean-instructions-v20231020
    type: chat_template
    split: train
    field_messages: conversations
    message_property_mappings:
      role: from
      content: value
    roles:
      user: ["human", "user"]
      assistant: ["gpt", "assistant", "bot"]
      system: ["system", "input"]

  - path: coastral/korean-writing-style-instruct
    type: chat_template
    split: train
    field_messages: conversations
    message_property_mappings:
      role: from
      content: value

  - path: devngho/korean-instruction-mix
    type: chat_template
    split: train
    field_messages: messages
    message_property_mappings:
      role: from
      content: value

dataset_prepared_path: last_run_prepared
val_set_size: 0.001
save_safetensors: true
sequence_len: 8192
sample_packing: false
pad_to_sequence_len: false
use_pose: true
pose_max_context_len: 65536

overrides_of_model_config:
  rope_theta: 1000000.0
  max_position_embeddings: 65536

gradient_accumulation_steps: 8
micro_batch_size: 16
num_epochs: 1
optimizer: muon
lr_scheduler: cosine
learning_rate: 3e-4

train_on_inputs: false
group_by_length: false
bf16: true
fp16:
tf32: true

gradient_checkpointing: false
gradient_checkpointing_kwargs:
  use_reentrant: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
sdp_attention:
s2_attention:

save_steps: 200
warmup_steps: 20
eval_steps: 200
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:

special_tokens:
  eos_token: '<|im_end|>'

plugins:
  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
  - axolotl.integrations.liger.LigerPlugin
  - axolotl.integrations.lm_eval.LMEvalPlugin

lm_eval_tasks:
  - gsm8k
  - hellaswag
  - arc_easy
  - arc_challenge
  - piqa
  - winogrande
  - openbookqa
  - wsc
  - boolq

liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_layer_norm: true
liger_fused_linear_cross_entropy: true

tiny-ko-187m-sft-250718

This model is a fine-tuned version of minpeter/tiny-ko-187m-base-250718 on the HuggingFaceTB/smol-smoltalk, the trillionlabs/multisystem-curated, the allenai/tulu-3-sft-personas-instruction-following, the lemon-mint/smol-koreantalk, the lemon-mint/Korean-FineTome-100k, the heegyu/open-korean-instructions-v20231020, the coastral/korean-writing-style-instruct and the devngho/korean-instruction-mix datasets. It achieves the following results on the evaluation set:

Loss: 1.6990

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0003
train_batch_size: 16
eval_batch_size: 16
seed: 42
distributed_type: multi-GPU
num_devices: 4
gradient_accumulation_steps: 8
total_train_batch_size: 512
total_eval_batch_size: 64
optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 20
training_steps: 3470

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0	0	2.1799
1.8649	0.0576	200	1.8603
1.8031	0.1153	400	1.8033
1.7128	0.1729	600	1.7709
1.7758	0.2306	800	1.7492
1.7084	0.2882	1000	1.7339
1.7258	0.3458	1200	1.7225
1.6972	0.4035	1400	1.7149
1.73	0.4611	1600	1.7091
1.7166	0.5188	1800	1.7051
1.688	0.5764	2000	1.7025
1.737	0.6341	2200	1.7010
1.7322	0.6917	2400	1.6998
1.7133	0.7493	2600	1.6994
1.6953	0.8070	2800	1.6992
1.7233	0.8646	3000	1.6990
1.733	0.9223	3200	1.6990
1.7017	0.9799	3400	1.6990

Framework versions

Transformers 4.53.2
Pytorch 2.7.1+cu126
Datasets 4.0.0
Tokenizers 0.21.2