Hyperparameter	Value
SFTConfig
`torch_dtype`	`bfloat16`
`seed`	`42`
`epoch`	`3`
`per_device_train_batch_size`	`2`
`per_device_eval_batch_size`	`2`
`learning_rate`	`0.0002`
`lr_scheduler_type`	`"linear"`
`max_grad_norm`	`1.0`
`neftune_noise_alpha`	`None`
`gradient_accumulation_steps`	`1`
`gradient_checkpointing`	`False`
`max_seq_length`	`1024`
LoraConfig
`r`	`16`
`lora_alpha`	`16`
`lora_dropout`	`0.1`
`target_modules`	`["q_proj", "v_proj"]`

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "jjae/Midm-KCulture-2.0-Base-Instruct"
model = AutoModelForCausalLM.from_pretrained(
      model_name,
      torch_dtype=torch.bfloat16,
      trust_remote_code=True,
      device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)