This model is a Korean-specific model trained in phi-2 by adding a Korean tokenizer and Korean data. (English is also available.) Although phi-2 performs very well, it does not support the Korean language and does not have a tokenizer trained on Korean corpous, so tokenizing Korean text will use many times more tokens than English tokens.

To overcome these limitations, I trained the model using an open-license Korean corpus and some English corpus. The reasons for using the English corpus together are as follows: 1. The goal is to preserve the excellent performance of the existing model by preventing catastrophic forgetting. 2. Mixing English and Korean prompts usually produces better results than using all prompts in Korean.

Since my role is not as a working developer, but as an solutions architect helping customers with quick PoCs/prototypes, and I was limited by AWS GPU resources available, I only trained with 5GB of data instead of hundreds of GB of massive data.

The dataset used for training is as follows. To prevent catastrophic forgetting, I included some English corpus as training data.

Note that performance is not guaranteed since only a small number of datasets were used for the experiment. The number of samples for training set is just around 5 million after tokenization. For distributed training, all weights were trained without adapter techniques, and sharding parallelization was performed with ZeRO-2. The presets are as follows.

Since this is a model that has not been fine-tuned, it is recommended to perform fine tuning such as instruction tuning/alignment tuning according to your use case.

Some hyperparameters are listed below.

batch_size: 2
num_epochs: 1
learning_rate: 3e-4
gradient_accumulation_steps: 8
lr_scheduler_type: "linear"
group_by_length: False

cc-by-sa 3.0; The license of phi-2 is MIT, but I considered the licensing of the dataset used for training.

This model was created as a personal experiment, unrelated to the organization I work for. The model may not operate correctly because separate verification was not performed. Please be careful unless it is for personal experimentation or PoC (Proof of Concept)!