KR-BERT-MEDIUM
A pretrained Korean-specific BERT model developed by Computational Linguistics Lab at Seoul National University.
It is based on our character-level
KR-BERT model which utilize WordPiece tokenizer.
Here, the model name has a suffix 'MEDIUM' since its training data grew from KR-BERT's original dataset. We have another additional model, KR-BERT-EXPANDED with more extensive training data expanded from those of KR-BERT-MEDIUM, so the suffix 'MEDIUM' is used.
Vocab, Parameters and Data
| Mulitlingual BERT
(Google) | KorBERT
(ETRI) | KoBERT
(SKT) | KR-BERT character | KR-BERT-MEDIUM |
|---|
| vocab size | 119,547 | 30,797 | 8,002 | 16,424 | 20,000 |
| parameter size | 167,356,416 | 109,973,391 | 92,186,880 | 99,265,066 | 102,015,010 |
| data size | -
(The Wikipedia data
for 104 languages) | 23GB
4.7B morphemes | -
(25M sentences,
233M words) | 2.47GB
20M sentences,
233M words | 12.37GB
91M sentences,
1.17B words |
The training data for this model is expanded from those of KR-BERT, texts from Korean Wikipedia, and news articles, by addition of legal texts crawled from the National Law Information Center and
Korean Comments dataset. This data expansion is to collect texts from more various domains than those of KR-BERT. The total data size is about 12.37GB, consisting of 91M and 1.17B words.
The user-generated comment dataset is expected to have similar stylistic properties to the task datasets of NSMC and HSD. Such text includes abbreviations, coinages, emoticons, spacing errors, and typos. Therefore, we added the dataset containing such on-line properties to our existing formal data such as news articles and Wikipedia texts to compose the training data for KR-BERT-MEDIUM. Accordingly, KR-BERT-MEDIUM reported better results in sentiment analysis than other models, and the performances improved with the model of the more massive, more various training data.
This model’s vocabulary size is 20,000, whose tokens are trained based on the expanded training data using the WordPiece tokenizer.
KR-BERT-MEDIUM is trained for 2M steps with the maxlen of 128, training batch size of 64, and learning rate of 1e-4, taking 22 hours to train the model using a Google Cloud TPU v3-8.
Models
TensorFlow
- BERT tokenizer, character-based model (download)
PyTorch
- You can import it from Transformers!
1# pytorch, transformers
2
3from transformers import AutoTokenizer, AutoModel
4
5tokenizer = AutoTokenizer.from_pretrained("snunlp/KR-Medium", do_lower_case=False)
6
7model = AutoModel.from_pretrained("snunlp/KR-Medium")
8
Requirements
- transformers == 4.0.0
- tensorflow < 2.0
Downstream tasks
- Movie Review Classification on Naver Sentiment Movie Corpus (NSMC)
- Hate Speech Detection (Moon et al., 2020)
tensorflow
- After downloading our pre-trained models, put them in a
models directory.
- Set the output directory (for fine-tuning)
- Select task name:
NSMC for Movie Review Classification, and HATE for Hate Speech Detection
1# tensorflow
2python3 run_classifier.py \
3 --task_name={NSMC, HATE} \
4 --do_train=true \
5 --do_eval=true \
6 --do_predict=true \
7 --do_lower_case=False\
8 --max_seq_length=128 \
9 --train_batch_size=128 \
10 --learning_rate=5e-05 \
11 --num_train_epochs=5.0 \
12 --output_dir={output_dir}
Performances
TensorFlow, test set performances
| multilingual BERT | KorBERT
character | KR-BERT
character
WordPiece | KR-BERT-MEDIUM |
|---|
| NSMC (Acc) | 86.82 | 89.81 | 89.74 | 90.29 |
| Hate Speech (F1) | 52.03 | 54.33 | 54.53 | 57.91 |
Contacts
