You can deploy this model and start earning money today!

Qishuai/
distilbert_punctuator_zh

Name: distilbert_punctuator_zh
Availability: InStock

Views

No views yet

Punctuator for Simplified Chinese

The model is fine-tuned based on DistilBertForTokenClassification for adding punctuations to plain text (simplified Chinese). The model is fine-tuned based on distilled model bert-base-chinese.

Usage

python

from transformers import DistilBertForTokenClassification, DistilBertTokenizerFast

model = DistilBertForTokenClassification.from_pretrained("Qishuai/distilbert_punctuator_zh")
tokenizer = DistilBertTokenizerFast.from_pretrained("Qishuai/distilbert_punctuator_zh")

Model Overview

Training data

Combination of following three dataset:

News articles of People's Daily 2014. Reference

Model Performance

Validation with MSRA training dataset. Reference

Metrics Report:

	precision	recall	f1-score	support
C_COMMA	0.67	0.59	0.63	91566
C_DUNHAO	0.50	0.37	0.42	21013
C_EXLAMATIONMARK	0.23	0.06	0.09	399
C_PERIOD	0.84	0.99	0.91	44258
C_QUESTIONMARK	0.00	1.00	0.00	0
micro avg	0.71	0.67	0.69	157236
macro avg	0.45	0.60	0.41	157236
weighted avg	0.69	0.67	0.68	157236