A Lite BERT for Self-supervised Learning of Language Representations for the Persian Language
ALBERT-Persian trained on a massive amount of public corpora (
Persian Wikidumps,
MirasText) and six other manually crawled text data from a various type of websites (
BigBang Page scientific,
Chetor lifestyle,
Eligasht itinerary,
Digikala digital magazine,
Ted Talks general conversational, Books
novels, storybooks, short stories from old to the contemporary era).
Please follow the
ALBERT-Persian repo for the latest information about previous and current models.
You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to
be fine-tuned on a downstream task. See the
model hub to look for
fine-tuned versions on a task that interests you.
1from transformers import AutoConfig, AutoTokenizer, TFAutoModel
2
3config = AutoConfig.from_pretrained("m3hrdadfi/albert-fa-base-v2")
4tokenizer = AutoTokenizer.from_pretrained("m3hrdadfi/albert-fa-base-v2")
5model = TFAutoModel.from_pretrained("m3hrdadfi/albert-fa-base-v2")
6
7text = "ما در هوشواره معتقدیم با انتقال صحیح دانش و آگاهی، همه افراد میتوانند از ابزارهای هوشمند استفاده کنند. شعار ما هوش مصنوعی برای همه است."
8tokenizer.tokenize(text)
9
10>>> ['▁ما', '▁در', '▁هوش', 'واره', '▁معتقد', 'یم', '▁با', '▁انتقال', '▁صحیح', '▁دانش', '▁و', '▁اگاه', 'ی', '،', '▁همه', '▁افراد', '▁می', '▁توانند', '▁از', '▁ابزارهای', '▁هوشمند', '▁استفاده', '▁کنند', '.', '▁شعار', '▁ما', '▁هوش', '▁مصنوعی', '▁برای', '▁همه', '▁است', '.']
11
1from transformers import AutoConfig, AutoTokenizer, AutoModel
2
3config = AutoConfig.from_pretrained("m3hrdadfi/albert-fa-base-v2")
4tokenizer = AutoTokenizer.from_pretrained("m3hrdadfi/albert-fa-base-v2")
5model = AutoModel.from_pretrained("m3hrdadfi/albert-fa-base-v2")
ALBERT-Persian is the first attempt on ALBERT for the Persian Language. The model was trained based on Google's ALBERT BASE Version 2.0 over various writing styles from numerous subjects (e.g., scientific, novels, news) with more than
3.9M documents,
73M sentences, and
1.3B words, like the way we did for
ParsBERT.
Objective goals during training are as below (after 140K steps).
1***** Eval results *****
2global_step = 140000
3loss = 2.0080082
4masked_lm_accuracy = 0.6141017
5masked_lm_loss = 1.9963315
6sentence_order_accuracy = 0.985
7sentence_order_loss = 0.06908702
The following tables summarize the F1 scores obtained by ALBERT-Persian as compared to other models and architectures.
1@misc{ALBERT-Persian,
2 author = {Mehrdad Farahani},
3 title = {ALBERT-Persian: A Lite BERT for Self-supervised Learning of Language Representations for the Persian Language},
4 year = {2020},
5 publisher = {GitHub},
6 journal = {GitHub repository},
7 howpublished = {\url{https://github.com/m3hrdadfi/albert-persian}},
8}
9
10@article{ParsBERT,
11 title={ParsBERT: Transformer-based Model for Persian Language Understanding},
12 author={Mehrdad Farahani, Mohammad Gharachorloo, Marzieh Farahani, Mohammad Manthouri},
13 journal={ArXiv},
14 year={2020},
15 volume={abs/2005.12515}
16}
Post a Github issue on the
ALBERT-Persian repo.
