Views
No views yet
NoteThis is a fork of thedistilbert-base-multilingual-cased-sentiments-studentmodel. The original model card can be found here. This is just a conversion of the model to the ONNX format so it can be used in JavaScript/TypeScript applications.
Teacher model: MoritzLaurer/mDeBERTa-v3-base-mnli-xnli
Teacher hypothesis template: "The sentiment of this text is {}."
Student model: distilbert-base-multilingual-cased1from transformers import pipeline
2
3distilled_student_sentiment_classifier = pipeline(
4 model="lxyuan/distilbert-base-multilingual-cased-sentiments-student",
5 return_all_scores=True
6)
7
8# english
9distilled_student_sentiment_classifier ("I love this movie and i would watch it again and again!")
10>> [[{'label': 'positive', 'score': 0.9731044769287109},
11 {'label': 'neutral', 'score': 0.016910076141357422},
12 {'label': 'negative', 'score': 0.009985478594899178}]]
13
14# malay
15distilled_student_sentiment_classifier("Saya suka filem ini dan saya akan menontonnya lagi dan lagi!")
16[[{'label': 'positive', 'score': 0.9760093688964844},
17 {'label': 'neutral', 'score': 0.01804516464471817},
18 {'label': 'negative', 'score': 0.005945465061813593}]]
19
20# japanese
21distilled_student_sentiment_classifier("私はこの映画が大好きで、何度も見ます！")
22>> [[{'label': 'positive', 'score': 0.9342429041862488},
23 {'label': 'neutral', 'score': 0.040193185210227966},
24 {'label': 'negative', 'score': 0.025563929229974747}]]
25
261python transformers/examples/research_projects/zero-shot-distillation/distill_classifier.py \
2--data_file ./multilingual-sentiments/train_unlabeled.txt \
3--class_names_file ./multilingual-sentiments/class_names.txt \
4--hypothesis_template "The sentiment of this text is {}." \
5--teacher_name_or_path MoritzLaurer/mDeBERTa-v3-base-mnli-xnli \
6--teacher_batch_size 32 \
7--student_name_or_path distilbert-base-multilingual-cased \
8--output_dir ./distilbert-base-multilingual-cased-sentiments-student \
9--per_device_train_batch_size 16 \
10--fp161###### modify L78 to disable fast tokenizer
2default=False,
3
4###### update dataset map part at L313
5dataset = dataset.map(tokenizer, input_columns="text", fn_kwargs={"padding": "max_length", "truncation": True, "max_length": 512})
6
7###### add following lines to L213
8del model
9print(f"Manually deleted Teacher model, free some memory for student model.")
10
11###### add following lines to L337
12trainer.push_to_hub()
13tokenizer.push_to_hub("distilbert-base-multilingual-cased-sentiments-student")
14 1
2Training completed. Do not forget to share your model on huggingface.co/models =)
3
4{'train_runtime': 2009.8864, 'train_samples_per_second': 73.0, 'train_steps_per_second': 4.563, 'train_loss': 0.6473459283913797, 'epoch': 1.0}
5100%|███████████████████████████████████████| 9171/9171 [33:29<00:00, 4.56it/s]
6[INFO|trainer.py:762] 2023-05-06 10:56:18,555 >> The following columns in the evaluation set don't have a corresponding argument in `DistilBertForSequenceClassification.forward` and have been ignored: text. If text are not expected by `DistilBertForSequenceClassification.forward`, you can safely ignore this message.
7[INFO|trainer.py:3129] 2023-05-06 10:56:18,557 >> ***** Running Evaluation *****
8[INFO|trainer.py:3131] 2023-05-06 10:56:18,557 >> Num examples = 146721
9[INFO|trainer.py:3134] 2023-05-06 10:56:18,557 >> Batch size = 128
10100%|███████████████████████████████████████| 1147/1147 [08:59<00:00, 2.13it/s]
1105/06/2023 11:05:18 - INFO - __main__ - Agreement of student and teacher predictions: 88.29%
12[INFO|trainer.py:2868] 2023-05-06 11:05:18,251 >> Saving model checkpoint to ./distilbert-base-multilingual-cased-sentiments-student
13[INFO|configuration_utils.py:457] 2023-05-06 11:05:18,251 >> Configuration saved in ./distilbert-base-multilingual-cased-sentiments-student/config.json
14[INFO|modeling_utils.py:1847] 2023-05-06 11:05:18,905 >> Model weights saved in ./distilbert-base-multilingual-cased-sentiments-student/pytorch_model.bin
15[INFO|tokenization_utils_base.py:2171] 2023-05-06 11:05:18,905 >> tokenizer config file saved in ./distilbert-base-multilingual-cased-sentiments-student/tokenizer_config.json
16[INFO|tokenization_utils_base.py:2178] 2023-05-06 11:05:18,905 >> Special tokens file saved in ./distilbert-base-multilingual-cased-sentiments-student/special_tokens_map.json
17