You can deploy this model and start earning money today!

bonadossou/
multitask_model_fon_False_multiplicative

Views

No views yet

Paper: FonMTL: Toward Building a Multi-Task Learning Model for Fon Language, accepted at WiNLP co-located at EMNLP 2023
Official Github: https://github.com/bonaventuredossou/multitask_fon
Build Multi-task Learning Model: For the shared layers (encoders) we used the following language model heads:
- AfroLM-Large
  - AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages (Dossou et.al., EMNLP 2022)
- XLMR-Large:
  - Unsupervised Cross-lingual Representation Learning at Scale (Conneau et.al., ACL 2020)
Evaluation:
- The goal primarily is to explore whether multitask learning improves performance on downstream tasks for Fon. We try two settings: (a) training only on Fon and evaluating on Fon, (b) training on all languages and evaluating on Fon. We evaluate the multi-task learning model on NER and POS tasks, and compare it with baselines (models finetuned and evaluated on single tasks)

How to get started

This command will:

Model	Task	Pretraining/Finetuning Dataset	Pretraining/Finetuning Language(s)	Evaluation Dataset	Metric	Metric's Value
`AfroLM-Large`	Single Task	MasakhaNER 2.0	All	FON NER	F1-Score	80.48
`AfriBERTa-Large`	Single Task	MasakhaNER 2.0	All	FON NER	F1-Score	79.90
`XLMR-Base`	Single Task	MasakhaNER 2.0	All	FON NER	F1-Score	81.90
`XLMR-Large`	Single Task	MasakhaNER 2.0	All	FON NER	F1-Score	81.60
`AfroXLMR-Base`	Single Task	MasakhaNER 2.0	All	FON NER	F1-Score	82.30
`AfroXLMR-Large`	Single Task	MasakhaNER 2.0	All	FON NER	F1-Score	82.70
:---:	:---:	:---:	:---:	:---:	:---:
`MTL Sum (ours)`	Multi-Task	MasakhaNER 2.0 & MasakhaPOS	All	FON NER	F1-Score	79.87
`MTL Weighted (ours)`	Multi-Task	MasakhaNER 2.0 & MasakhaPOS	All	FON NER	F1-Score	81.92
`MTL Weighted (ours)`	Multi-Task	MasakhaNER 2.0 & MasakhaPOS	Fon Data	FON NER	F1-Score	64.43

Model	Task	Pretraining/Finetuning Dataset	Pretraining/Finetuning Language(s)	Evaluation Dataset	Metric	Metric's Value
`AfroLM-Large`	Single Task	MasakhaPOS	All	FON POS	Accuracy	82.40
`AfriBERTa-Large`	Single Task	MasakhaPOS	All	FON POS	Accuracy	88.40
`XLMR-Base`	Single Task	MasakhaPOS	All	FON POS	Accuracy	90.10
`XLMR-Large`	Single Task	MasakhaPOS	All	FON POS	Accuracy	90.20
`AfroXLMR-Base`	Single Task	MasakhaPOS	All	FON POS	Accuracy	90.10
`AfroXLMR-Large`	Single Task	MasakhaPOS	All	FON POS	Accuracy	90.40
:---:	:---:	:---:	:---:	:---:	:---:
`MTL Sum (ours)`	Multi-Task	MasakhaNER 2.0 & MasakhaPOS	All	FON POS	Accuracy	82.45
`MTL Weighted (ours)`	Multi-Task	MasakhaNER 2.0 & MasakhaPOS	All	FON POS	Accuracy	89.20
`MTL Weighted (ours)`	Multi-Task	MasakhaNER 2.0 & MasakhaPOS	Fon Data	FON POS	Accuracy	80.85

Merging Type	Models	Task	Metric	Metric's Value
Multiplicative	MTL Weighted (multi-task; ours; *)	NER	F1-Score	81.92
Multiplicative	MTL Weighted (multi-task; ours; +)	NER	F1-Score	64.43
:---:	:---:	:---:	:---:	:---:
Multiplicative	MTL Weighted (multi-task; ours; *)	POS	Accuracy	89.20
Multiplicative & MTL Weighted (multi-task; ours; +)	POS	Accuracy	80.85
:---:	:---:	:---:	:---:	:---:
Additive	MTL Weighted (multi-task; ours; *)	NER	F1-Score	78.91
Additive	MTL Weighted (multi-task; ours; +)	NER	F1-Score	60.93
:---:	:---:	:---:	:---:	:---:
Additive	MTL Weighted (multi-task; ours; *)	POS	Accuracy	86.99
Additive	MTL Weighted (multi-task; ours; +)	POS	Accuracy	78.25

multitask_model_fon_False_multiplicative.bin is the MTL Fon Model which has been pre-trained on all MasakhaNER 2.0 and MasakhaPOS datasets, and merging representations in a multiplicative way.
multitask_model_fon_True_multiplicative.bin is the MTL Fon Model which has been pre-trained only on Fon data from the MasakhaNER 2.0 and MasakhaPOS datasets, and merging representations in a multiplicative way.

To run inference with the model(s), you can use the testing block defined in our MultitaskFON class.