CrisisTransformers were evaluated on 18 public crisis-specific datasets against strong baselines. Our pre-trained models outperform the baselines across all 18 datasets in classification tasks, and our best-performing sentence-encoder (mono-lingual) outperforms the state-of-the-art by more than 17% in sentence encoding tasks. The multi-lingual sentence encoders (support 50+ languages; see associated paper) are designed to approximate the embedding space of the best-performing mono-lingual sentence encoder.
Uses
CrisisTransformers has 8 pre-trained models, 1 mono-lingual and 2 multi-lingual sentence encoders. The pre-trained models should be finetuned for downstream tasks just like BERT and RoBERTa. The sentence encoders can be used out-of-the-box just like Sentence-Transformers for sentence encoding to facilitate tasks such as semantic search, clustering, topic modelling.
Models and naming conventions
CT-M1 models were trained from scratch up to 40 epochs, while CT-M2 models were initialized with pre-trained RoBERTa's weights and CT-M3 models were initialized with pre-trained BERTweet's weights and both trained for up to 20 epochs. OneLook represents the checkpoint after 1 epoch, BestLoss represents the checkpoint with the lowest loss during training, and Complete represents the checkpoint after completing all epochs. SE represents sentence encoder.
Languages supported by the multi-lingual sentence encoders: Albanian, Arabic, Armenian, Bulgarian, Catalan, Chinese (Simplified), Chinese (Traditional), Croatian, Czech, Danish, Dutch, Estonian, Finnish, French, French (Canada), Galician, Georgian, German, Greek, Gujarati, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Kurdish (Sorani), Latvian, Lithuanian, Macedonian, Malay, Marathi, Mongolian, Myanmar (Burmese), Norwegian, Persian, Polish, Portuguese, Portuguese (Brazil), Romanian, Russian, Serbian, Slovak, Slovenian, Spanish, Swedish, Thai, Turkish, Ukrainian, Urdu, and Vietnamese.
Citation
If you use CrisisTransformers and the mono-lingual sentence encoder, please cite the following paper:
@article{lamsal2023crisistransformers,
title={CrisisTransformers: Pre-trained language models and sentence encoders for crisis-related social media texts},
author={Rabindra Lamsal and
Maria Rodriguez Read and
Shanika Karunasekera},
journal={Knowledge-Based Systems},
pages={111916},
year={2024},
publisher={Elsevier}
}
If you use the multi-lingual sentence encoders, please cite the following paper:
@article{lamsal2024semantically,
title={Semantically Enriched Cross-Lingual Sentence Embeddings for Crisis-related Social Media Texts},
author={Rabindra Lamsal and
Maria Rodriguez Read and
Shanika Karunasekera},
year={2024},
eprint={2403.16614},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
