Luxembourgish adaptation of sentence-transformers/paraphrase-multilingual-mpnet-base-v2
This is a sentence-transformers model finetuned from sentence-transformers/paraphrase-multilingual-mpnet-base-v2 further adapted to support Historical and Contemporary Luxembourgish. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for (cross-lingual) semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
Model Details
This model is specialised to perform cross-lingual semantic search to and from Historical/Contemporary Luxembourgish. This model would be particularly useful for libraries and archives that want to perform semantic search and longitudinal studies within their collections.
This model only supports inputs of up to 128 subtokens long.
We also release a model that performs better (7.5pp) on Historical Bitext Mining and natively supports long context (8192 subtokens). For most usecases we reccomend using histlux-gte-multilingual-base
However, this model exhibits superior performance (by 18pp) on the adversarial paraphrase discrimination task ParaLUX.
1from sentence_transformers import SentenceTransformer
23# Download from the 🤗 Hub4model = SentenceTransformer("impresso-project/histlux-paraphrase-multilingual-mpnet-base-v2")5# Run inference6sentences =[7'The cross-border workers should also receive more wages.',8"D'grenzarbechetr missten och me' lo'n kre'en.",9"De Néckel: Firun! Dât ass jo ailes, wèll 't get dach neischt un der Bréck gemâcht!",10]11embeddings = model.encode(sentences)12print(embeddings.shape)13# [3, 768]1415# Get the similarity scores for the embeddings16similarities = model.similarity(embeddings, embeddings)17print(similarities.shape)18# [3, 3]
Evaluation
Metrics
(see introducing paper)
Historical Bitext Mining (Accuracy):
LB -> FR: 88.6
FR -> LB: 90.0
LB -> EN: 88.7
EN -> LB: 90.4
LB -> DE: 91.1
DE -> LB: 91.8
Contemporary LB (Accuracy):
ParaLUX: 80.5
SIB-200(LB): 59.4
Training Details
Training Dataset
LB-EN (Historical, Modern)
Dataset: lb-en (mixed)
Size: 40,000 training samples
Columns: english, luxembourgish, and label (teacher's en embeddings)
Approximate statistics based on the first 1000 samples:
Adapting Multilingual Embedding Models to Historical Luxembourgish (introducing paper)
bibtex
1@misc{michail2025adaptingmultilingualembeddingmodels,
2 title={Adapting Multilingual Embedding Models to Historical Luxembourgish},
3 author={Andrianos Michail and Corina Julia Raclé and Juri Opitz and Simon Clematide},
4 year={2025},
5 eprint={2502.07938},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL},
8 url={https://arxiv.org/abs/2502.07938},
9}
Multilingual Knowledge Distillation
bibtex
1@inproceedings{reimers-2020-multilingual-sentence-bert,
2 title = "Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation",
3 author = "Reimers, Nils and Gurevych, Iryna",
4 booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing",
5 month = "11",
6 year = "2020",
7 publisher = "Association for Computational Linguistics",
8 url = "https://arxiv.org/abs/2004.09813",
9}
About Impresso
Impresso project
Impresso - Media Monitoring of the Past is an interdisciplinary research project that aims to develop and consolidate tools for processing and exploring large collections of media archives across modalities, time, languages and national borders. The first project (2017-2021) was funded by the Swiss National Science Foundation under grant No. CRSII5_173719 and the second project (2023-2027) by the SNSF under grant No. CRSII5_213585 and the Luxembourg National Research Fund under grant No. 17498891.
