mBART50 XL-Sum
mbart50-xlsum is a multilingual abstractive news summarization model fine-tuned from
facebook/mbart-large-50-many-to-many-mmt on 15 language subsets of
GEM/xlsum. It was developed as part of a multilingual news aggregation and summarization project.
This checkpoint is the first of two mBART-50 training variants produced by the project. It uses a standard multilingual mixture, a 96-token target limit, and a compute-conscious two-epoch training setup. Compared with the second variant, it is slightly more source-grounded in the project's automatic diagnostics, while producing very similar reference-overlap scores.
Related Project
This model was developed as part of
samurAI, a multilingual news aggregation and summarization system with RSS ingestion, local transformer summarization, SQLite storage, a Flask web interface, JSON APIs, and evaluation notebooks.
Model Details
| Property | Value |
|---|
| Task | Multilingual abstractive news summarization |
| Base model | facebook/mbart-large-50-many-to-many-mmt |
| Architecture | MBartForConditionalGeneration |
| Encoder / decoder layers | 12 / 12 |
| Attention heads | 16 |
| Hidden size | 1,024 |
| Vocabulary size | 250,054 |
| Maximum model positions | 1,024 |
| Fine-tuning source length | 512 tokens |
| Fine-tuning target length | 96 tokens |
| Framework | PyTorch / Transformers |
Languages
The fine-tuning mixture contains the following 15 languages:
| Language | mBART code | Language | mBART code |
|---|
| Arabic | ar_AR | Chinese (Simplified) | zh_CN |
| English | en_XX | French | fr_XX |
| Hindi | hi_IN | Indonesian | id_ID |
| Japanese | ja_XX | Korean | ko_KR |
| Persian | fa_IR | Portuguese | pt_XX |
| Russian | ru_RU | Spanish | es_XX |
| Turkish | tr_TR | Ukrainian | uk_UA |
| Vietnamese | vi_VN | | |
Quantitative evaluation was completed for 11 of these languages: Arabic, Chinese (Simplified), English, French, Hindi, Japanese, Korean, Russian, Spanish, Turkish, and Vietnamese. Results for Indonesian, Persian, Portuguese, and Ukrainian are not reported in the final common evaluation, so performance claims should not be extrapolated to them.
Intended Use
The model is intended for:
- summarizing news articles in a supported source language;
- research on multilingual abstractive summarization;
- comparing multilingual training and language-balancing strategies;
- integration into news-processing pipelines with human review.
The model is not intended to be a factual authority, to make editorial decisions autonomously, or to summarize high-stakes medical, legal, financial, or emergency information without verification against the source.
Training Data
The checkpoint was trained on
GEM/xlsum, a multilingual collection of BBC news articles paired with short reference summaries. Examples from the 15 selected language subsets were combined, shuffled, and annotated with their corresponding mBART language codes.
The final benchmark below is separate from these training-time subsets and uses the full executable test subsets from
csebuetnlp/xlsum.
| Split | Examples used |
|---|
| Train | 818,129 |
| Validation | 3,000 (up to 200 per language) |
| Test | 3,000 (up to 200 per language) |
Training Procedure
| Hyperparameter | Value |
|---|
| Epochs | 2 |
| Learning rate | 3e-5 |
| Per-device training batch size | 32 |
| Gradient accumulation | 8 |
| Effective batch size | 256 |
| Weight decay | 0.01 |
| Warmup ratio | 0.03 |
| Precision | bfloat16 when CUDA was available |
| Maximum source length | 512 |
| Maximum target length | 96 |
| Seed | 42 |
Generation-based validation was disabled during training to control compute cost, and no best-checkpoint selection was performed. Training completed 6,390 optimization steps with a final reported training loss of 2.2078 over approximately 4 hours 44 minutes on CUDA hardware.
Evaluation
Protocol
The final evaluation used the complete available test subsets of csebuetnlp/xlsum for 11 languages, totaling 52,219 examples per model. No per-language sample cap was applied. Scores below are macro-averaged across languages and reported on a 0-1 scale.
Articles were truncated to 512 tokens. Evaluation used batch size 8 and mixed precision on CUDA. The common mBART decoding configuration was:
1max_length=90
2min_length=24
3num_beams=5
4length_penalty=1.1
5no_repeat_ngram_size=3
6repetition_penalty=1.05
7early_stopping=True
Turkish used a language-specific configuration with max_length=84, min_length=26, 6 beams, length penalty 1.3, 4-gram repetition blocking, and repetition penalty 1.2. The target language token was forced for every language.
Overall Results
| Metric | Score |
|---|
| ROUGE-1 F1 | 0.3559 |
| ROUGE-2 F1 | 0.1735 |
| ROUGE-L F1 | 0.2785 |
| BLEU | 0.1202 |
| METEOR-lite | 0.3157 |
METEOR-lite is the project's lightweight exact-token implementation and must not be interpreted as the canonical METEOR package score.
Results by Language
| Language | Samples | ROUGE-1 | ROUGE-2 | ROUGE-L | BLEU | METEOR-lite |
|---|
| Arabic | 4,689 | 0.2674 | 0.1104 | 0.2280 | 0.0726 | 0.2249 |
| English | 11,535 | 0.3489 | 0.1333 | 0.2769 | 0.0957 | 0.3024 |
| Spanish | 4,763 | 0.3032 | 0.1013 | 0.2251 | 0.0675 | 0.2523 |
| French | 1,086 | 0.3121 | 0.1345 | 0.2504 | 0.0953 | 0.2678 |
| Hindi | 8,847 | 0.5677 | 0.2905 | 0.4052 | 0.2122 | 0.5301 |
| Japanese | 889 | 0.4997 | 0.3074 | 0.3669 | 0.2133 | 0.4444 |
| Korean | 550 | 0.4295 | 0.2493 | 0.3429 | 0.1817 | 0.4038 |
| Russian | 7,780 | 0.2401 | 0.0926 | 0.1996 | 0.0630 | 0.1932 |
| Turkish | 3,397 | 0.2291 | 0.0983 | 0.2065 | 0.0704 | 0.1991 |
| Vietnamese | 4,013 | 0.3283 | 0.1575 | 0.2507 | 0.0947 | 0.2895 |
| Chinese (Simplified) | 4,670 | 0.3884 | 0.2335 | 0.3111 | 0.1552 | 0.3654 |
Supplementary Diagnostics
These diagnostics are included to describe model behavior, not as established factuality or human-quality measures.
| Diagnostic | Value |
|---|
| Mean latency per example | 0.1917 s |
| Mean compression ratio | 23.2527 |
| Mean source coverage | 0.8481 |
| Mean novel bigram ratio | 0.4229 |
| Mean repeated trigram ratio | 0.0039 |
| Mean generated summary length | 31.17 tokens |
| Automatic completeness proxy | 0.4525 |
| Automatic factuality proxy | 0.8522 |
Latency was measured in one CUDA evaluation environment and is not portable across hardware, software versions, or batch configurations. The completeness and factuality values are project-specific automatic proxies; they do not prove semantic completeness or factual correctness.
Usage
1import torch
2from transformers import MBart50TokenizerFast, MBartForConditionalGeneration
3
4model_id = "mskayacioglu/mbart50-xlsum"
5language_code = "tr_TR"
6
7tokenizer = MBart50TokenizerFast.from_pretrained(model_id)
8model = MBartForConditionalGeneration.from_pretrained(model_id)
9model.eval()
10
11device = "cuda" if torch.cuda.is_available() else "cpu"
12model.to(device)
13
14article = "Özetlenecek Türkçe haber metni buraya yazılır."
15tokenizer.src_lang = language_code
16
17inputs = tokenizer(
18 article,
19 return_tensors="pt",
20 max_length=512,
21 truncation=True,
22).to(device)
23
24with torch.inference_mode():
25 output_ids = model.generate(
26 **inputs,
27 forced_bos_token_id=tokenizer.lang_code_to_id[language_code],
28 max_length=84,
29 min_length=26,
30 num_beams=6,
31 length_penalty=1.3,
32 no_repeat_ngram_size=4,
33 repetition_penalty=1.2,
34 early_stopping=True,
35 )
36
37summary = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0]
38print(summary)
For languages other than Turkish, the evaluation configuration used max_length=90, min_length=24, 5 beams, length penalty 1.1, 3-gram repetition blocking, and repetition penalty 1.05.
Limitations and Risks
- The model can hallucinate facts, names, dates, quantities, or causal relationships.
- Inputs longer than 512 tokens were truncated during fine-tuning and evaluation; important information near the end of long articles may be omitted.
- XL-Sum is based on BBC news. Performance may decline on other publishers, informal writing, specialist domains, or structurally noisy web text.
- Automatic overlap metrics reward lexical similarity and do not establish factual consistency or summary usefulness.
- Evaluation coverage is uneven across languages and includes only 11 languages in the final common benchmark.
- The training mixture is imbalanced across languages, so high-resource languages can influence the model more strongly.
- The model may reproduce social, geographic, and editorial biases present in its pretrained parameters and news data.
- Always verify generated summaries against the original article before publication or high-stakes use.
License
XL-Sum is distributed under CC BY-NC-SA 4.0. This model card therefore declares the same license for the fine-tuned checkpoint. Users are responsible for checking the terms of the base model and dataset and for ensuring that their intended use is permitted. In particular, the dataset license restricts commercial use.
Citation
If you use the model, cite the XL-Sum dataset and mBART-50:
1@inproceedings{hasan-etal-2021-xl,
2 title = {{XL-Sum}: Large-Scale Multilingual Abstractive Summarization for 44 Languages},
3 author = {Hasan, Tahmid and Bhattacharjee, Abhik and Islam, Md. Saiful and Samin, Kazi and Li, Yuan-Fang and Kang, Yong-Bin and Rahman, M. Sohel and Shahriyar, Rifat},
4 booktitle = {Findings of ACL-IJCNLP 2021},
5 year = {2021},
6 pages = {4693--4703},
7 eprint = {2106.13822},
8 archivePrefix = {arXiv}
9}
10
11@article{tang2020multilingual,
12 title = {Multilingual Translation with Extensible Multilingual Pretraining and Finetuning},
13 author = {Tang, Yuqing and Tran, Chau and Li, Xian and Chen, Peng-Jen and Goyal, Naman and Chaudhary, Vishrav and Gu, Jiatao and Fan, Angela},
14 journal = {arXiv preprint arXiv:2008.00401},
15 year = {2020},
16 eprint = {2008.00401},
17 archivePrefix = {arXiv}
18}
Model Card Author
Mert Samet Kayacıoğlu
