Dhee-Vagmi-Core — v4

Why this model exists

What "better" means

1. QUALITY. chrF++ macro-averaged over ALL 380 directions -- an unweighted mean, so a data-starved direction cannot hide behind a well-resourced one -- within reach of NLLB-200-distilled-600M, the model it replaces. Non-English directions are held to the same bar as English ones from v4 on: they are trained explicitly now, not reached zero-shot, so a weak Tamil<->Bengali is a result and no longer an excuse.
2. LICENCE. Every training corpus must be commercially usable (Samanantar is CC-BY). A single CC-BY-NC or GPL source contaminates the point of the model and fails the run regardless of the score.
3. ON-DEVICE. Exports to INT8 ONNX and runs under onnxruntime-mobile at a size comparable to the current NLLB bundle. A checkpoint that scores well but will not quantise/export is not shippable and has not succeeded.

This version

• Run: 18c59747-6bd9-4ccd-b718-c08072654e6c
• Config: {"version": "v4", "max_steps": 140000, "grad_accum": 32, "pin_worker": "17b2c978-23e2-4dc0-a624-ab28982f240b", "vocab_size": 64000, "corpus_from": "/home/dheeyantra/vagmi-text/corpus20", "batch_tokens": 32, "min_vram_mib": 13000, "corpus_sha256": "860ce033e13f72114db5cbb8655ec72241e640936e16ab9c143f4f25c3879c1d", "vram_fraction": 0.5}
• Eval: {"BLEU": 34.53, "best": "eng_Latn-spa_Latn", "beams": 5, "worst": "eng_Latn-tam_Taml", "chrF++": 53.49, "directions": 380, "chrF++_en_xx": 51.1, "chrF++_xx_en": 55.51, "per_direction": {"arb_Arab-asm_Beng": {"n": 100, "BLEU": 22.41, "chrF++": 42.2}, "arb_Arab-ben_Beng": {"n": 100, "BLEU": 23.19, "chrF++": 44.58}, "arb_Arab-deu_Latn": {"n": 100, "BLEU": 45.56, "chrF++": 64.57}, "arb_Arab-eng_Latn": {"n": 100, "BLEU": 57.7, "chrF++": 71.85}, "arb_Arab-fra_Latn": {"n": 100, "BLEU": 52.64, "chrF++": 68.55}, "arb_Arab-guj_Gujr": {"n": 100, "BLEU": 31.58, "chrF++": 51.77}, "arb_Arab-hin_Deva": {"n": 100, "BLEU": 33.86, "chrF++": 50.16}, "arb_Arab-jpn_Jpan": {"n": 100, "BLEU": 46.11, "chrF++": 58.81}, "arb_Arab-kan_Knda": {"n": 100, "BLEU": 27.08, "chrF++": 51.72}, "arb_Arab-mal_Mlym": {"n": 100, "BLEU": 29.98, "chrF++": 49.07}, "arb_Arab-mar_Deva": {"n": 100, "BLEU": 26.75, "chrF++": 48.17}, "arb_Arab-ory_Orya": {"n": 100, "BLEU": 27.2, "chrF++": 50.3}, "arb_Arab-pan_Guru": {"n": 100, "BLEU": 33.17, "chrF++": 49.34}, "arb_Arab-por_Latn": {"n": 100, "BLEU": 54.03, "chrF++": 69.51}, "arb_Arab-rus_Cyrl": {"n": 100, "BLEU": 45.79, "chrF++": 63.54}, "arb_Arab-spa_Latn": {"n": 100, "BLEU": 55.36, "chrF++": 71.33}, "arb_Arab-tam_Taml": {"n": 100, "BLEU": 23.55, "chrF++": 46.4}, "arb_Arab-tel_Telu": {"n": 100, "BLEU": 18.68, "chrF++": 42.48}, "arb_Arab-zho_Hans": {"n": 100, "BLEU": 41.82, "chrF++": 47.86}, "asm_Beng-arb_Arab": {"n": 100, "BLEU": 37.61, "chrF++": 55.04}, "asm_Beng-ben_Beng": {"n": 100, "BLEU": 27.89, "chrF++": 46.24}, "asm_Beng-deu_Latn": {"n": 100, "BLEU": 35.09, "chrF++": 56.22}, "asm_Beng-eng_Latn": {"n": 100, "BLEU": 21.21, "chrF++": 37.1}, "asm_Beng-fra_Latn": {"n": 100, "BLEU": 40.74, "chrF++": 59.25}, "asm_Beng-guj_Gujr": {"n": 100, "BLEU": 29.45, "chrF++": 48.32}, "asm_Beng-hin_Deva": {"n": 100, "BLEU": 29.52, "chrF++": 46.11}, "asm_Beng-jpn_Jpan": {"n": 100, "BLEU": 36.02, "chrF++": 48.65}, "asm_Beng-kan_Knda": {"n": 100, "BLEU": 25.64, "chrF++": 48.9}, "asm_Beng-mal_Mlym": {"n": 100, "BLEU": 23.98, "chrF++": 44.09}, "asm_Beng-mar_Deva": {"n": 100, "BLEU": 27.38, "chrF++": 47.5}, "asm_Beng-ory_Orya": {"n": 100, "BLEU": 26.27, "chrF++": 47.55}, "asm_Beng-pan_Guru": {"n": 100, "BLEU": 27.37, "chrF++": 44.78}, "asm_Beng-por_Latn": {"n": 100, "BLEU": 40.37, "chrF++": 57.7}, "asm_Beng-rus_Cyrl": {"n": 100, "BLEU": 31.72, "chrF++": 51.68}, "asm_Beng-spa_Latn": {"n": 100, "BLEU": 41.78, "chrF++": 60.59}, "asm_Beng-tam_Taml": {"n": 100, "BLEU": 21.65, "chrF++": 45.32}, "asm_Beng-tel_Telu": {"n": 100, "BLEU": 18.16, "chrF++": 42.37}, "asm_Beng-zho_Hans": {"n": 100, "BLEU": 34.98, "chrF++": 41.01}, "ben_Beng-arb_Arab": {"n": 100, "BLEU": 40.41, "chrF++": 53.84}, "ben_Beng-asm_Beng": {"n": 100, "BLEU": 24.24, "chrF++": 43.07}, "ben_Beng-deu_Latn": {"n": 100, "BLEU": 36.26, "chrF++": 53.98}, "ben_Beng-eng_Latn": {"n": 100, "BLEU": 21.2, "chrF++": 43.7}, "ben_Beng-fra_Latn": {"n": 100, "BLEU": 40.6, "chrF++": 58.06}, "ben_Beng-guj_Gujr": {"n": 100, "BLEU": 28.76, "chrF++": 46.99}, "ben_Beng-hin_Deva": {"n": 100, "BLEU": 28.63, "chrF++": 44.88}, "ben_Beng-jpn_Jpan": {"n": 100, "BLEU": 29.42, "chrF++": 47.02}, "ben_Beng-kan_Knda": {"n": 100, "BLEU": 24.24, "chrF++": 48.01}, "ben_Beng-mal_Mlym": {"n": 100, "BLEU": 29.48, "chrF++": 47.84}, "ben_Beng-mar_Deva": {"n": 100, "BLEU": 26.19, "chrF++": 44.97}, "ben_Beng-ory_Orya": {"n": 100, "BLEU": 23.26, "chrF++": 44.67}, "ben_Beng-pan_Guru": {"n": 100, "BLEU": 33.28, "chrF++": 48.55}, "ben_Beng-por_Latn": {"n": 100, "BLEU": 41.07, "chrF++": 58.2}, "ben_Beng-rus_Cyrl": {"n": 100, "BLEU": 35.72, "chrF++": 53.17}, "ben_Beng-spa_Latn": {"n": 100, "BLEU": 41.99, "chrF++": 59.32}, "ben_Beng-tam_Taml": {"n": 100, "BLEU": 22.48, "chrF++": 44.97}, "ben_Beng-tel_Telu": {"n": 100, "BLEU": 19.67, "chrF++": 41.55}, "ben_Beng-zho_Hans": {"n": 100, "BLEU": 33.26, "chrF++": 40.18}, "deu_Latn-arb_Arab": {"n": 100, "BLEU": 49.16, "chrF++": 66.39}, "deu_Latn-asm_Beng": {"n": 100, "BLEU": 22.92, "chrF++": 44.15}, "deu_Latn-ben_Beng": {"n": 100, "BLEU": 24.93, "chrF++": 46.71}, "deu_Latn-eng_Latn": {"n": 100, "BLEU": 60.18, "chrF++": 74.48}, "deu_Latn-fra_Latn": {"n": 100, "BLEU": 58.31, "chrF++": 73.51}, "deu_Latn-guj_Gujr": {"n": 100, "BLEU": 31.74, "chrF++": 52.51}, "deu_Latn-hin_Deva": {"n": 100, "BLEU": 38.23, "chrF++": 54.69}, "deu_Latn-jpn_Jpan": {"n": 100, "BLEU": 41.87, "chrF++": 58.77}, "deu_Latn-kan_Knda": {"n": 100, "BLEU": 28.56, "chrF++": 54.09}, "deu_Latn-mal_Mlym": {"n": 100, "BLEU": 30.98, "chrF++": 50.12}, "deu_Latn-mar_Deva": {"n": 100, "BLEU": 28.21, "chrF++": 51.01}, "deu_Latn-ory_Orya": {"n": 100, "BLEU": 28.2, "chrF++": 52.43}, "deu_Latn-pan_Guru": {"n": 100, "BLEU": 34.92, "chrF++": 51.98}, "deu_Latn-por_Latn": {"n": 100, "BLEU": 59.85, "chrF++": 72.55}, "deu_Latn-rus_Cyrl": {"n": 100, "BLEU": 47.05, "chrF++": 65.84}, "deu_Latn-spa_Latn": {"n": 100, "BLEU": 60.02, "chrF++": 74.6}, "deu_Latn-tam_Taml": {"n": 100, "BLEU": 22.94, "chrF++": 46.74}, "deu_Latn-tel_Telu": {"n": 100, "BLEU": 20.22, "chrF++": 47.44}, "deu_Latn-zho_Hans": {"n": 100, "BLEU": 45.35, "chrF++": 52.83}, "eng_Latn-arb_Arab": {"n": 100, "BLEU": 53.9, "chrF++": 70.92}, "eng_Latn-asm_Beng": {"n": 100, "BLEU": 16.82, "chrF++": 36.74}, "eng_Latn-ben_Beng": {"n": 100, "BLEU": 10.11, "chrF++": 34.11}, "eng_Latn-deu_Latn": {"n": 100, "BLEU": 56.87, "chrF++": 74.21}, "eng_Latn-fra_Latn": {"n": 100, "BLEU": 63.35, "chrF++": 77.03}, "eng_Latn-guj_Gujr": {"n": 100, "BLEU": 12.01, "chrF++": 38.05}, "eng_Latn-hin_Deva": {"n": 100, "BLEU": 16.24, "chrF++": 40.46}, "eng_Latn-jpn_Jpan": {"n": 100, "BLEU": 49.19, "chrF++": 63.86}, "eng_Latn-kan_Knda": {"n": 100, "BLEU": 10.8, "chrF++": 37.81}, "eng_Latn-mal_Mlym": {"n": 100, "BLEU": 9.26, "chrF++": 35.49}, "eng_Latn-mar_Deva": {"n": 100, "BLEU": 12.21, "chrF++": 35.3}, "eng_Latn-ory_Orya": {"n": 100, "BLEU": 11.84, "chrF++": 38.66}, "eng_Latn-pan_Guru": {"n": 100, "BLEU": 14.3, "chrF++": 34.68}, "eng_Latn-por_Latn": {"n": 100, "BLEU": 67.21, "chrF++": 78.93}, "eng_Latn-rus_Cyrl": {"n": 100, "BLEU": 50.84, "chrF++": 68.6}, "eng_Latn-spa_Latn": {"n": 100, "BLEU": 68.91, "chrF++": 80.7}, "eng_Latn-tam_Taml": {"n": 100, "BLEU": 9.73, "chrF++": 33.58}, "eng_Latn-tel_Telu": {"n": 100, "BLEU": 11.63, "chrF++": 36.59}, "eng_Latn-zho_Hans": {"n": 100, "BLEU": 39.44, "chrF++": 55.14}, "fra_Latn-arb_Arab": {"n": 100, "BLEU": 52.18, "chrF++": 67.87}, "fra_Latn-asm_Beng": {"n": 100, "BLEU": 23.39, "chrF++": 44.51}, "fra_Latn-ben_Beng": {"n": 100, "BLEU": 24.8, "chrF++": 47.16}, "fra_Latn-deu_Latn": {"n": 100, "BLEU": 53.16, "chrF++": 70.97}, "fra_Latn-eng_Latn": {"n": 100, "BLEU": 62.53, "chrF++": 75.63}, "fra_Latn-guj_Gujr": {"n": 100, "BLEU": 34.61, "chrF++": 55.75}, "fra_Latn-hin_Deva": {"n": 100, "BLEU": 32.98, "chrF++": 51.68}, "fra_Latn-jpn_Jpan": {"n": 100, "BLEU": 41.17, "chrF++": 58.77}, "fra_Latn-kan_Knda": {"n": 100, "BLEU": 28.03, "chrF++": 53.85}, "fra_Latn-mal_Mlym": {"n": 100, "BLEU": 31.41, "chrF++": 50.03}, "fra_Latn-mar_Deva": {"n": 100, "BLEU": 27.27, "chrF++": 50.01}, "fra_Latn-ory_Orya": {"n": 100, "BLEU": 31.35, "chrF++": 53.5}, "fra_Latn-pan_Guru": {"n": 100, "BLEU": 35.19, "chrF++": 50.78}, "fra_Latn-por_Latn": {"n": 100, "BLEU": 61.25, "chrF++": 74.22}, "fra_Latn-rus_Cyrl": {"n": 100, "BLEU": 47.47, "chrF++": 65.1}, "fra_Latn-spa_Latn": {"n": 100, "BLEU": 63.63, "chrF++": 77.53}, "fra_Latn-tam_Taml": {"n": 100, "BLEU": 25.64, "chrF++": 49.51}, "fra_Latn-tel_Telu": {"n": 100, "BLEU": 19.12, "chrF++": 46.24}, "fra_Latn-zho_Hans": {"n": 100, "BLEU": 41.14, "chrF++": 50.85}, "guj_Gujr-arb_Arab": {"n": 100, "BLEU": 42.79, "chrF++": 59.94}, "guj_Gujr-asm_Beng": {"n": 100, "BLEU": 24.62, "chrF++": 44.49}, "guj_Gujr-ben_Beng": {"n": 100, "BLEU": 24.18, "chrF++": 45.11}, "guj_Gujr-deu_Latn": {"n": 100, "BLEU": 44.88, "chrF++": 63.78}, "guj_Gujr-eng_Latn": {"n": 100, "BLEU": 19.49, "chrF++": 41.24}, "guj_Gujr-fra_Latn": {"n": 100, "BLEU": 49.74, "chrF++": 66.34}, "guj_Gujr-hin_Deva": {"n": 100, "BLEU": 32.0, "chrF++": 50.58}, "guj_Gujr-jpn_Jpan": {"n": 100, "BLEU": 35.05, "chrF++": 52.72}, "guj_Gujr-kan_Knda": {"n": 100, "BLEU": 25.66, "chrF++": 50.91}, "guj_Gujr-mal_Mlym": {"n": 100, "BLEU": 33.09, "chrF++": 50.45}, "guj_Gujr-mar_Deva": {"n": 100, "BLEU": 31.52, "chrF++": 50.72}, "guj_Gujr-ory_Orya": {"n": 100, "BLEU": 29.39, "chrF++": 52.23}, "guj_Gujr-pan_Guru": {"n": 100, "BLEU": 35.73, "chrF++": 51.0}, "guj_Gujr-por_Latn": {"n": 100, "BLEU": 53.01, "chrF++": 67.57}, "guj_Gujr-rus_Cyrl": {"n": 100, "BLEU": 42.86, "chrF++": 61.28}, "guj_Gujr-spa_Latn": {"n": 100, "BLEU": 51.75, "chrF++": 67.89}, "guj_Gujr-tam_Taml": {"n": 100, "BLEU": 28.16, "chrF++": 48.97}, "guj_Gujr-tel_Telu": {"n": 100, "BLEU": 18.62, "chrF++": 44.56}, "guj_Gujr-zho_Hans": {"n": 100, "BLEU": 31.29, "chrF++": 44.89}, "hin_Deva-arb_Arab": {"n": 100, "BLEU": 43.06, "chrF++": 59.4}, "hin_Deva-asm_Beng": {"n": 100, "BLEU": 23.23, "chrF++": 41.67}, "hin_Deva-ben_Beng": {"n": 100, "BLEU": 25.33, "chrF++": 46.1}, "hin_Deva-deu_Latn": {"n": 100, "BLEU": 40.09, "chrF++": 60.16}, "hin_Deva-eng_Latn": {"n": 100, "BLEU": 22.18, "chrF++": 45.65}, "hin_Deva-fra_Latn": {"n": 100, "BLEU": 49.46, "chrF++": 65.37}, "hin_Deva-guj_Gujr": {"n": 100, "BLEU": 35.11, "chrF++": 54.79}, "hin_Deva-jpn_Jpan": {"n": 100, "BLEU": 37.25, "chrF++": 54.45}, "hin_Deva-kan_Knda": {"n": 100, "BLEU": 27.63, "chrF++": 52.16}, "hin_Deva-mal_Mlym": {"n": 100, "BLEU": 30.64, "chrF++": 49.35}, "hin_Deva-mar_Deva": {"n": 100, "BLEU": 30.64, "chrF++": 49.25}, "hin_Deva-ory_Orya": {"n": 100, "BLEU": 29.01, "chrF++": 52.54}, "hin_Deva-pan_Guru": {"n": 100, "BLEU": 33.81, "chrF++": 50.3}, "hin_Deva-por_Latn": {"n": 100, "BLEU": 52.97, "chrF++": 66.51}, "hin_Deva-rus_Cyrl": {"n": 100, "BLEU": 41.22, "chrF++": 60.0}, "hin_Deva-spa_Latn": {"n": 100, "BLEU": 51.44, "chrF++": 67.44}, "hin_Deva-tam_Taml": {"n": 100, "BLEU": 27.76, "chrF++": 48.52}, "hin_Deva-tel_Telu": {"n": 100, "BLEU": 19.71, "chrF++": 44.45}, "hin_Deva-zho_Hans": {"n": 100, "BLEU": 35.98, "chrF++": 47.25}, "jpn_Jpan-arb_Arab": {"n": 100, "BLEU": 44.09, "chrF++": 61.76}, "jpn_Jpan-asm_Beng": {"n": 100, "BLEU": 24.99, "chrF++": 44.94}, "jpn_Jpan-ben_Beng": {"n": 100, "BLEU": 24.86, "chrF++": 46.29}, "jpn_Jpan-deu_Latn": {"n": 100, "BLEU": 43.18, "chrF++": 64.88}, "jpn_Jpan-eng_Latn": {"n": 100, "BLEU": 54.22, "chrF++": 69.74}, "jpn_Jpan-fra_Latn": {"n": 100, "BLEU": 51.9, "chrF++": 69.26}, "jpn_Jpan-guj_Gujr": {"n": 100, "BLEU": 35.12, "chrF++": 54.68}, "jpn_Jpan-hin_Deva": {"n": 100, "BLEU": 32.52, "chrF++": 51.48}, "jpn_Jpan-kan_Knda": {"n": 100, "BLEU": 25.22, "chrF++": 51.37}, "jpn_Jpan-mal_Mlym": {"n": 100, "BLEU": 29.29, "chrF++": 48.34}, "jpn_Jpan-mar_Deva": {"n": 100, "BLEU": 29.36, "chrF++": 50.25}, "jpn_Jpan-ory_Orya": {"n": 100, "BLEU": 25.4, "chrF++": 50.19}, "jpn_Jpan-pan_Guru": {"n": 100, "BLEU": 30.67, "chrF++": 48.3}, "jpn_Jpan-por_Latn": {"n": 100, "BLEU": 50.74, "chrF++": 66.79}, "jpn_Jpan-rus_Cyrl": {"n": 100, "BLEU": 43.78, "chrF++": 63.0}, "jpn_Jpan-spa_Latn": {"n": 100, "BLEU": 54.45, "chrF++": 71.06}, "jpn_Jpan-tam_Taml": {"n": 100, "BLEU": 25.03, "chrF++": 48.39}, "jpn_Jpan-tel_Telu": {"n": 100, "BLEU": 18.18, "chrF++": 45.17}, "jpn_Jpan-zho_Hans": {"n": 100, "BLEU": 40.46, "chrF++": 51.32}, "kan_Knda-arb_Arab": {"n": 100, "BLEU": 40.8, "chrF++": 58.36}, "kan_Knda-asm_Beng": {"n": 100, "BLEU": 24.21, "chrF++": 44.73}, "kan_Knda-ben_Beng": {"n": 100, "BLEU": 25.36, "chrF++": 44.87}, "kan_Knda-deu_Latn": {"n": 100, "BLEU": 40.46, "chrF++": 60.62}, "kan_Knda-eng_Latn": {"n": 100, "BLEU": 22.75, "chrF++": 43.79}, "kan_Knda-fra_Latn": {"n": 100, "BLEU": 46.92, "chrF++": 63.81}, "kan_Knda-guj_Gujr": {"n": 100, "BLEU": 32.98, "chrF++": 52.4}, "kan_Knda-hin_Deva": {"n": 100, "BLEU": 29.57, "chrF++": 48.19}, "kan_Knda-jpn_Jpan": {"n": 100, "BLEU": 38.94, "chrF++": 54.61}, "kan_Knda-mal_Mlym": {"n": 100, "BLEU": 30.9, "chrF++": 49.29}, "kan_Knda-mar_Deva": {"n": 100, "BLEU": 27.6, "chrF++": 48.55}, "kan_Knda-ory_Orya": {"n": 100, "BLEU": 25.3, "chrF++": 48.55}, "kan_Knda-pan_Guru": {"n": 100, "BLEU": 32.09, "chrF++": 48.54}, "kan_Knda-por_Latn": {"n": 100, "BLEU": 46.49, "chrF++": 62.42}, "kan_Knda-rus_Cyrl": {"n": 100, "BLEU": 38.27, "chrF++": 57.08}, "kan_Knda-spa_Latn": {"n": 100, "BLEU": 50.22, "chrF++": 65.65}, "kan_Knda-tam_Taml": {"n": 100, "BLEU": 22.45, "chrF++": 45.51}, "kan_Knda-tel_Telu": {"n": 100, "BLEU": 20.87, "chrF++": 44.92}, "kan_Knda-zho_Hans": {"n": 100, "BLEU": 33.84, "chrF++": 44.46}, "mal_Mlym-arb_Arab": {"n": 100, "BLEU": 35.26, "chrF++": 55.5}, "mal_Mlym-asm_Beng": {"n": 100, "BLEU": 19.41, "chrF++": 39.17}, "mal_Mlym-ben_Beng": {"n": 100, "BLEU": 22.9, "chrF++": 44.33}, "mal_Mlym-deu_Latn": {"n": 100, "BLEU": 33.17, "chrF++": 58.12}, "mal_Mlym-eng_Latn": {"n": 100, "BLEU": 19.44, "chrF++": 41.55}, "mal_Mlym-fra_Latn": {"n": 100, "BLEU": 40.2, "chrF++": 59.81}, "mal_Mlym-guj_Gujr": {"n": 100, "BLEU": 31.57, "chrF++": 51.18}, "mal_Mlym-hin_Deva": {"n": 100, "BLEU": 30.55, "chrF++": 47.76}, "mal_Mlym-jpn_Jpan": {"n": 100, "BLEU": 38.62, "chrF++": 51.3}, "mal_Mlym-kan_Knda": {"n": 100, "BLEU": 22.44, "chrF++": 48.82}, "mal_Mlym-mar_Deva": {"n": 100, "BLEU": 22.91, "chrF++": 45.67}, "mal_Mlym-ory_Orya": {"n": 100, "BLEU": 23.9, "chrF++": 47.31}, "mal_Mlym-pan_Guru": {"n": 100, "BLEU": 32.03, "chrF++": 48.82}, "mal_Mlym-por_Latn": {"n": 100, "BLEU": 40.3, "chrF++": 60.34}, "mal_Mlym-rus_Cyrl": {"n": 100, "BLEU": 32.7, "chrF++": 55.18}, "mal_Mlym-spa_Latn": {"n": 100, "BLEU": 43.06, "chrF++": 62.8}, "mal_Mlym-tam_Taml": {"n": 100, "BLEU": 24.37, "chrF++": 47.06}, "mal_Mlym-tel_Telu": {"n": 100, "BLEU": 18.13, "chrF++": 42.48}, "mal_Mlym-zho_Hans": {"n": 100, "BLEU": 30.95, "chrF++": 42.51}, "mar_Deva-arb_Arab": {"n": 100, "BLEU": 40.07, "chrF++": 56.6}, "mar_Deva-asm_Beng": {"n": 100, "BLEU": 22.37, "chrF++": 41.44}, "mar_Deva-ben_Beng": {"n": 100, "BLEU": 23.15, "chrF++": 44.47}, "mar_Deva-deu_Latn": {"n": 100, "BLEU": 37.39, "chrF++": 58.29}, "mar_Deva-eng_Latn": {"n": 100, "BLEU": 19.32, "chrF++": 39.82}, "mar_Deva-fra_Latn": {"n": 100, "BLEU": 44.38, "chrF++": 62.42}, "mar_Deva-guj_Gujr": {"n": 100, "BLEU": 33.03, "chrF++": 51.71}, "mar_Deva-hin_Deva": {"n": 100, "BLEU": 31.41, "chrF++": 48.71}, "mar_Deva-jpn_Jpan": {"n": 100, "BLEU": 34.21, "chrF++": 51.0}, "mar_Deva-kan_Knda": {"n": 100, "BLEU": 23.98, "chrF++": 49.87}, "mar_Deva-mal_Mlym": {"n": 100, "BLEU": 30.8, "chrF++": 48.89}, "mar_Deva-ory_Orya": {"n": 100, "BLEU": 26.8, "chrF++": 47.87}, "mar_Deva-pan_Guru": {"n": 100, "BLEU": 34.47, "chrF++": 50.45}, "mar_Deva-por_Latn": {"n": 100, "BLEU": 46.27, "chrF++": 61.77}, "mar_Deva-rus_Cyrl": {"n": 100, "BLEU": 37.57, "chrF++": 57.52}, "mar_Deva-spa_Latn": {"n": 100, "BLEU": 45.26, "chrF++": 63.22}, "mar_Deva-tam_Taml": {"n": 100, "BLEU": 26.6, "chrF++": 47.36}, "mar_Deva-tel_Telu": {"n": 100, "BLEU": 20.76, "chrF++": 43.21}, "mar_Deva-zho_Hans": {"n": 100, "BLEU": 29.26, "chrF++": 42.06}, "ory_Orya-arb_Arab": {"n": 100, "BLEU": 43.32, "chrF++": 60.08}, "ory_Orya-asm_Beng": {"n": 100, "BLEU": 22.66, "chrF++": 42.37}, "ory_Orya-ben_Beng": {"n": 100, "BLEU": 21.99, "chrF++": 42.75}, "ory_Orya-deu_Latn": {"n": 100, "BLEU": 42.68, "chrF++": 60.69}, "ory_Orya-eng_Latn": {"n": 100, "BLEU": 27.22, "chrF++": 49.64}, "ory_Orya-fra_Latn": {"n": 100, "BLEU": 48.84, "chrF++": 64.98}, "ory_Orya-guj_Gujr": {"n": 100, "BLEU": 34.25, "chrF++": 52.31}, "ory_Orya-hin_Deva": {"n": 100, "BLEU": 31.79, "chrF++": 48.6}, "ory_Orya-jpn_Jpan": {"n": 100, "BLEU": 47.24, "chrF++": 55.2}, "ory_Orya-kan_Knda": {"n": 100, "BLEU": 27.73, "chrF++": 52.34}, "ory_Orya-mal_Mlym": {"n": 100, "BLEU": 30.78, "chrF++": 48.47}, "ory_Orya-mar_Deva": {"n": 100, "BLEU": 29.91, "chrF++": 49.41}, "ory_Orya-pan_Guru": {"n": 100, "BLEU": 31.68, "chrF++": 48.52}, "ory_Orya-por_Latn": {"n": 100, "BLEU": 47.62, "chrF++": 63.12}, "ory_Orya-rus_Cyrl": {"n": 100, "BLEU": 38.52, "chrF++": 57.49}, "ory_Orya-spa_Latn": {"n": 100, "BLEU": 47.22, "chrF++": 63.96}, "ory_Orya-tam_Taml": {"n": 100, "BLEU": 24.98, "chrF++": 47.78}, "ory_Orya-tel_Telu": {"n": 100, "BLEU": 18.78, "chrF++": 42.91}, "ory_Orya-zho_Hans": {"n": 100, "BLEU": 38.84, "chrF++": 47.58}, "pan_Guru-arb_Arab": {"n": 100, "BLEU": 45.53, "chrF++": 61.0}, "pan_Guru-asm_Beng": {"n": 100, "BLEU": 22.36, "chrF++": 41.51}, "pan_Guru-ben_Beng": {"n": 100, "BLEU": 23.73, "chrF++": 44.22}, "pan_Guru-deu_Latn": {"n": 100, "BLEU": 39.84, "chrF++": 59.95}, "pan_Guru-eng_Latn": {"n": 100, "BLEU": 18.51, "chrF++": 38.79}, "pan_Guru-fra_Latn": {"n": 100, "BLEU": 44.54, "chrF++": 63.46}, "pan_Guru-guj_Gujr": {"n": 100, "BLEU": 32.87, "chrF++": 53.24}, "pan_Guru-hin_Deva": {"n": 100, "BLEU": 31.69, "chrF++": 49.06}, "pan_Guru-jpn_Jpan": {"n": 100, "BLEU": 43.18, "chrF++": 54.16}, "pan_Guru-kan_Knda": {"n": 100, "BLEU": 27.09, "chrF++": 53.0}, "pan_Guru-mal_Mlym": {"n": 100, "BLEU": 29.35, "chrF++": 46.33}, "pan_Guru-mar_Deva": {"n": 100, "BLEU": 27.67, "chrF++": 48.75}, "pan_Guru-ory_Orya": {"n": 100, "BLEU": 26.85, "chrF++": 49.17}, "pan_Guru-por_Latn": {"n": 100, "BLEU": 49.72, "chrF++": 65.48}, "pan_Guru-rus_Cyrl": {"n": 100, "BLEU": 38.38, "chrF++": 56.97}, "pan_Guru-spa_Latn": {"n": 100, "BLEU": 48.71, "chrF++": 66.08}, "pan_Guru-tam_Taml": {"n": 100, "BLEU": 24.98, "chrF++": 48.16}, "pan_Guru-tel_Telu": {"n": 100, "BLEU": 19.25, "chrF++": 43.86}, "pan_Guru-zho_Hans": {"n": 100, "BLEU": 34.31, "chrF++": 43.69}, "por_Latn-arb_Arab": {"n": 100, "BLEU": 51.83, "chrF++": 68.44}, "por_Latn-asm_Beng": {"n": 100, "BLEU": 25.71, "chrF++": 46.58}, "por_Latn-ben_Beng": {"n": 100, "BLEU": 24.88, "chrF++": 46.8}, "por_Latn-deu_Latn": {"n": 100, "BLEU": 51.31, "chrF++": 70.4}, "por_Latn-eng_Latn": {"n": 100, "BLEU": 63.68, "chrF++": 77.3}, "por_Latn-fra_Latn": {"n": 100, "BLEU": 60.23, "chrF++": 74.69}, "por_Latn-guj_Gujr": {"n": 100, "BLEU": 33.7, "chrF++": 54.52}, "por_Latn-hin_Deva": {"n": 100, "BLEU": 35.62, "chrF++": 53.32}, "por_Latn-jpn_Jpan": {"n": 100, "BLEU": 44.33, "chrF++": 60.63}, "por_Latn-kan_Knda": {"n": 100, "BLEU": 27.24, "chrF++": 52.63}, "por_Latn-mal_Mlym": {"n": 100, "BLEU": 31.58, "chrF++": 51.04}, "por_Latn-mar_Deva": {"n": 100, "BLEU": 29.65, "chrF++": 51.71}, "por_Latn-ory_Orya": {"n": 100, "BLEU": 30.37, "chrF++": 52.95}, "por_Latn-pan_Guru": {"n": 100, "BLEU": 34.23, "chrF++": 50.63}, "por_Latn-rus_Cyrl": {"n": 100, "BLEU": 48.84, "chrF++": 67.68}, "por_Latn-spa_Latn": {"n": 100, "BLEU": 66.01, "chrF++": 79.74}, "por_Latn-tam_Taml": {"n": 100, "BLEU": 27.16, "chrF++": 50.33}, "por_Latn-tel_Telu": {"n": 100, "BLEU": 19.64, "chrF++": 46.41}, "por_Latn-zho_Hans": {"n": 100, "BLEU": 39.28, "chrF++": 51.86}, "rus_Cyrl-arb_Arab": {"n": 100, "BLEU": 49.01, "chrF++": 64.88}, "rus_Cyrl-asm_Beng": {"n": 100, "BLEU": 23.98, "chrF++": 44.54}, "rus_Cyrl-ben_Beng": {"n": 100, "BLEU": 24.31, "chrF++": 45.02}, "rus_Cyrl-deu_Latn": {"n": 100, "BLEU": 47.41, "chrF++": 65.25}, "rus_Cyrl-eng_Latn": {"n": 100, "BLEU": 58.83, "chrF++": 72.55}, "rus_Cyrl-fra_Latn": {"n": 100, "BLEU": 52.59, "chrF++": 69.09}, "rus_Cyrl-guj_Gujr": {"n": 100, "BLEU": 31.89, "chrF++": 52.54}, "rus_Cyrl-hin_Deva": {"n": 100, "BLEU": 33.53, "chrF++": 51.13}, "rus_Cyrl-jpn_Jpan": {"n": 100, "BLEU": 35.28, "chrF++": 56.68}, "rus_Cyrl-kan_Knda": {"n": 100, "BLEU": 27.31, "chrF++": 51.98}, "rus_Cyrl-mal_Mlym": {"n": 100, "BLEU": 29.77, "chrF++": 48.83}, "rus_Cyrl-mar_Deva": {"n": 100, "BLEU": 30.55, "chrF++": 51.59}, "rus_Cyrl-ory_Orya": {"n": 100, "BLEU": 26.12, "chrF++": 50.69}, "rus_Cyrl-pan_Guru": {"n": 100, "BLEU": 33.66, "chrF++": 50.4}, "rus_Cyrl-por_Latn": {"n": 100, "BLEU": 56.42, "chrF++": 70.69}, "rus_Cyrl-spa_Latn": {"n": 100, "BLEU": 55.99, "chrF++": 71.5}, "rus_Cyrl-tam_Taml": {"n": 100, "BLEU": 23.46, "chrF++": 47.02}, "rus_Cyrl-tel_Telu": {"n": 100, "BLEU": 18.57, "chrF++": 43.88}, "rus_Cyrl-zho_Hans": {"n": 100, "BLEU": 40.98, "chrF++": 50.48}, "spa_Latn-arb_Arab": {"n": 100, "BLEU": 51.73, "chrF++": 69.06}, "spa_Latn-asm_Beng": {"n": 100, "BLEU": 26.47, "chrF++": 46.91}, "spa_Latn-ben_Beng": {"n": 100, "BLEU": 24.64, "chrF++": 46.36}, "spa_Latn-deu_Latn": {"n": 100, "BLEU": 52.64, "chrF++": 70.56}, "spa_Latn-eng_Latn": {"n": 100, "BLEU": 64.71, "chrF++": 78.28}, "spa_Latn-fra_Latn": {"n": 100, "BLEU": 60.05, "chrF++": 75.18}, "spa_Latn-guj_Gujr": {"n": 100, "BLEU": 36.43, "chrF++": 56.58}, "spa_Latn-hin_Deva": {"n": 100, "BLEU": 34.37, "chrF++": 52.4}, "spa_Latn-jpn_Jpan": {"n": 100, "BLEU": 44.21, "chrF++": 59.55}, "spa_Latn-kan_Knda": {"n": 100, "BLEU": 30.29, "chrF++": 55.66}, "spa_Latn-mal_Mlym": {"n": 100, "BLEU": 31.48, "chrF++": 49.89}, "spa_Latn-mar_Deva": {"n": 100, "BLEU": 29.59, "chrF++": 51.85}, "spa_Latn-ory_Orya": {"n": 100, "BLEU": 29.08, "chrF++": 53.07}, "spa_Latn-pan_Guru": {"n": 100, "BLEU": 35.0, "chrF++": 51.22}, "spa_Latn-por_Latn": {"n": 100, "BLEU": 66.07, "chrF++": 77.85}, "spa_Latn-rus_Cyrl": {"n": 100, "BLEU": 47.53, "chrF++": 67.4}, "spa_Latn-tam_Taml": {"n": 100, "BLEU": 23.35, "chrF++": 49.27}, "spa_Latn-tel_Telu": {"n": 100, "BLEU": 19.6, "chrF++": 46.48}, "spa_Latn-zho_Hans": {"n": 100, "BLEU": 37.53, "chrF++": 52.42}, "tam_Taml-arb_Arab": {"n": 100, "BLEU": 42.44, "chrF++": 59.56}, "tam_Taml-asm_Beng": {"n": 100, "BLEU": 21.42, "chrF++": 39.89}, "tam_Taml-ben_Beng": {"n": 100, "BLEU": 22.3, "chrF++": 42.51}, "tam_Taml-deu_Latn": {"n": 100, "BLEU": 38.69, "chrF++": 58.38}, "tam_Taml-eng_Latn": {"n": 100, "BLEU": 17.07, "chrF++": 36.51}, "tam_Taml-fra_Latn": {"n": 100, "BLEU": 43.96, "chrF++": 61.11}, "tam_Taml-guj_Gujr": {"n": 100, "BLEU": 32.18, "chrF++": 51.16}, "tam_Taml-hin_Deva": {"n": 100, "BLEU": 30.77, "chrF++": 47.19}, "tam_Taml-jpn_Jpan": {"n": 100, "BLEU": 32.0, "chrF++": 49.86}, "tam_Taml-kan_Knda": {"n": 100, "BLEU": 21.82, "chrF++": 48.08}, "tam_Taml-mal_Mlym": {"n": 100, "BLEU": 26.62, "chrF++": 45.59}, "tam_Taml-mar_Deva": {"n": 100, "BLEU": 29.18, "chrF++": 47.33}, "tam_Taml-ory_Orya": {"n": 100, "BLEU": 25.48, "chrF++": 48.43}, "tam_Taml-pan_Guru": {"n": 100, "BLEU": 33.48, "chrF++": 48.78}, "tam_Taml-por_Latn": {"n": 100, "BLEU": 42.95, "chrF++": 60.48}, "tam_Taml-rus_Cyrl": {"n": 100, "BLEU": 35.68, "chrF++": 53.88}, "tam_Taml-spa_Latn": {"n": 100, "BLEU": 43.76, "chrF++": 62.48}, "tam_Taml-tel_Telu": {"n": 100, "BLEU": 18.62, "chrF++": 41.1}, "tam_Taml-zho_Hans": {"n": 100, "BLEU": 33.89, "chrF++": 43.29}, "tel_Telu-arb_Arab": {"n": 100, "BLEU": 45.31, "chrF++": 59.77}, "tel_Telu-asm_Beng": {"n": 100, "BLEU": 25.68, "chrF++": 45.38}, "tel_Telu-ben_Beng": {"n": 100, "BLEU": 25.08, "chrF++": 45.78}, "tel_Telu-deu_Latn": {"n": 100, "BLEU": 45.29, "chrF++": 63.88}, "tel_Telu-eng_Latn": {"n": 100, "BLEU": 22.47, "chrF++": 45.56}, "tel_Telu-fra_Latn": {"n": 100, "BLEU": 50.02, "chrF++": 66.94}, "tel_Telu-guj_Gujr": {"n": 100, "BLEU": 35.52, "chrF++": 54.85}, "tel_Telu-hin_Deva": {"n": 100, "BLEU": 35.78, "chrF++": 52.95}, "tel_Telu-jpn_Jpan": {"n": 100, "BLEU": 36.35, "chrF++": 52.98}, "tel_Telu-kan_Knda": {"n": 100, "BLEU": 26.44, "chrF++": 53.39}, "tel_Telu-mal_Mlym": {"n": 100, "BLEU": 31.98, "chrF++": 49.73}, "tel_Telu-mar_Deva": {"n": 100, "BLEU": 32.25, "chrF++": 53.13}, "tel_Telu-ory_Orya": {"n": 100, "BLEU": 27.4, "chrF++": 51.87}, "tel_Telu-pan_Guru": {"n": 100, "BLEU": 33.86, "chrF++": 50.65}, "tel_Telu-por_Latn": {"n": 100, "BLEU": 51.5, "chrF++": 65.76}, "tel_Telu-rus_Cyrl": {"n": 100, "BLEU": 41.81, "chrF++": 61.22}, "tel_Telu-spa_Latn": {"n": 100, "BLEU": 51.24, "chrF++": 67.4}, "tel_Telu-tam_Taml": {"n": 100, "BLEU": 28.09, "chrF++": 50.3}, "tel_Telu-zho_Hans": {"n": 100, "BLEU": 39.43, "chrF++": 46.13}, "zho_Hans-arb_Arab": {"n": 100, "BLEU": 43.24, "chrF++": 61.78}, "zho_Hans-asm_Beng": {"n": 100, "BLEU": 21.0, "chrF++": 41.85}, "zho_Hans-ben_Beng": {"n": 100, "BLEU": 21.27, "chrF++": 43.71}, "zho_Hans-deu_Latn": {"n": 100, "BLEU": 45.28, "chrF++": 63.95}, "zho_Hans-eng_Latn": {"n": 100, "BLEU": 56.4, "chrF++": 71.44}, "zho_Hans-fra_Latn": {"n": 100, "BLEU": 52.19, "chrF++": 68.65}, "zho_Hans-guj_Gujr": {"n": 100, "BLEU": 33.94, "chrF++": 54.02}, "zho_Hans-hin_Deva": {"n": 100, "BLEU": 31.65, "chrF++": 50.19}, "zho_Hans-jpn_Jpan": {"n": 100, "BLEU": 39.56, "chrF++": 54.45}, "zho_Hans-kan_Knda": {"n": 100, "BLEU": 24.29, "chrF++": 50.31}, "zho_Hans-mal_Mlym": {"n": 100, "BLEU": 28.82, "chrF++": 48.44}, "zho_Hans-mar_Deva": {"n": 100, "BLEU": 27.64, "chrF++": 47.89}, "zho_Hans-ory_Orya": {"n": 100, "BLEU": 23.7, "chrF++": 47.67}, "zho_Hans-pan_Guru": {"n": 100, "BLEU": 31.08, "chrF++": 48.07}, "zho_Hans-por_Latn": {"n": 100, "BLEU": 51.69, "chrF++": 67.67}, "zho_Hans-rus_Cyrl": {"n": 100, "BLEU": 41.51, "chrF++": 60.92}, "zho_Hans-spa_Latn": {"n": 100, "BLEU": 53.57, "chrF++": 69.9}, "zho_Hans-tam_Taml": {"n": 100, "BLEU": 23.14, "chrF++": 47.31}, "zho_Hans-tel_Telu": {"n": 100, "BLEU": 18.9, "chrF++": 43.58}}, "per_direction_sentences": 100}
• ONNX: exported + verified under onnxruntime (encoder.onnx + decoder.onnx)