A general language translation model for translating between Welsh and English.
This model was trained using custom DVC pipeline employing
Marian NMT,
the datasets prepared were generated from the following sources:
The data was split into train, validation and test sets; the test comprising of a random slice of 20% of the total dataset. Segments were selected randomly form
of text and TMX from the datasets described above.
The datasets were cleaned, without any pre-tokenisation, utilising a SentencePiece vocabulary model, and then fed into a 10 separate Marian NMT training processes, the data having been split into
split into 10 training and validation sets.
The BLEU evaluation score was produced using the python library
SacreBLEU.
1import trnasformers
2model_id = "mgrbyte/mt-general-cy-en"
3tokenizer = transformers.AutoTokenizer.from_pretrained(model_id)
4model = transformers.AutoModelForSeq2SeqLM.from_pretrained(model_id)
5translate = transformers.pipeline("translation", model=model, tokenizer=tokenizer)
6translated = translate(
7 "Mae gan Lywodraeth Cymru targed i gyrraedd miliwn o siariadwyr Cymraeg erbyn y flwyddyn 2020."
8)
9print(translated["translation_text"])
