Views
No views yet
google/mt5-base architecture to support translation between English and Dhivehi in both directions. The model is instruction-prefixed using "2dv" for English→Dhivehi and "2en" for Dhivehi→English.Input: Hello, how are you?
Output: ހެލޯ، ކިހިނެއްވީ؟
Input: I love reading books.
Output: އަހަރެން ފޮތް ކިޔަން ވަރަށް ލޯބިވޭ.Input: ކިހިނެއްތަ އުޅެނީ؟
Output: how's it going?
Input: ރާއްޖެއަކީ ރީތި ޤައުމެކެވެ.
Output: sri lanka is a beautiful country. (Note: dataset quality may affect accuracy)1from transformers import MT5ForConditionalGeneration, MT5Tokenizer
2
3# Load model and tokenizer
4model = MT5ForConditionalGeneration.from_pretrained("./mt5-base-dv-en/best_model")
5tokenizer = MT5Tokenizer.from_pretrained("./mt5-base-dv-en/best_model")
6
7# English to Dhivehi
8def translate_to_dhivehi(text):
9 inputs = tokenizer("2dv" + text, return_tensors="pt", max_length=512, truncation=True)
10 outputs = model.generate(
11 **inputs,
12 max_length=100,
13 num_beams=5,
14 length_penalty=2.5,
15 repetition_penalty=1.5,
16 early_stopping=True,
17 no_repeat_ngram_size=2
18 )
19 return tokenizer.decode(outputs[0], skip_special_tokens=True)
20
21# Dhivehi to English
22def translate_to_english(text):
23 inputs = tokenizer("2en" + text, return_tensors="pt", max_length=512, truncation=True)
24 outputs = model.generate(
25 **inputs,
26 max_length=100,
27 num_beams=5,
28 length_penalty=2.5,
29 repetition_penalty=1.5,
30 early_stopping=True,
31 no_repeat_ngram_size=2
32 )
33 return tokenizer.decode(outputs[0], skip_special_tokens=True)
34
35# Example usage
36print(translate_to_dhivehi("Hello, how are you?"))
37print(translate_to_english("ކިހިނެއްތަ އުޅެނީ؟"))google/mt5-base"2dv" for English → Dhivehi"2en" for Dhivehi → English