We present BLOOMZ & mT0, a family of models capable of following human instructions in dozens of languages zero-shot. We finetune BLOOM & mT5 pretrained multilingual language models on our crosslingual task mixture (xP3) and find our resulting models capable of crosslingual generalization to unseen tasks & languages.
| Multitask finetuned on xP3. Recommended for prompting in English. | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Parameters | 300M | 580M | 1.2B | 3.7B | 13B | 560M | 1.1B | 1.7B | 3B | 7.1B | 176B |
| Finetuned Model | mt0-small | mt0-base | mt0-large | mt0-xl | mt0-xxl | bloomz-560m | bloomz-1b1 | bloomz-1b7 | bloomz-3b | bloomz-7b1 | bloomz |
| Multitask finetuned on xP3mt. Recommended for prompting in non-English. | |||||||||||
| Finetuned Model | mt0-xxl-mt | bloomz-7b1-mt | bloomz-mt | ||||||||
| Multitask finetuned on P3. Released for research purposes only. Strictly inferior to above models! | |||||||||||
| Finetuned Model | mt0-xxl-p3 | bloomz-7b1-p3 | bloomz-p3 | ||||||||
| Original pretrained checkpoints. Not recommended. | |||||||||||
| Pretrained Model | mt5-small | mt5-base | mt5-large | mt5-xl | mt5-xxl | bloom-560m | bloom-1b1 | bloom-1b7 | bloom-3b | bloom-7b1 | bloom |
1# pip install -q transformers
2from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
3
4checkpoint = "bigscience/mt0-xxl"
5
6tokenizer = AutoTokenizer.from_pretrained(checkpoint)
7model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)
8
9inputs = tokenizer.encode("Translate to English: Je t’aime.", return_tensors="pt")
10outputs = model.generate(inputs)
11print(tokenizer.decode(outputs[0]))1# pip install -q transformers accelerate
2from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
3
4checkpoint = "bigscience/mt0-xxl"
5
6tokenizer = AutoTokenizer.from_pretrained(checkpoint)
7model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint, torch_dtype="auto", device_map="auto")
8
9inputs = tokenizer.encode("Translate to English: Je t’aime.", return_tensors="pt").to("cuda")
10outputs = model.generate(inputs)
11print(tokenizer.decode(outputs[0]))1# pip install -q transformers accelerate bitsandbytes
2from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
3
4checkpoint = "bigscience/mt0-xxl"
5
6tokenizer = AutoTokenizer.from_pretrained(checkpoint)
7model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint, device_map="auto", load_in_8bit=True)
8
9inputs = tokenizer.encode("Translate to English: Je t’aime.", return_tensors="pt").to("cuda")
10outputs = model.generate(inputs)
11print(tokenizer.decode(outputs[0]))config.json file1@article{muennighoff2022crosslingual,
2 title={Crosslingual generalization through multitask finetuning},
3 author={Muennighoff, Niklas and Wang, Thomas and Sutawika, Lintang and Roberts, Adam and Biderman, Stella and Scao, Teven Le and Bari, M Saiful and Shen, Sheng and Yong, Zheng-Xin and Schoelkopf, Hailey and others},
4 journal={arXiv preprint arXiv:2211.01786},
5 year={2022}
6}