This repository provides large language models trained by
SB Intuitions.
1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline, set_seed
3
4model = AutoModelForCausalLM.from_pretrained("sbintuitions/sarashina2.2-1b", torch_dtype=torch.bfloat16, device_map="auto")
5tokenizer = AutoTokenizer.from_pretrained("sbintuitions/sarashina2.2-1b")
6generator = pipeline("text-generation", model=model, tokenizer=tokenizer)
7set_seed(123)
8
9text = generator(
10 "おはようございます、今日の天気は",
11 max_length=30,
12 do_sample=True,
13 pad_token_id=tokenizer.pad_token_id,
14 num_return_sequences=3,
15)
16
17for t in text:
18 print(t)
19
20
We constructed the Sarashina2.2-1B model, which consists of about 1 billion parameters (excluding embeddings and the LM head when calculating the number of parameters), using a three-phase training process.
First, we trained the model on 10 trillion tokens, including Japanese, English, and code data extracted from web corpora.
Next, we trained the model using synthetic data to improve its performance on math and coding tasks.
Finally, we trained the model with a small amount of data to enhance its performance on various application tasks.
The following tables show the model's performance on Japanese tasks.
For reference, we also present the performance of our previous LLMs.
As shown in the table, our Sarashina2.2-3B outperforms Sarashina2-7B in Japanese QA tasks such as NIILC and JMMLU.
In addition, Sarashina2.2-3B outperforms Sarashina2-70B in Japanese math and coding tasks such as MGSM-ja and JHumanEval.
This repository contains the pre-trained model, which has not yet been tuned to follow instructions.
Therefore, this model may generate meaningless sequences, inaccurate instances, or biased/objectionable outputs.
As post-trained Sarashina2.2 models, we have published
Sarashina2.2-0.5B-instruct-v0.1,
Sarashina2.2-1B-instruct-v0.1, and
Sarashina2.2-3B-instruct-v0.1.
