This repository provides large language models trained by
SB Intuitions.
1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline, set_seed
3
4model = AutoModelForCausalLM.from_pretrained("sbintuitions/sarashina2.1-1b", torch_dtype=torch.bfloat16, device_map="auto")
5tokenizer = AutoTokenizer.from_pretrained("sbintuitions/sarashina2.1-1b")
6generator = pipeline("text-generation", model=model, tokenizer=tokenizer)
7set_seed(123)
8
9text = generator(
10 "おはようございます、今日の天気は",
11 max_length=30,
12 do_sample=True,
13 pad_token_id=tokenizer.pad_token_id,
14 num_return_sequences=3,
15)
16
17for t in text:
18 print(t)
19
20
We constructed this Sarashina2.1-1B model, which consists of 1 billion parameters, using a two-phase training process.
First, we trained the model on 10 trillion tokens, including Japanese and English data extracted from web corpora.
Then, we trained the model using 1 trillion tokens, predominantly consisting of Japanese data, to enhance its performance in Japanese.
The following tables show the model's performance on Japanese and English tasks.
We also show the performance of other public LLMs for reference.
Sarashina2.1 has not been tuned to follow an instruction yet.
Therefore, sarashina2.1 might generate some meaningless sequences, some inaccurate instances or biased/objectionable outputs.
Before using sarashina2.1, we would like developers to tune models based on human preferences and safety considerations.
