We conduct continual pre-training of
qwen-7b on
30B tokens from a mixture of Japanese and English datasets. The continual pre-training significantly improves the model's performance on Japanese tasks. It also enjoys the following great features provided by the original Qwen model.
The name
nekomata comes from the Japanese word
猫又/ねこまた/Nekomata, which is a kind of Japanese mythical creature (
妖怪/ようかい/Youkai).
1import torch
2from transformers import AutoTokenizer, AutoModelForCausalLM
3
4tokenizer = AutoTokenizer.from_pretrained("rinna/nekomata-7b", trust_remote_code=True)
5
6# Use GPU with bf16
7# model = AutoModelForCausalLM.from_pretrained("rinna/nekomata-7b", device_map="auto", trust_remote_code=True, bf16=True)
8
9# Use GPU with fp16
10# model = AutoModelForCausalLM.from_pretrained("rinna/nekomata-7b", device_map="auto", trust_remote_code=True, fp16=True)
11
12# Use CPU
13# model = AutoModelForCausalLM.from_pretrained("rinna/nekomata-7b", device_map="cpu", trust_remote_code=True)
14
15# Automatically select device and precision
16model = AutoModelForCausalLM.from_pretrained("rinna/nekomata-7b", device_map="auto", trust_remote_code=True)
17
18text = "西田幾多郎は、"
19token_ids = tokenizer.encode(text, add_special_tokens=False, return_tensors="pt")
20
21with torch.no_grad():
22 output_ids = model.generate(
23 token_ids.to(model.device),
24 max_new_tokens=200,
25 min_new_tokens=200,
26 do_sample=True,
27 temperature=1.0,
28 top_p=0.95,
29 pad_token_id=tokenizer.pad_token_id,
30 bos_token_id=tokenizer.bos_token_id,
31 eos_token_id=tokenizer.eos_token_id
32 )
33
34output = tokenizer.decode(output_ids.tolist()[0])
35print(output)
The model uses the original Qwen tokenizer. It augments the
cl100k tiktoken tokenizer and has a vocabulary size of 151,936. The inclusive vocabulary helps the model to reach a better tokenization efficiency, especially for Japanese texts.
1@misc{rinna-nekomata-7b,
2 title = {rinna/nekomata-7b},
3 author = {Zhao, Tianyu and Kaga, Akio and Sawada, Kei},
4 url = {https://huggingface.co/rinna/nekomata-7b}
5}
6
7@inproceedings{sawada2024release,
8 title = {Release of Pre-Trained Models for the {J}apanese Language},
9 author = {Sawada, Kei and Zhao, Tianyu and Shing, Makoto and Mitsui, Kentaro and Kaga, Akio and Hono, Yukiya and Wakatsuki, Toshiaki and Mitsuda, Koh},
10 booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
11 month = {5},
12 year = {2024},
13 pages = {13898--13905},
14 url = {https://aclanthology.org/2024.lrec-main.1213},
15 note = {\url{https://arxiv.org/abs/2404.01657}}
16}
1@software{gpt-neox-library,
2 title = {{GPT}-{N}eo{X}: Large Scale Autoregressive Language Modeling in {P}y{T}orch},
3 author = {Andonian, Alex and Anthony, Quentin and Biderman, Stella and Black, Sid and Gali, Preetham and Gao, Leo and Hallahan, Eric and Levy-Kramer, Josh and Leahy, Connor and Nestler, Lucas and Parker, Kip and Pieler, Michael and Purohit, Shivanshu and Songz, Tri and Phil, Wang and Weinbach, Samuel},
4 doi = {10.5281/zenodo.5879544},
5 month = {8},
6 year = {2021},
7 version = {0.0.1},
8 url = {https://www.github.com/eleutherai/gpt-neox}
9}
