This repository provides large language models developed by
LLM-jp, a collaborative project launched in Japan.
1import torch
2from transformers import AutoTokenizer, AutoModelForCausalLM
3tokenizer = AutoTokenizer.from_pretrained("llm-jp/llm-jp-13b-v2.0")
4model = AutoModelForCausalLM.from_pretrained("llm-jp/llm-jp-13b-v2.0", device_map="auto", torch_dtype=torch.bfloat16)
5text = "自然言語処理とは何か"
6tokenized_input = tokenizer.encode(text, add_special_tokens=False, return_tensors="pt").to(model.device)
7with torch.no_grad():
8 output = model.generate(
9 tokenized_input,
10 max_new_tokens=100,
11 do_sample=True,
12 top_p=0.95,
13 temperature=0.7,
14 repetition_penalty=1.05,
15 )[0]
16print(tokenizer.decode(output))
The tokenizer of this model is based on
huggingface/tokenizers Unigram byte-fallback model.
The vocabulary entries were converted from
llm-jp-tokenizer v2.2 (100k: code20K_en40K_ja60K.ver2.2).
Please refer to
README.md of
llm-ja-tokenizer for details on the vocabulary construction procedure (the pure SentencePiece training does not reproduce our vocabulary).
The models have been pre-trained using a blend of the following datasets.
The models have been fine-tuned on the following datasets.
You can view the evaluation results of several LLMs on this
leaderboard. We used
llm-jp-eval (v1.3.0) for the evaluation.
Besides, we used LLM-as-a-judge frameworks,
Japanese Vicuna QA Benchmark and
Japanese MT Bench, for evaluation.
For details, please refer to
our technical blog (in Japanese).
The models released here are still in the early stages of our research and development and have not been tuned to ensure outputs align with human intent and safety considerations.
Namgi Han, Tatsuya Hiraoka, Hirokazu Kiyomaru, Takashi Kodama, and Hiroshi Matsuda.
