If you would like to learn more about the pretraining of the LLM-jp-3 MoE series, please refer to this blog post.
Tokenizer
The tokenizer of this model is based on huggingface/tokenizers Unigram byte-fallback model.
The vocabulary entries were converted from llm-jp-tokenizer v3.0.
Please refer to README.md of llm-jp-tokenizer for details on the vocabulary construction procedure (the pure SentencePiece training does not reproduce our vocabulary).
Datasets
Pre-training
The models have been pre-trained using a blend of the following datasets.
We evaluated the models using gpt-4o-2024-08-06.
The scores represent the average values obtained from five rounds of inference and evaluation.
For more details, please refer to the codes.
AnswerCarefully-Eval assesses the safety of Japanese language model outputs using the LLM-as-a-Judge approach, based on the test set from llm-jp/AnswerCarefully.
We evaluated the models using gpt-4-0613.
The scores represent the average values obtained from five rounds of inference and evaluation.
The models released here are in the early stages of our research and development and have not been tuned to ensure outputs align with human intent and safety considerations.
If you find our work helpful, please feel free to cite the paper.
@inproceedings{
nakamura2025dropupcycling,
title={Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization},
author={Taishi Nakamura and Takuya Akiba and Kazuki Fujii and Yusuke Oda and Rio Yokota and Jun Suzuki},
booktitle={The Thirteenth International Conference on Learning Representations},
year={2025},
url={https://openreview.net/forum?id=gx1wHnf5Vp}
}
Model Card Authors
The names are listed in alphabetical order.
Hirokazu Kiyomaru, Takashi Kodama and Taishi Nakamura.
