Base models are trained with pre-training and mid-training only.
Post-trained models are aligned using supervised fine-tuning (SFT) and direct preference optimization (DPO), without reinforcement learning.
For practical usage examples and detailed instructions on how to use the models, please also refer to our cookbook.
To support the continued development of LLM-jp, we would greatly appreciate it if you could share how you utilize LLM-jp outcomes via the survey form.
Usage
Please refer to our cookbook for practical usage examples and detailed instructions on how to use the models.
Model Details
Model type: Transformer-based Language Model
Architectures:
Dense model:
Params
Layers
Hidden size
Heads
Context length
Embedding parameters
Non-embedding parameters
Total parameters
8B
32
4,096
32
65,536
805,306,368
7,784,894,464
8,590,200,832
MoE model:
Params
Layers
Hidden size
Heads
Routed Experts
Activated Experts
Context length
Embedding parameters
Non-embedding parameters
Activated parameters
Total parameters
32B-A3B
32
2,560
40
128
8
65,536
503,316,480
31,635,712,512
3,827,476,992
32,139,028,992
Tokenizer
The tokenizer of this model is based on huggingface/tokenizers Unigram byte-fallback model.
The vocabulary entries were converted from llm-jp-tokenizer v4.0.
Please refer to README.md of llm-jp-tokenizer for details on the vocabulary construction procedure (the pure SentencePiece training does not reproduce our vocabulary).
[!NOTE]
The chat template of this model is designed to be compatible with the OpenAI Harmony response format.
However, the tokenizer differs from the one assumed by the openai-harmony library, and therefore direct tokenization with openai-harmony is not supported.
For correct behavior, please use the tokenizer provided with this model. For detailed usage, please refer to our cookbook.
Training
Pre-training
This model is trained through a multi-stage pipeline consisting of pre-training and mid-training phases, using a total of 11.7T tokens.
pretraining_overview
The corpora used for pre-training and mid-training are publicly available at the following links:
We evaluated the model on a variety of tasks using an LLM-as-a-Judge framework. The descriptions of each task are as follows.
MT-Bench (JA/EN): A benchmark for measuring multi-turn conversational task-solving ability.
AnswerCarefully: A benchmark for evaluating safety in Japanese. We used 336 questions from the v2.0 test set.
llm-jp-instructions: A set of human-created single-turn question–answer pairs. We used 400 questions from the test set.
We evaluated the models using gpt-5.4-2026-03-05.
[!NOTE]
Note: In earlier evaluations of the llm-jp-3 series, we used gpt-4o-2024-08-06. The newer evaluator gpt-5.4-2026-03-05 provides a stricter and more reliable assessment, which results in lower scores on benchmarks such as MT-Bench compared to those reported for the llm-jp-3 series.
The scores represent the average values obtained from three rounds of inference and evaluation.
For more details, please refer to the codes.
The models released here are in the early stages of our research and development and have not been tuned to ensure outputs align with human intent and safety considerations.
