GPT-OSS-Swallow v0.1 is a family of large language models available in 20B and 120B parameter sizes. Built as bilingual Japanese-English models, they were developed through Continual Pre-Training (CPT), Supervised Fine-Tuning (SFT), and Reinforcement Learning with Verifiable Rewards (RLVR) based on the GPT-OSS [OpenAI, 2025].
This repository provides the 20B RL model in MXFP4 format.
In addition to enhancing Japanese language proficiency and Japanese-English translation capabilities, we minimized the performance degradation in math and coding tasks—often seen during CPT—by utilizing high-quality math and code datasets alongside custom-built data during SFT. Subsequently, we further enhanced the models' math and coding capabilities through RLVR, achieving performance comparable to or surpassing GPT-OSS.
We are releasing four GPT-OSS-Swallow models: two SFT models and two RL models (excluding CPT models). MXFP4 variants of the RL models are also available.
The complete list is as follows:
Model type: Please refer to gpt-oss model card for details on the model architecture.
Language(s): Japanese, English
Tokenizer: Please refer to gpt-oss model card for details on the tokenizer.
Contact: swallow[at]nlp.c.titech.ac.jp
Model Performance
For comprehensive details on the evaluation tasks and the resulting scores, please refer to the Swallow LLM Leaderboard.
[!IMPORTANT]
The evaluation scores for gpt-oss and gpt-oss-swallow were measured with the reasoning effort set to medium.
The following results were measured with stochastic inference (temperature=0.6, top_p=0.95).
Japanese tasks
Japanese Performance
English tasks
English Performance
Usage
vLLM
[!TIP]
This model has been primarily developed and evaluated using vLLM.
For the most reliable and reproducible behavior, we strongly recommend running inference with vLLM.
vLLM recommends using uv to manage the Python environment.
Once the server is running, you can send requests using the OpenAI-compatible API:
python
1from openai import OpenAI
23# Note: Replace with the actual model path/name you are using4model_name ="tokyotech-llm/GPT-OSS-Swallow-20B-RL-v0.1-MXFP4"56client = OpenAI(7 base_url="http://localhost:8000/v1",8 api_key="EMPTY"9)1011result = client.chat.completions.create(12 model=model_name,13 messages=[14{"role":"user","content":"Create a casual one-day Tokyo itinerary in Japanese."}15],16 max_tokens=4096,17 temperature=0.6,18 top_p=0.95,19 extra_body={20"top_k":20,21"min_p":0,22}23)2425print("Reasoning:")26print(result.choices[0].message.reasoning)27print("\nResponse:")28print(result.choices[0].message.content)
There is no default system message.
Best Practices
We recommend specifying following generation parameters: Temperature=0.6, TopP=0.95, TopK=20, and MinP=0, which are the default values specified in generation_config.json.
You may omit manually specifying these parameters when using inference frameworks or clients that respect generation_config.json by default.
We also recommend specifying a max context length of 32,768 or less.
Unvalidated use cases
GPT-OSS-Swallow may not be suitable for the following use cases or features.
Tool Use (Function Calling): We did not explicitly train the models for tool use. Users who wish to leverage function-calling capabilities will need to perform custom post-training.
Model Identity: Our training recipe does not account for the "model identity" parameter in the chat template. The model may not consistently identify itself as a specific version ("You are ChatGPT, a large language model trained by OpenAI.").
Reasoning Effort Control: We did not train the model with variations in the "reasoning effort" parameter. For stable results, we strongly recommend keeping the reasoning effort set to medium during inference.
Long Context: We did not explicitly train the models beyond 32k tokens or evaluate performance on long-context tasks, although the model supports context length extension using YaRN, following the original GPT-OSS models.
Training Datasets
CPT (Continual Pre-Training)
The following datasets were used for Continual Pre-Training (CPT).
Training was conducted using NVIDIA NeMo with a context size of 32K (32,768) over a total of 419.4 billion tokens.
The following datasets were used for Supervised Fine-Tuning (SFT). These datasets cover general chat in Japanese and English (GPT-OSS-LMSYS), as well as math, coding, and science domains (Swallow-Nemotron). The reasoning traces and assistant responses in these datasets were generated using gpt-oss-120b.
SFT was conducted using NVIDIA Automodel with a context size of 32K (32,768). The total training dataset size was 1.1M samples.
The following datasets were used for RLVR.
RLVR was conducted using slime, with its codebase adapted for GPT-OSS support. During RL training, the maximum number of output tokens was set to 24,576 (input prompt tokens are not included).
This repository provides the RL model in MXFP4 format, following the native GPT-OSS weight format.
Risks and Limitations
The models released here are still in the early stages of our research and development and have not been tuned to ensure outputs align with human intent and safety considerations.
Acknowledgements
We thank the OpenAI Team for releasing GPT-OSS under a generous open license.
This work is based on results obtained from AIST policy-based budget project "R&D on Generative AI Foundation Models for the Physical Domain".
This work was supported by the “R&D Hub Aimed at Ensuring Transparency and Reliability of Generative AI Models” project of the Ministry of Education, Culture, Sports, Science and Technology.
We used ABCI 3.0 provided by AIST and AIST Solutions with support from "ABCI 3.0 Development Acceleration Use".
This study was carried out using the TSUBAME4.0 supercomputer at Institute of Science Tokyo.
If you find our work helpful, please feel free to cite these papers.
Continual Pre-Training
@inproceedings{
fujii2024continual,
title={Continual Pre-Training for Cross-Lingual {LLM} Adaptation: Enhancing Japanese Language Capabilities},
author={Kazuki Fujii and Taishi Nakamura and Mengsay Loem and Hiroki Iida and Masanari Ohi and Kakeru Hattori and Hirai Shota and Sakae Mizuki and Rio Yokota and Naoaki Okazaki},
booktitle={First Conference on Language Modeling},
year={2024}
}
Supervised Fine-Tuning
@inproceedings{
ma2025building,
title={Building Instruction-Tuning Datasets from Human-Written Instructions with Open-Weight Large Language Models},
author={Youmi Ma and Sakae Mizuki and Kazuki Fujii and Taishi Nakamura and Masanari Ohi and Hinari Shimada and Taihei Shiotani and Koshiro Saito and Koki Maeda and Kakeru Hattori and Takumi Okamoto and Shigeki Ishida and Rio Yokota and Hiroya Takamura and Naoaki Okazaki},
booktitle={Second Conference on Language Modeling},
year={2025}
}
