Qwen3-Swallow v0.2 is a family of large language models available in 8B, 30B-A3B, and 32B parameter sizes. Built as bilingual Japanese-English models, they were developed through Continual Pre-Training (CPT), Supervised Fine-Tuning (SFT), and Reinforcement Learning with Verifiable Rewards (RLVR) based on the Qwen3 [Yang, 2025].
In addition to enhancing Japanese language proficiency and Japanese-English translation capabilities, we significantly improved or maintained performance on math and coding tasks during CPT by using high-quality math and code datasets with reasoning traces, along with custom-built datasets during SFT. Subsequently, we further improved the models' math and coding performance by enhancing reasoning capabilities through RLVR.
Feb 23, 2026: We made the GPTQ-quantized models private due to significant performance degradation. Please use the AWQ-quantized model instead.
HF Model Family
We are releasing nine Qwen3-Swallow models: three each for CPT, SFT, and RL models.
Quantized versions of RL models are also available.
The complete list is as follows:
Quantized model performance is coming soon.
For comprehensive details on the evaluation tasks and the original model scores, please refer to the Swallow LLM Leaderboard.
Once the server is running, you can send requests using the OpenAI-compatible API:
python
1from openai import OpenAI
2model_name ="tokyotech-llm/Qwen3-Swallow-32B-RL-v0.2-AWQ-INT4"3client = OpenAI(4 base_url="http://localhost:8000/v1",5 api_key="EMPTY"6)7result = client.chat.completions.create(8 model=model_name,9 messages=[10{"role":"user","content":"Create a casual one-day Tokyo itinerary in Japanese."}11],12 max_tokens=4096,13 temperature=0.6,14 top_p=0.95,15 extra_body={16"top_k":20,17"min_p":0,18}19)20print("Reasoning:")21print(result.choices[0].message.reasoning)22print("
23Response:")24print(result.choices[0].message.content)
Best Practices
We recommend specifying following generation parameters: Temperature=0.6, TopP=0.95, TopK=20, and MinP=0, which are the default values specified in generation_config.json.
You may omit manually specifying these parameters when using inference frameworks or clients that respect generation_config.json by default.
We also recommend specifying a max context length of 32,768 or less.
Unvalidated use cases
Tool Use: Please note that Qwen3-Swallow has not been explicitly trained for Tool Use (Function Calling). Users wishing to utilize Tool Use capabilities should consider performing custom post-training.
Reasoning Toggle: Qwen3-Swallow does not support toggling a "Reasoning ON/OFF" feature. Both the SFT and RL models are designed to perform reasoning by default.
Training Datasets
CPT (Continual Pre-Training)
The following datasets were used for Continual Pre-Training (CPT).
Training was conducted using NVIDIA Megatron-LM with a context size of 32K (32,768) over a total of 209.7 billion tokens.
The following datasets were used for Supervised Fine-Tuning (SFT).
SFT was conducted using NVIDIA Megatron-LM with a context size of 32K (32,768). The training dataset sizes were as follows: 2.1M samples for 8B model, and 1.1M samples for both 30B-A3B and 32B models.
The following datasets were used for RLVR.
RLVR was conducted using slime. During RL training, the maximum number of output tokens was set to 24,576 (input prompt tokens are not included).
We provide AWQ-INT4 quantized variants of the RL models.
We generated outputs from RL dataset prompts, validated them, and used only complete, well-formed generations (excluding poor or incomplete outputs) for quantization calibration.
Risks and Limitations
The models released here are still in the early stages of our research and development and have not been tuned to ensure outputs align with human intent and safety considerations.
Acknowledgements
We thank Qwen Team for releasing Qwen3 under a generous open license.
This work is based on results obtained from a project, JPNP25006, subsidized by the New Energy and Industrial Technology Development Organization (NEDO).
This work was supported by the "R&D Hub Aimed at Ensuring Transparency and Reliability of Generative AI Models" project of the Ministry of Education, Culture, Sports, Science and Technology.
We used ABCI 3.0 provided by AIST and AIST Solutions with support from "ABCI 3.0 Development Acceleration Use".
This study was carried out using the TSUBAME4.0 supercomputer at Institute of Science Tokyo.
If you find our work helpful, please feel free to cite these papers.
The Qwen3-Swallow and GPT-OSS-Swallow Technical Paper (Training Details) will be released in March.
Continual Pre-Training
@inproceedings{
fujii2024continual,
title={Continual Pre-Training for Cross-Lingual {LLM} Adaptation: Enhancing Japanese Language Capabilities},
author={Kazuki Fujii and Taishi Nakamura and Mengsay Loem and Hiroki Iida and Masanari Ohi and Kakeru Hattori and Hirai Shota and Sakae Mizuki and Rio Yokota and Naoaki Okazaki},
booktitle={First Conference on Language Modeling},
year={2024}
}
Supervised Fine-Tuning
@inproceedings{
ma2025building,
title={Building Instruction-Tuning Datasets from Human-Written Instructions with Open-Weight Large Language Models},
author={Youmi Ma and Sakae Mizuki and Kazuki Fujii and Taishi Nakamura and Masanari Ohi and Hinari Shimada and Taihei Shiotani and Koshiro Saito and Koki Maeda and Kakeru Hattori and Takumi Okamoto and Shigeki Ishida and Rio Yokota and Hiroya Takamura and Naoaki Okazaki},
booktitle={Second Conference on Language Modeling},
year={2025}
}
