You can deploy this model and start earning money today!

amd/
Instella-MoE-16B-A3B-Pretrain

Name: Instella-MoE-16B-A3B-Pretrain
Availability: InStock

text-generation-inference

endpoints_compatible

Views

No views yet

Instella-MoE✨: Fully Open State-of-the-Art Mixture-of-Experts Language Model

Instella-MoE is a state-of-the-art fully open Mixture-of-Experts (MoE) language model with 16 billion total parameters and 2.8 billion active parameters trained end-to-end from pre-training to RL. Trained from scratch on AMD Instinct™ MI300X and MI325X GPUs using AMD's Primus framework, Instella-MoE combines a sparsely activated MoE design with architectural innovations such as Gated Multi-head Latent Attention (Gated MLA) and FarSkip-Collective.

Figure 1: Pre-trained and Post-trained Instella-MoE model performance compared with other similar size state-of-the-art models.

Takeaways

Instella-MoE is a new state-of-the-art fully open Mixture-of-Experts language model developed by AMD, with 16 billion total parameters and 2.8 billion active parameters per token, trained from scratch on AMD Instinct™ MI300X and MI325X GPUs.
The Instella-MoE model checkpoint release spans every stage of the model training pipeline, including pre-training, mid-training, long-context extension, SFT, DPO and RL.
Built entirely on the AMD ROCm™ software stack on top of the Primus training and Miles RL frameworks, Instella-MoE incorporates cutting-edge architecture and systems innovations—including Gated Multi-head Latent Attention (Gated MLA) and extreme communication-computation overlap through FarSkip-Collective—for efficient large-scale training and inference on AMD hardware.
Fully open and accessible: we provide our complete training recipe across all training stages, including training frameworks, data mixtures, intermediate checkpoints and inference code.

The release includes checkpoints from the following Instella-MoE training pipeline stages as shown in Table 1 below:

Model	Stage	Description
Instella-MoE-16B-A3B-Pretrain (Link)	Pre-training	MoE base model trained from scratch on a large and diverse training corpus.
Instella-MoE-16B-A3B-Midtrain (Link)	Mid-training	Pretrained model further trained on high-quality data mixtures to refine key capabilities.
Instella-MoE-16B-A3B-Base (Link)	Long-context	Long-context training to extend the model’s ability to process and reason over longer sequences. We use this as our final base checkpoint.
Instella-MoE-16B-A3B-SFT (Link)	SFT	Base checkpoint extended via supervised fine-tuning (SFT) to enable instruction following and chain-of-thought reasoning capabilities.
Instella-MoE-16B-A3B-DPO (Link)	DPO	Direct preference optimization (DPO) on contrastive preference data to improve model performance.
Instella-MoE-16B-A3B-Think (Link)	RL	Final thinking checkpoint refined with reinforcement learning (RL) to further strengthen instruction following and overall response quality.

Table 1: Instella-MoE-16B-A3B models and training stages.

Model Summary

Parameter	Value
Total Parameters	16B
Active Parameters per Token	2.8B
Decoder Layers	27
Hidden Size	2048
Attention Heads	16
Number of Experts	64
Shared Experts	2
Activated Experts per Token	6
Vocabulary Size	128,896
Attention	Gated Multi-head Latent Attention (Gated MLA)
MoE Connectivity	FarSkip-Collective

Results

Pretraining Results

Table 2: Instella-MoE-16B-A3B-Base Results on Standard Benchmarks.

Table 3: Instella-MoE-16B-A3B-Base Results on Long Context HELMET and RULER Benchmarks.

Post-training Results

Table 4: Instella-MoE-Think results. We evaluate all models using the OLMES framework, generating up to a maximum of 32768 tokens.

Getting Started

Example Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "amd/Instella-MoE-16B-A3B-Think"

tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto", trust_remote_code=True)

prompt = [{"role": "user", "content": "What are the computational benefits of Mixture-of-Experts models?"}]
inputs = tokenizer.apply_chat_template(
    prompt,
    add_generation_prompt=True,
    return_tensors='pt'
)

tokens = model.generate(
    inputs.to(model.device),
    max_new_tokens=1024,
    temperature=0.6,
    top_p=0.95,
    do_sample=True
)

print(tokenizer.decode(tokens[0], skip_special_tokens=False))

For high-throughput inference with SGLang, please refer to the setup and usage instructions in our GitHub repository.

Training Details

Instella-MoE was trained end-to-end on AMD Instinct™ MI300X and MI325X GPUs using the AMD ROCm™ software stack, built on top of the Primus training framework and the Miles RL framework. Training proceeds through a multi-stage pipeline—pre-training, mid-training, long-context extension, SFT, DPO, and RL—with each stage progressively strengthening the model's capabilities.

For the complete training recipe, including per-stage data mixtures, hyperparameters, training frameworks, and inference code, please refer to our GitHub repository and technical blog.

Acknowledgements

We are deeply grateful to the LLM360 team and the Miles team for their invaluable support throughout the development of our model.

License

The Instella-MoE models are licensed for academic and research purposes under a ResearchRAIL license.
Refer to the LICENSE for more information.

Bias, Risks, and Limitations

The models are being released for research purposes only. They are not intended for use cases requiring high levels of factual accuracy, safety-critical applications, or health and medical applications. They must not be used to generate false information or facilitate toxic conversations.
Model checkpoints are made accessible without any safety promises. It is crucial for users to conduct comprehensive evaluations and implement safety filtering mechanisms as per their respective use cases.
It may be possible to prompt the model to generate content that may be factually inaccurate, harmful, violent, toxic, biased, or otherwise objectionable. Such content may also be generated in response to prompts that were not intended to elicit it. Users are thus requested to be aware of this and exercise caution and responsible thinking when using the model.
The multilingual abilities of the models have not been tested and thus may misunderstand and generate erroneous responses across different languages.

Contributors

Core contributors: Jiang Liu, Sudhanshu Ranjan, Prakamya Mishra, Yonatan Dukler, Gowtham Ramesh, Zicheng Liu

Contributors: Jialian Wu, Ximeng Sun, Wen Xie, Chaojun Hou, Vikram Appia, Zhenyu Gu, Emad Barsoum

Citations

If you use Instella-MoE, please cite:

text

@article{instella-moe,
  title={Instella-MoE Technical Report},
  author={Liu, Jiang and Ranjan, Sudhanshu and Mishra, Prakamya and Dukler, Yonatan and Ramesh, Gowtham and Wu, Jialian and Sun, Ximeng and Xie, Wen and Hou, Chaojun and Appia, Vikram and Gu, Zhenyu and Liu, Zicheng and Barsoum, Emad},
  journal={arXiv preprint arXiv:2609.00791},
  year={2026}
}

@inproceedings{
dukler2026farskipcollective,
title={FarSkip-Collective: Unhobbling Blocking Communication in Mixture of Experts Models},
author={Yonatan Dukler and Guihong Li and Deval Shah and Jiang Liu and Vikram Appia and Emad Barsoum},
booktitle={Ninth Conference on Machine Learning and Systems},
year={2026},
url={https://openreview.net/forum?id=ruOpvLzsGV}
}

amd/Instella-MoE-16B-A3B-Pretrain