inclusionAI/

Ling-3.0-tiny-base-midtrain

Introduction

• Pretrained checkpoint have completed large-scale pretraining but have not undergone mid-training, WSM merging (or learning-rate decay), or post-training.
• Mid-trained checkpoint have completed mid-training but have not undergone WSM merging (or learning-rate decay) or post-training.
• Merged checkpoints have undergone WSM merging (or learning-rate decay) based on the mid-training checkpoints but have not undergone post-training.

Model Overview

Key features

• Highly sparse (1/64) MoE architecture: 512 routed experts, with only 8 routed experts and 1 shared expert activated per token. This enables broad model capabilities while activating just 5.1B (Non-emb) parameters per token;
• Native hybrid linear attention: Ling-3.0 series adopt a native hybrid linear attention architecture from the very start of pretraining by combining KDA with Gated MLA to enable efficient processing of long-context inputs.
• Warmup-Stable and Merge: We replace conventional learning-rate decay with weighted checkpoint merging. By eliminating the decay phase, our Base Model is better suited for continual pretraining and dynamic data expansion, while enabling offline exploration of different decay profiles without rerunning costly experiments for each strategy.
• Scale Seamlessly: Ling-3.0-tiny-base and Ling-3.0-flash-base share the same training recipe, enabling community to experiment on the Ling-3.0-tiny-base first and then scale validated training strategies to the larger Ling-3.0-flash-base.

Model Type	Base (final checkpoint of mid-training)
Architecture	Hybrid-linear MoE
Parameter Scale	Totoal 7.9B, Activated 1.3B
Transformer Layers	18 KDA + 6 Gated MLA (3:1)
Number of Dense Layers	1
Number of Routed Experts	128
Number of Shared Experts	1
Number of Activated Experts	8
Attention Heads	16
Hidden Size	1536
Expert Intermediate Size	512
Dense Intermediate Size	4608
Vocabulary Size	157,184

Base Model Evaluation

Intended Use

• Continued pre-training
• Mid-training
• Supervised fine-tuning for domain adaptation
• Preference optimization and RL post-training Distillation research
• Long-context and MoE systems research

• Direct end-user chat deployment
• Safety-critical applications without additional alignment and evaluation
• Production use without post-training and task-specific validation

Usage

FAQ

License