This repository contains a structured pruned variant of DeepSeek-V2-Lite using the MoNE (Mixture-of-Novice Experts) framework proposed in our paper.
*## Model Overview
Base Model: DeepSeek-V2-Lite
Method: MoNE structured expert pruning
Remaining Experts: 48
Calibration Set: c4-1000
Architecture: Mixture-of-Experts (MoE)
Framework: Transformers-compatible
This checkpoint replaces redundant experts with lightweight novice experts via structured pruning, aiming to reduce compute while preserving performance.
Paper
Title:MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE Authors: Geng Zhang, Yuxuan Han, Yuxuan Lou, Yiqi Zhang, Wangbo Zhao, Yang You arXiv: arXiv:2507.00390
