An experimental Mixture of Experts (MoE) model combining four specialized Llama 3.2 3B fine-tunes into a single 12B parameter model with ~3B active parameters per token.

For more information, please see theprint/theprint-12B-MoE-3A.