This model is a mixture of experts (MoE) using the RhuiDih/moetify library with various task-specific experts. All relevant expert models, LoRA adapters, and datasets are available at Moecule Ingredients.

Key Features

Zero Additional Training: Combine existing domain-specific / task-specific experts into a powerful MoE model without additional training!

System Requirements

Steps	System Requirements
MoE Creation	> 44.8 GB System RAM
Inference (fp16)	GPU with > 12.4 GB VRAM

MoE Creation

To reproduce this model, run the following command:

shell

# git clone moetify fork that fixes dependency issue
!git clone -b fix-transformers-4.47.1-FlashA2-dependency --single-branch https://github.com/davzoku/moetify.git

!cd moetify && pip install -e .

python -m moetify.mix \
 --output_dir ./moecule-2x3b-m11-fk \
 --model_path unsloth/llama-3.2-3b-Instruct \
 --modules mlp q_proj \
 --ingredients \
 davzoku/finqa_expert_3b \
 davzoku/kyc_expert_3b

Model Parameters

shell

INFO:root:Stem parameters: 1228581888
INFO:root:Experts parameters: 4756340736
INFO:root:Routers parameters: 344064
INFO:root:MOE total parameters (numel): 5985266688
INFO:root:MOE total parameters : 5985266688
INFO:root:MOE active parameters: 5985266688

Inference

To run an inference with this model, you can use the following code snippet:

python

# git clone moetify fork that fixes dependency issue
!git clone -b fix-transformers-4.47.1-FlashA2-dependency --single-branch https://github.com/davzoku/moetify.git

!cd moetify && pip install -e .

model = AutoModelForCausalLM.from_pretrained(<model-name>, device_map='auto', trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(<model-name>)

def format_instruction(row):
    return f"""### Question: {row}"""

greedy_generation_config = GenerationConfig(
    temperature=0.1,
    top_p=0.75,
    top_k=40,
    num_beams=1,
    max_new_tokens=128,
    repetition_penalty=1.2
)


input_text = "In what ways did Siemens's debt restructuring on March 06, 2024 reflect its strategic priorities?"
formatted_input = format_instruction(input_text)
inputs = tokenizer(formatted_input, return_tensors="pt").to('cuda')

with torch.no_grad():
    outputs = model.generate(
        input_ids=inputs.input_ids,
        attention_mask=inputs.attention_mask,
        generation_config=greedy_generation_config
    )

generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)

The Team

CHOCK Wan Kee
Farlin Deva Binusha DEVASUGIN MERLISUGITHA
GOH Bao Sheng
Jessica LEK Si Jia
Sinha KHUSHI
TENG Kok Wai (Walter)

davzoku/moecule-2x3b-m11-fk

🫐 Moecule 2x3B M11 FK

Model Details