Base Model: Just Merged ~ No Training Gates After Merge
Model Overview
I have developed a Mixture of Experts (MoE) architecture with two always-active experts designed to work together for Python instruction tuning. Each expert possesses a distinct skill:
- Expert 1: Specializes in generating Mermaid diagrams, primarily from Python code, which requires a deep understanding of code structures and logic.
- Expert 2: Focuses on strict context obedience, ensuring that the model only generates outputs based on the provided instructions.
Why Always-Active MoE is Optimal
In this model, both experts are always active for each token, allowing them to complement each other:
- Expert 1’s competence in Python structures enhances the model's ability to generate correct and structured Python code.
- Expert 2’s context obedience ensures that the output remains aligned with the user’s instructions, preventing unnecessary or irrelevant outputs, such as Mermaid diagrams, unless explicitly requested.
This setup allows me to efficiently train the model for Python instruction following. By leveraging both experts simultaneously, I ensure that the model generates syntactically correct Python code while strictly adhering to user prompts.
