This is the retained 200M WSD/OPD milestone from the MiMo second-model arm of the full-attention-to-Gated-DeltaNet conversion study. It contains a 1:4 uniform hybrid of Qwen2 attention and Gated DeltaNet layers.

Actual consumed generation/loss tokens: 200,049,316
Training rollout horizon: 16,384
Model context window: 65,536 tokens
RULER multikey at 32K: 85.00%
RULER health gate: healthy (n=100)

Evaluation

Metric	Score
AIME 2024 accuracy / empirical pass@1	35.42%
AIME 2024 pass@8	63.33%
AIME 2025 accuracy / empirical pass@1	28.75%
AIME 2025 pass@8	46.67%
MATH-500 thinking	88.20%
MATH-500 no-thinking	64.40%
GSM8K strict	58.68%
MMLU 5-shot	53.64%
NIAH multikey 16K	97.00%
NIAH multikey 32K	85.00%

The repository includes the compact raw evaluation JSONs under full_eval/. Generation dumps and optimizer/trainer recovery state are intentionally omitted.

Transformers usage

The custom architecture implementation and auto_map are bundled, so the checkpoint can be loaded without access to the private training repository:

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "arianraje/mimo-7b-gdn-hybrid-200M-OPD"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

The model requires the pinned Qwen3-Next GDN implementation in Transformers and Flash Linear Attention; see requirements.txt. For vLLM 0.11.2, download the repository snapshot, put it on sys.path, call vllm_mimo_gdn.register(), then construct vllm.LLM from that snapshot.

Limitations

This is a research checkpoint using custom recurrent linear-attention kernels. It has not been safety-tuned beyond the inherited MiMo behavior. Long-context quality varies by task; consult the included RULER cells rather than treating the nominal context window as a uniform quality guarantee.