Views
No views yet
"The OpenThoughts3 traces used in the Phase 2 reasoning stage were already present in MV pretraining, so the unique gain may reflect amplification of seen data rather than general receptivity, and the ablation removes the whole reasoning subset rather than that overlap."
MV-noinstruct ablation removes MixtureVitae's entire instruction/reasoning
subset (175.1B tokens, 42% of the corpus), so it cannot isolate the OT3 pretrain/post-train
overlap. This model removes only OpenThoughts3 — 18.69B tokens, 4.53% of MV — keeping
everything else, including the multilingual shards.| base | Phase 2 | tests |
|---|---|---|
| MV (full) | OT3 30k | published result |
| this model | OT3 30k | is the gain OT3 re-exposure, or general reasoning receptivity? |
open_thoughts-0/1/2.
Weights proportional to shard size, so every kept shard retains its natural share: this is
plain MV minus OT3, not a reweighted mixture. 393.92B tokens available; 100B consumed.open-sci-ref architecture, GPT-NeoX tokenizer. 100B tokens, 23,842 iters,
seq len 4096, RoPE theta 100k, WSD schedule, lr 4e-3, 20% linear cooldown, global batch 1024,
warmup 1000. 64 nodes x 4 GH200. Hyperparameters identical to the companion
nemotron-hq[-mv_reasoning]-100B-4096 models and to MV-100BT, so all are comparable.Note: not labelledv0.02— the published open-sci-ref v0.02 recipe uses global batch 1008 and warmup 25000, and exists only at 300B/1T budgets. This follows the project's 100B convention instead.
modeling_opensci.py: OpensciModel.forward accepts **kwargs. Recent transformers
passes num_items_in_batch from Trainer for loss averaging, and OpensciForCausalLM.forward
forwards **kwargs into both self.model(...) and self.loss_function(...). Without this,
training raises TypeError: unexpected keyword argument 'num_items_in_batch'.config.json: max_position_embeddings cast to int. YaRN writes it as a float
(16384.0), and lm-eval uses it as a slice index -> TypeError: slice indices must be integers. Affects the 16k variant.| task | wo_ot3 | MV-100BT (full) |
|---|---|---|
| MMLU | 31.93 | 32.47 |
| HellaSwag | 48.90 | 48.46 |
| ARC-C / ARC-E | 35.32 / 66.20 | 34.39 / 64.60 |
| PIQA | 68.12 | 68.44 |
| BoolQ | 67.43 | 70.86 |
| WinoGrande | 57.38 | 57.70 |
| OBQA | 35.20 | 33.80 |
| COPA | 68.00 | 67.00 |
| LAMBADA | 44.96 | 43.82 |
| CommonsenseQA | 35.05 | 31.45 |
| Social-IQA | 42.94 | 42.22 |
| MEAN | 50.12 | 49.60 |
MV-noinstruct, which removes 42% of the
corpus and does degrade.