模型继承了YModel1.1的自蒸馏结构，在层间设置余弦相似度损失，使得模型倾向于将知识压缩到浅层。
模型将并行的线性层融合，经二阶优化器实验显示模型最终效果不会与不融合的情况更差。
模型采用最强大的SiMuon优化器训练。SiMuon是从Muon（ https://kellerjordan.github.io/posts/muon ）改进得到的（ https://www.arxiv.org/abs/2507.11005 ）。这个模型使用的SiMuon与原始的Muon以下区别：1：在执行NS操作的l2 norm前，执行sign操作。2：将NS迭代从5步减少到2步。
模型的tokenlizer与词嵌入层使用的是预训练权重，来自MiniMind2（ https://github.com/jingyaogong/minimind ）
模型在预训练阶段训练了0.4B token，在微调阶段训练了2B token。
模型在512长度上进行预训练，学习率为1e-3（SiMuon，使用0.2*sqrt(max(fan in, fan out))进行学习率缩放）与1e-4（AdamW）。采用bf16 amp加速。
模型在512/1e-5、1024/3e-6、2048/1e-6、2048/5e-7（长度/学习率）上进行sft微调。采用bf16 amp加速。

Model Description

YModel2 is the most powerful Large Language Model (LLM) trained by SnifferCaptain to date (as of 11/23/2025). It demonstrates significant improvements in inference speed, mathematical capabilities, coding abilities, and common sense reasoning compared to the YModel1.x versions.

Model Details

The model incorporates optimization ideas from MFA ( https://arxiv.org/abs/2412.19255 ), upgrading PEGA (Position Embedding Gate Attention) to PEGA2. This new version achieves performance on par with or even surpassing the original PEGA, while delivering nearly a 3x speedup.
The model utilizes GeGLU in its Feed-Forward Network (FFN) blocks.

Training Details

The model inherits the self-distillation structure from YModel1.1, employing an inter-layer cosine similarity loss that encourages the model to compress knowledge into shallower layers.
Parallel linear layers have been fused. Experiments with second-order optimizers indicate that this fusion does not degrade the model's final performance compared to the non-fused counterpart.
The model was trained using the powerful SiMuon optimizer, an improved version of Muon ( https://kellerjordan.github.io/posts/muon ) as detailed in ( https://www.arxiv.org/abs/2507.11005 ). The SiMuon implementation used in this model differs from the original Muon in the following ways:
1. A sign operation is applied before the L2 norm in the NS iteration.
2. The number of NS iterations is reduced from 5 to 2.
The model's tokenizer and embedding layers are initialized with pre-trained weights from MiniMind2 ( https://github.com/jingyaogong/minimind ).
The model was trained on 0.4 billion tokens during the pre-training phase and 2 billion tokens during the fine-tuning phase.
Pre-training was conducted at a sequence length of 512 with a learning rate of 1e-3 (for SiMuon, scaled using 0.2*sqrt(max(fan_in, fan_out))) and 1e-4 (for AdamW). The process was accelerated using bf16 AMP.
For Supervised Fine-Tuning (SFT), the model was trained at the following sequence length and learning rate combinations: 512/1e-5, 1024/3e-6, 2048/1e-6, and 2048/5e-7 (Length/LR). This stage was also accelerated with bf16 AMP.

模型性能 Model Performance

模型在多个数据集上跑分，仅供娱乐参考：

模型跑分结果如下，使用lm_eval框架：

Groups	Version	Filter	n-shot	Metric		Value		Stderr
ceval-valid	2	none	0	acc	↑	0.2303	±	0.0115
cmmlu	1	none	5	acc	↑	0.2515	±	0.004

ceval bench result

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
ceval-valid	2	none		acc	↑	0.2303	±	0.0115
ceval-valid_accountant	2	none	0	acc	↑	0.2245	±	0.0602
ceval-valid_advanced_mathematics	2	none	0	acc	↑	0.3158	±	0.1096
ceval-valid_art_studies	2	none	0	acc	↑	0.4545	±	0.0880
ceval-valid_basic_medicine	2	none	0	acc	↑	0.0526	±	0.0526
ceval-valid_business_administration	2	none	0	acc	↑	0.2424	±	0.0758
ceval-valid_chinese_language_and_literature	2	none	0	acc	↑	0.2174	±	0.0879
ceval-valid_civil_servant	2	none	0	acc	↑	0.2553	±	0.0643
ceval-valid_clinical_medicine	2	none	0	acc	↑	0.2273	±	0.0914
ceval-valid_college_chemistry	2	none	0	acc	↑	0.1667	±	0.0777
ceval-valid_college_economics	2	none	0	acc	↑	0.2909	±	0.0618
ceval-valid_college_physics	2	none	0	acc	↑	0.2105	±	0.0961
ceval-valid_college_programming	2	none	0	acc	↑	0.2432	±	0.0715
ceval-valid_computer_architecture	2	none	0	acc	↑	0.2857	±	0.1010
ceval-valid_computer_network	2	none	0	acc	↑	0.1053	±	0.0723
ceval-valid_discrete_mathematics	2	none	0	acc	↑	0.3750	±	0.1250
ceval-valid_education_science	2	none	0	acc	↑	0.2414	±	0.0809
ceval-valid_electrical_engineer	2	none	0	acc	↑	0.2162	±	0.0686
ceval-valid_environmental_impact_assessment_engineer	2	none	0	acc	↑	0.1613	±	0.0672
ceval-valid_fire_engineer	2	none	0	acc	↑	0.2581	±	0.0799
ceval-valid_high_school_biology	2	none	0	acc	↑	0.3684	±	0.1137
ceval-valid_high_school_chemistry	2	none	0	acc	↑	0.2105	±	0.0961
ceval-valid_high_school_chinese	2	none	0	acc	↑	0.2105	±	0.0961
ceval-valid_high_school_geography	2	none	0	acc	↑	0.2105	±	0.0961
ceval-valid_high_school_history	2	none	0	acc	↑	0.3000	±	0.1051
ceval-valid_high_school_mathematics	2	none	0	acc	↑	0.2222	±	0.1008
ceval-valid_high_school_physics	2	none	0	acc	↑	0.2105	±	0.0961
ceval-valid_high_school_politics	2	none	0	acc	↑	0.2105	±	0.0961
ceval-valid_ideological_and_moral_cultivation	2	none	0	acc	↑	0.2632	±	0.1038
ceval-valid_law	2	none	0	acc	↑	0.2500	±	0.0903
ceval-valid_legal_professional	2	none	0	acc	↑	0.0435	±	0.0435
ceval-valid_logic	2	none	0	acc	↑	0.1818	±	0.0842
ceval-valid_mao_zedong_thought	2	none	0	acc	↑	0.3333	±	0.0983
ceval-valid_marxism	2	none	0	acc	↑	0.2632	±	0.1038
ceval-valid_metrology_engineer	2	none	0	acc	↑	0.1250	±	0.0690
ceval-valid_middle_school_biology	2	none	0	acc	↑	0.1905	±	0.0878
ceval-valid_middle_school_chemistry	2	none	0	acc	↑	0.1500	±	0.0819
ceval-valid_middle_school_geography	2	none	0	acc	↑	0.0833	±	0.0833
ceval-valid_middle_school_history	2	none	0	acc	↑	0.1818	±	0.0842
ceval-valid_middle_school_mathematics	2	none	0	acc	↑	0.1579	±	0.0859
ceval-valid_middle_school_physics	2	none	0	acc	↑	0.2105	±	0.0961
ceval-valid_middle_school_politics	2	none	0	acc	↑	0.2857	±	0.1010
ceval-valid_modern_chinese_history	2	none	0	acc	↑	0.1739	±	0.0808
ceval-valid_operating_system	2	none	0	acc	↑	0.1579	±	0.0859
ceval-valid_physician	2	none	0	acc	↑	0.2653	±	0.0637
ceval-valid_plant_protection	2	none	0	acc	↑	0.3182	±	0.1016
ceval-valid_probability_and_statistics	2	none	0	acc	↑	0.1111	±	0.0762
ceval-valid_professional_tour_guide	2	none	0	acc	↑	0.3448	±	0.0898
ceval-valid_sports_science	2	none	0	acc	↑	0.1579	±	0.0859
ceval-valid_tax_accountant	2	none	0	acc	↑	0.1633	±	0.0533
ceval-valid_teacher_qualification	2	none	0	acc	↑	0.2955	±	0.0696
ceval-valid_urban_and_rural_planner	2	none	0	acc	↑	0.2174	±	0.0615
ceval-valid_veterinary_medicine	2	none	0	acc	↑	0.2174	±	0.0879

cmmlu bench result

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
cmmlu	1	none		acc	↑	0.2515	±	0.0040
- cmmlu_agronomy	1	none	5	acc	↑	0.2544	±	0.0336
- cmmlu_anatomy	1	none	5	acc	↑	0.2432	±	0.0354
- cmmlu_ancient_chinese	1	none	5	acc	↑	0.2317	±	0.0330
- cmmlu_arts	1	none	5	acc	↑	0.2562	±	0.0346
- cmmlu_astronomy	1	none	5	acc	↑	0.2485	±	0.0337
- cmmlu_business_ethics	1	none	5	acc	↑	0.2344	±	0.0294
- cmmlu_chinese_civil_service_exam	1	none	5	acc	↑	0.2562	±	0.0346
- cmmlu_chinese_driving_rule	1	none	5	acc	↑	0.2519	±	0.0381
- cmmlu_chinese_food_culture	1	none	5	acc	↑	0.2574	±	0.0376
- cmmlu_chinese_foreign_policy	1	none	5	acc	↑	0.2710	±	0.0432
- cmmlu_chinese_history	1	none	5	acc	↑	0.2508	±	0.0242
- cmmlu_chinese_literature	1	none	5	acc	↑	0.2549	±	0.0306
- cmmlu_chinese_teacher_qualification	1	none	5	acc	↑	0.2458	±	0.0323
- cmmlu_clinical_knowledge	1	none	5	acc	↑	0.2532	±	0.0283
- cmmlu_college_actuarial_science	1	none	5	acc	↑	0.3019	±	0.0448
- cmmlu_college_education	1	none	5	acc	↑	0.3178	±	0.0452
- cmmlu_college_engineering_hydrology	1	none	5	acc	↑	0.3019	±	0.0448
- cmmlu_college_law	1	none	5	acc	↑	0.1667	±	0.0360
- cmmlu_college_mathematics	1	none	5	acc	↑	0.2381	±	0.0418
- cmmlu_college_medical_statistics	1	none	5	acc	↑	0.2547	±	0.0425
- cmmlu_college_medicine	1	none	5	acc	↑	0.2234	±	0.0253
- cmmlu_computer_science	1	none	5	acc	↑	0.2500	±	0.0304
- cmmlu_computer_security	1	none	5	acc	↑	0.2632	±	0.0338
- cmmlu_conceptual_physics	1	none	5	acc	↑	0.2721	±	0.0368
- cmmlu_construction_project_management	1	none	5	acc	↑	0.2230	±	0.0354
- cmmlu_economics	1	none	5	acc	↑	0.2264	±	0.0333
- cmmlu_education	1	none	5	acc	↑	0.2454	±	0.0338
- cmmlu_electrical_engineering	1	none	5	acc	↑	0.2500	±	0.0331
- cmmlu_elementary_chinese	1	none	5	acc	↑	0.2778	±	0.0283
- cmmlu_elementary_commonsense	1	none	5	acc	↑	0.2273	±	0.0299
- cmmlu_elementary_information_and_technology	1	none	5	acc	↑	0.2731	±	0.0289
- cmmlu_elementary_mathematics	1	none	5	acc	↑	0.2304	±	0.0278
- cmmlu_ethnology	1	none	5	acc	↑	0.2519	±	0.0375
- cmmlu_food_science	1	none	5	acc	↑	0.2448	±	0.0361
- cmmlu_genetics	1	none	5	acc	↑	0.2443	±	0.0325
- cmmlu_global_facts	1	none	5	acc	↑	0.2617	±	0.0361
- cmmlu_high_school_biology	1	none	5	acc	↑	0.2485	±	0.0333
- cmmlu_high_school_chemistry	1	none	5	acc	↑	0.2273	±	0.0366
- cmmlu_high_school_geography	1	none	5	acc	↑	0.2542	±	0.0403
- cmmlu_high_school_mathematics	1	none	5	acc	↑	0.2500	±	0.0339
- cmmlu_high_school_physics	1	none	5	acc	↑	0.2545	±	0.0417
- cmmlu_high_school_politics	1	none	5	acc	↑	0.2378	±	0.0357
- cmmlu_human_sexuality	1	none	5	acc	↑	0.2619	±	0.0393
- cmmlu_international_law	1	none	5	acc	↑	0.2432	±	0.0316
- cmmlu_journalism	1	none	5	acc	↑	0.2442	±	0.0329
- cmmlu_jurisprudence	1	none	5	acc	↑	0.2506	±	0.0214
- cmmlu_legal_and_moral_basis	1	none	5	acc	↑	0.2290	±	0.0288
- cmmlu_logical	1	none	5	acc	↑	0.2602	±	0.0397
- cmmlu_machine_learning	1	none	5	acc	↑	0.2623	±	0.0400
- cmmlu_management	1	none	5	acc	↑	0.2667	±	0.0306
- cmmlu_marketing	1	none	5	acc	↑	0.2722	±	0.0333
- cmmlu_marxist_theory	1	none	5	acc	↑	0.2434	±	0.0313
- cmmlu_modern_chinese	1	none	5	acc	↑	0.2500	±	0.0404
- cmmlu_nutrition	1	none	5	acc	↑	0.2138	±	0.0342
- cmmlu_philosophy	1	none	5	acc	↑	0.2476	±	0.0423
- cmmlu_professional_accounting	1	none	5	acc	↑	0.2571	±	0.0331
- cmmlu_professional_law	1	none	5	acc	↑	0.2749	±	0.0308
- cmmlu_professional_medicine	1	none	5	acc	↑	0.2553	±	0.0225
- cmmlu_professional_psychology	1	none	5	acc	↑	0.2500	±	0.0285
- cmmlu_public_relations	1	none	5	acc	↑	0.2529	±	0.0330
- cmmlu_security_study	1	none	5	acc	↑	0.2370	±	0.0367
- cmmlu_sociology	1	none	5	acc	↑	0.2699	±	0.0296
- cmmlu_sports_science	1	none	5	acc	↑	0.2485	±	0.0337
- cmmlu_traditional_chinese_medicine	1	none	5	acc	↑	0.2595	±	0.0323
- cmmlu_virology	1	none	5	acc	↑	0.2485	±	0.0333
- cmmlu_world_history	1	none	5	acc	↑	0.2857	±	0.0357
- cmmlu_world_religions	1	none	5	acc	↑	0.2625	±	0.0349

以下是模型的问答输出：

[Q]你知道长江吗？
[A]是的，我知道长江。长江，位于中国东部沿海地区，是中国第一大河，也是亚洲第一大河，全长约6300公里。它发源于青藏高原的唐古拉山脉，流经中国11个省份，最终注入东海。长江不仅是中国的母亲河，也是亚洲最重要的河流之一，对中国的气候、水资源和生态环境有着深远的影响。

[Q]世界上最高的山峰是什么？
[A]珠穆朗玛峰是世界上海拔最高的山峰，位于喜马拉雅山脉，位于尼泊尔和中国边境。它的海拔高度为8,848.86米（29,031.7英尺）。

[Q]万有引力是谁提出的？
[A]万有引力是由艾萨克·牛顿在1687年提出的。他在这本书中详细阐述了万有引力定律，并通过实验验证了这一理论的正确性。

[Q]你知道大熊猫的主要食物是什么吗？
[A]当然，大熊猫主要以竹子为食。它们每天需要消耗大量的竹子来满足能量需求。除了竹子，大熊猫也会吃一些其他植物，如野生植物、果实和坚果。

[Q]海水为什么是咸的？
[A]海水之所以是咸的，主要是因为地球上的水循环过程。海洋是地球上最大的生态系统，它通过蒸发、降水、地表径流和地下水的补给等过程，将大量的溶解在水中的盐分转化为无量的盐分，从而维持着海洋生态系统的平衡。

蒸发：地球表面的水体（如海洋）在吸收和释放热量后，会以水蒸气的形式从海洋中蒸发出来。这个过程称为蒸发。
降水：随着水体的不断循环，水蒸气在大气中凝结成云，最终以降水的形式返回海洋。降水是海洋生物生长和繁殖的基础。
地表径流：在某些情况下，海水中的溶解物质（如溶解的矿物质、有机物等）会通过河流、湖泊、海洋表面等地理环境传递到海洋中，形成海水的“咸水”。
地下水补给：在一些深海区域，由于水压的增加，海水中的溶解盐分（如氯化钠、硫酸盐等）会被提取出来，然后通过地下水系统输送到淡水湖泊、河流、海洋中的淡水中，再通过河流或湖泊进行补给。
综上所述，海水之所以是咸的，是由于地球上的水循环过程，以及地球表面的盐分和溶解物质的积累。这些过程共同作用，使得海洋成为地球上最大的生态系统之一。

[Q]写一段python的快速排序代码
[A]快速排序是一种高效的排序算法，采用分治法策略。以下是使用Python实现的快速排序代码：

python

def quick_sort(arr):
    if len(arr) <= 1:
        return arr
    pivot = arr[len(arr) // 2]
    left = [x for x in arr if x < pivot]
    middle = [x for x in arr if x == pivot]
    right = [x for x in arr if x > pivot]
    return quick_sort(left) + middle + quick_sort(right)
# 示例
arr = [3, 6, 8, 10, 1, 2, 1]
print("原始数组:", arr)
sorted_arr = quick_sort(arr)
print("排序后的数组:", sorted_arr)

这段代码首先检查输入数组的长度，如果长度小于等于1，则直接返回数组（因为长度为1或空的数组已经是排序好的）。然后选择一个基准值（这里选择数组中间的元素），将数组分为三部分：小于基准值的元素、等于基准值的元素和大于基准值的元素。接着对小于和大于基准值的两部分递归地进行快速排序，并将结果与等于基准值的部分合并。
在示例中，我们创建了一个数组arr，并调用quick_sort函数对其进行排序。最后，打印排序前后的结果。

[Q]截至2025年2月9日14时31分，电影《哪吒之魔童闹海》累计票房（含预售）突破78.09亿元，超过《星球大战：天行者崛起》的票房成绩，成为首部进入全球票房榜前40的亚洲电影。这一成就不仅标志着中国电影在国际市场的突破，也展示了中国动画电影的巨大潜力和市场吸引力。从这段文字总结20字的摘要给我。
[A]《哪吒之魔童闹海》在2025年2月9日14时31分累计票房突破78.09亿元，成为首部进入全球票房榜前40的亚洲电影。

[Q]Justice has 4 ferns, 6 palms, and 8 succulent plants in her house. If she wants a total of 25 plants in her home, how many more plants does she need?
[A]Justice has 4 ferns, 6 palms, and 8 succulent plants. So, the total number of plants she has is 4 + 6 + 8 = 18. Since she wants a total of 25 plants, she needs 25 - 18 = 7 more plants. The answer is 7.

[Q]请写一篇关于环保的社论，要求言之有物、具有说服力。
[A]好的，以下是我为您撰写的社论：
随着工业化和城市化的快速发展，环境问题越来越受到人们的关注。我们必须认识到，保护环境是每个人的责任。为了实现可持续发展，我们需要采取行动。
首先，我们应该减少污染。这包括减少废弃物的产生和处理，以及使用更环保的产品和材料。同时，我们也应该鼓励人们使用公共交通工具、骑自行车或步行等方式出行，以减少汽车尾气的排放。
其次，我们应该重视水资源的保护。水是生命之源，但它是地球上最宝贵的资源之一。我们应该避免过度开采地下水，并确保我们的水源清洁。此外，我们还应该注意节约用水，例如减少洗澡时间、关闭水龙头等。
最后，我们应该重视垃圾分类和回收。垃圾分类可以有效地减少垃圾的数量，并将可回收的物品进行回收利用。同时，回收废纸、塑料和金属等材料可以减少对自然资源的消耗。
综上所述，保护环境是我们每个人的责任。我们应该采取行动，减少污染，保护水资源，并重视垃圾分类和回收。只有这样，我们才能共同创造一个更加美好的未来。