Views
No views yet
1name: SuperMerge-LayeredTIES-v1
2merge_method: della_linear
3base_model: CultriX/Enhanced-TIES-Base-v1 # Referencing the TIES base model defined below (now inlined)
4tokenizer_source: base
5dtype: float32
6out_dtype: bfloat16
7parameters:
8 int8_mask: true
9 normalize: true
10 rescale: false
11 t: [0.1, 0.3, 0.7, 0.7, 0.4, 0.2]
12
13slices:
14 - sources:
15 - model: CultriX/Enhanced-TIES-Base-v1 # Referencing inlined TIES base
16 layer_range: [0, 8]
17 parameters:
18 weight: 0.7
19 - model: arcee-ai/Virtuoso-Small-v2
20 layer_range: [0, 8]
21 parameters:
22 weight: 0.3
23 - model: sthenno/tempesthenno-ppo-ckpt40
24 layer_range: [0, 8]
25 parameters:
26 weight: 0.0
27 - model: sometimesanotion/Qwenvergence-14B-v3-Prose
28 layer_range: [0, 8]
29 parameters:
30 weight: 0.0
31 - sources:
32 - model: CultriX/Enhanced-TIES-Base-v1 # Referencing inlined TIES base
33 layer_range: [8, 16]
34 parameters:
35 weight: 0.4
36 - model: arcee-ai/Virtuoso-Small-v2
37 layer_range: [8, 16]
38 parameters:
39 weight: 0.3
40 - model: sthenno/tempesthenno-ppo-ckpt40
41 layer_range: [8, 16]
42 parameters:
43 weight: 0.3
44 - model: sometimesanotion/Qwenvergence-14B-v3-Prose
45 layer_range: [8, 16]
46 parameters:
47 weight: 0.0
48 - sources:
49 - model: CultriX/Enhanced-TIES-Base-v1 # Referencing inlined TIES base
50 layer_range: [16, 24]
51 parameters:
52 weight: 0.2
53 - model: arcee-ai/Virtuoso-Small-v2
54 layer_range: [16, 24]
55 parameters:
56 weight: 0.2
57 - model: sthenno/tempesthenno-ppo-ckpt40
58 layer_range: [16, 24]
59 parameters:
60 weight: 0.5
61 - model: sometimesanotion/Qwenvergence-14B-v3-Prose
62 layer_range: [16, 24]
63 parameters:
64 weight: 0.1
65 - sources:
66 - model: CultriX/Enhanced-TIES-Base-v1 # Referencing inlined TIES base
67 layer_range: [24, 32]
68 parameters:
69 weight: 0.25
70 - model: arcee-ai/Virtuoso-Small-v2
71 layer_range: [24, 32]
72 parameters:
73 weight: 0.1
74 - model: sthenno/tempesthenno-ppo-ckpt40
75 layer_range: [24, 32]
76 parameters:
77 weight: 0.4
78 - model: sometimesanotion/Qwenvergence-14B-v3-Prose
79 layer_range: [24, 32]
80 parameters:
81 weight: 0.25
82 - sources:
83 - model: CultriX/Enhanced-TIES-Base-v1 # Referencing inlined TIES base
84 layer_range: [32, 40]
85 parameters:
86 weight: 0.4
87 - model: arcee-ai/Virtuoso-Small-v2
88 layer_range: [32, 40]
89 parameters:
90 weight: 0.0
91 - model: sthenno/tempesthenno-ppo-ckpt40
92 layer_range: [32, 40]
93 parameters:
94 weight: 0.2
95 - model: sometimesanotion/Qwenvergence-14B-v3-Prose
96 layer_range: [32, 40]
97 parameters:
98 weight: 0.4
99 - sources:
100 - model: CultriX/Enhanced-TIES-Base-v1 # Referencing inlined TIES base
101 layer_range: [40, 48]
102 parameters:
103 weight: 0.6
104 - model: arcee-ai/Virtuoso-Small-v2
105 layer_range: [40, 48]
106 parameters:
107 weight: 0.0
108 - model: sthenno/tempesthenno-ppo-ckpt40
109 layer_range: [40, 48]
110 parameters:
111 weight: 0.1
112 - model: sometimesanotion/Qwenvergence-14B-v3-Prose
113 layer_range: [40, 48]
114 parameters:
115 weight: 0.3
116
117
118
119# Commentary:
120# =============================================================================
121# SuperMerge-LayeredTIES-v1 Commentary:
122#
123# This configuration combines the strengths of both Enhanced-LayeredSlerp-v1 and SuperMerge-Enhanced-v1.
124# It leverages the robust foundation of a TIES-merged base model (Enhanced-TIES-Base-v1) and applies
125# the layer-wise module approach and fine-grained weight control from SuperMerge-Enhanced-v1 in a SLERP merge.
126#
127# Key Features:
128# - TIES-Merged Base Foundation: Uses 'Enhanced-TIES-Base-v1' as the base model for the SLERP merge.
129# This TIES base provides a selectively merged and potentially more efficient starting point, incorporating
130# strengths from multiple models (Virtuoso, Phi-4, Qwenvergence, DeepSeek) with density control.
131#
132# - Layer-wise Module Integration in SLERP: Maintains the module-based slice structure from SuperMerge-Enhanced-v1.
133# The SLERP merge now combines the TIES-merged base with specialized modules for Reasoning, IFEval, and MATH/Knowledge
134# at different layer ranges, using explicit weights for fine-grained control.
135#
136# - Benchmark-Driven Iterative Weight Tuning: The configuration is designed to be optimized through a
137# benchmark-driven iterative weight tuning process (as described in the refined SuperMerge-Enhanced-v1 approach).
138# The initial weights provided are starting points and need to be systematically tuned based on benchmark results.
139#
140# Tuning Process (Same as Refined SuperMerge-Enhanced-v1):
141# 1. Initial Benchmarking: Run a full benchmark suite.
142# 2. Performance Analysis: Examine per-benchmark scores and compare to source models.
143# 3. Targeted Weight Adjustments: Adjust layer weights based on performance analysis (e.g., increase IFEval module weight
144# in early layers if IFEval is weak).
145# 4. Iterate: Repeat steps 1-3. Make small, incremental adjustments in each iteration.
146#
147# Rationale:
148# - By using a TIES-merged base, we aim to create a more robust and potentially efficient foundation for the SLERP merge.
149# - The layer-wise module approach and fine-grained weights in SLERP still allow for precise control over the blending
150# of specialized capabilities at different network depths, building upon the solid TIES base.
151# - The emphasis on a benchmark-driven iterative weight tuning process remains crucial for achieving optimal performance.
152#
153# Next Steps:
154# - Implement this configuration using MergeKit.
155# - Run initial benchmarks to establish a baseline.
156# - Begin the iterative benchmark-driven weight tuning process to optimize performance.
157# =============================================================================