Views
No views yet
1slices:
2 - sources:
3 - model: bamec66557/mergekit-model_stock-zdaysvi
4 layer_range: [0, 20]
5 - model: bamec66557/MISCHIEVOUS-12B-Mix_0.4v
6 layer_range: [0, 20]
7 parameters:
8 t:
9 - value: 0.8
10
11 - sources:
12 - model: bamec66557/mergekit-model_stock-zdaysvi
13 layer_range: [20, 40]
14 - model: bamec66557/MISCHIEVOUS-12B-Mix_0.4v
15 layer_range: [20, 40]
16 parameters:
17 t:
18 - value: 1.0
19 - filter: self_attn
20 value: [0.8, 0.9, 1.0, 1.1, 1.2]
21
22merge_method: slerp # Preserve merge method
23
24base_model: bamec66557/mergekit-model_stock-zdaysvi # Base model
25
26dtype: bfloat16 # Data types for fast merges
27
28# Additional options
29regularization:
30 - method: weight_clipping
31 clip_range: [-0.1, 0.1]
32
33postprocessing:
34 - operation: gaussian_smoothing
35 sigma: 1.5 # Gaussian smoothing intensity
36 - operation: smoothing
37 parameters:
38 adaptive: true
39 range: [0.8, 1.2] # Adaptively adjust
40 kernel_size: 5 # Smoothing larger ranges with increased kernel size
41 - operation: normalize # Normalise after merge
42| Metric | Value |
|---|---|
| Avg. | 26.05 |
| IFEval (0-Shot) | 63.65 |
| BBH (3-Shot) | 30.51 |
| MATH Lvl 5 (4-Shot) | 11.25 |
| GPQA (0-shot) | 8.50 |
| MuSR (0-shot) | 12.44 |
| MMLU-PRO (5-shot) | 29.96 |