Views
No views yet
1slices:
2 - sources:
3 - model: bamec66557/MISCHIEVOUS-12B-Mix_0.4v
4 layer_range: [0, 8]
5 - model: bamec66557/MISCHIEVOUS-12B-Mix_III_IV_V
6 layer_range: [0, 8]
7 parameters:
8 t:
9 - value: 0.72
10 - sources:
11 - model: bamec66557/MISCHIEVOUS-12B-Mix_0.4v
12 layer_range: [8, 16]
13 - model: bamec66557/MISCHIEVOUS-12B-Mix_III_IV_V
14 layer_range: [8, 16]
15 parameters:
16 t:
17 - value: [0.75, 0.85, 0.75]
18 - sources:
19 - model: bamec66557/MISCHIEVOUS-12B-Mix_0.4v
20 layer_range: [16, 24]
21 - model: bamec66557/MISCHIEVOUS-12B-Mix_III_IV_V
22 layer_range: [16, 24]
23 parameters:
24 t:
25 - value: [0.85, 1.0, 0.85]
26 - filter: feed_forward
27 value: [0.9, 1.0, 1.1]
28 - sources:
29 - model: bamec66557/MISCHIEVOUS-12B-Mix_0.4v
30 layer_range: [24, 32]
31 - model: bamec66557/MISCHIEVOUS-12B-Mix_III_IV_V
32 layer_range: [24, 32]
33 parameters:
34 t:
35 - value: [0.95, 1.0, 0.95]
36 - sources:
37 - model: bamec66557/MISCHIEVOUS-12B-Mix_0.4v
38 layer_range: [32, 40]
39 - model: bamec66557/MISCHIEVOUS-12B-Mix_III_IV_V
40 layer_range: [32, 40]
41 parameters:
42 t:
43 - value: 1.0
44 - filter: self_attn
45 value: [0.92, 1.0, 1.08]
46merge_method: slerp
47base_model: bamec66557/MISCHIEVOUS-12B-Mix_0.4v
48regularization:
49 - method: weight_clipping
50 clip_range: [-0.04, 0.04]
51postprocessing:
52 - operation: gaussian_smoothing
53 sigma: 0.9
54 - operation: normalize
55 - operation: quantize
56 target_dtype: int8
57dtype: bfloat16
58| Metric | Value |
|---|---|
| Avg. | 23.84 |
| IFEval (0-Shot) | 43.66 |
| BBH (3-Shot) | 34.73 |
| MATH Lvl 5 (4-Shot) | 12.31 |
| GPQA (0-shot) | 10.40 |
| MuSR (0-shot) | 12.34 |
| MMLU-PRO (5-shot) | 29.58 |