I have adopted a new recipe for merging this 120B model (I tried to expand the recipe to 124B, but experienced a performance decline). Compared to the original 124B version, it has 4B fewer parameters but seems to have improved performance (at least that is my subjective impression). It exhibits fewer hallucinations, better comprehension, and clearer logic than the old version of the 124B model (although I am not sure by how much, as my judgement is based on limited subjectively use). It still cannot (in most time) solve some of my high-difficulty reasoning questions I use for testing, but it seems less likely to get confused and makes more slightly mistakes in the same questions.
Coming soon...
1dtype: float16
2merge_method: passthrough
3slices:
4- sources:
5 - layer_range: [0, 20]
6 model: Qwen\Qwen1.5-72B-Chat
7- sources:
8 - layer_range: [5, 30]
9 model: Qwen\Qwen1.5-72B-Chat
10- sources:
11 - layer_range: [10, 35]
12 model: Qwen\Qwen1.5-72B-Chat
13- sources:
14 - layer_range: [30, 50]
15 model: Qwen\Qwen1.5-72B-Chat
16- sources:
17 - layer_range: [40, 60]
18 model: Qwen\Qwen1.5-72B-Chat
19- sources:
20 - layer_range: [55, 80]
21 model: Qwen\Qwen1.5-72B-Chat
I feel its understanding and logical reasoning abilities are better than the 124B version(subjectively), but I'm not clear about other aspects of its performance (for example, writing ability, as most normal 120B+ models have decent writing, making it difficult to discern superiority).If you believe in this model's performance, feel free to test it out or offer evaluations. Everyone's tests or evaluations are welcome.
