A Goliath-120b style frankenmerge of aurelian-v0.5-70b-32K and WinterGoddess-1.4x-70b. The goal is to have similar performance with an extended context size. Important: Use a positional embeddings compression factor (compress_pos_emb) of 8 when loading this model.
Llama2 and Alpaca.
1- range 0, 16
2 aurelian
3- range 8, 24
4 WinterGoddess
5- range 17, 32
6 aurelian
7- range 25, 40
8 WinterGoddess
9- range 33, 48
10 aurelian
11- range 41, 56
12 WinterGoddess
13- range 49, 64
14 aurelian
15- range 57, 72
16 WinterGoddess
17- range 65, 80
18 aurelian
@grimulkan For creating aurelian-v0.5-70b-32K
@Sao10K For creating WinterGoddess
@alpindale For creating the original Goliath
