RWKV Architecture "Finch" based 11.2B Parameters Model.
継続トレーニング中です。実験なので性能評価はしていません。
"YORINOBU"
Based on RWKV6-World v2.1 7b 53% Model, we have applied a layer expansion approach and tuned it as a 48-layer, 4096-dimensional model.
I added 8 layers to the 40-layer model, froze layers 0 to 39, and continued pre-training layers 40 to 47, along with the Embedding and Head layers, using a Japanese corpus.
Since it is an experimental approach, it may exhibit unpredictable behavior.
