RWKV-4 430M

Use RWKV-4 models (NOT RWKV-4a, NOT RWKV-4b) unless you know what you are doing.

Use RWKV-4 models (NOT RWKV-4a, NOT RWKV-4b) unless you know what you are doing.

Use RWKV-4 models (NOT RWKV-4a, NOT RWKV-4b) unless you know what you are doing.

Model Description

• Pile loss 2.2621
• LAMBADA ppl 13.04, acc 45.16%
• PIQA acc 67.52%
• SC2016 acc 63.87%
• Hellaswag acc_norm 40.90%

• Pile loss 2.2394
• LAMBADA ppl 10.54, acc 50.20%
• PIQA acc 68.12%
• SC2016 acc 63.55%
• Hellaswag acc_norm 40.82%

• Pile loss 2.2026
• LAMBADA ppl 10.48, acc 51.35%
• PIQA acc 68.06%
• SC2016 acc 63.17%
• Hellaswag acc_norm 42.09%