Winner-RWKV-7-285M-20260601 – AI Model by Jellyfish042 | AlphaNeural AI

You can deploy this model and start earning money today!

Jellyfish042/
Winner-RWKV-7-285M-20260601

like

0

Views

No views yet

Winner-RWKV-7-285M-20260601

RWKV-7 checkpoint trained in Jellyfish042/GatedDeltaNet-2.

Architecture: RWKV-7 TimeMix, rwkv7_time_1024x12
Parameters printed by the training script: 219,611,136
Pretraining data: globally shuffled FineWeb-Edu packed 100B-token run
Sequence length: 4096
Global batch size: 1024
Seed: 3407
Checkpoint file: pytorch_model.bin

This is a project checkpoint from pretrain.py, not a Hugging Face Transformers checkpoint.

lm-eval-harness snapshot

Task	Metric	Value
`lambada_openai`	acc	`0.3728`
`lambada_openai`	perplexity	`25.6649`
`wikitext`	word_perplexity	`24.0226`
`wikitext`	byte_perplexity	`1.8121`
`wikitext`	bits_per_byte	`0.8577`