RWKV-x070-2B9-CJE-Instruct Model Card

Model Overview

• Model Name: RWKV-x070-2B9-CJE-Instruct
• Description: An instruction-tuned model specialized for Japanese, Chinese, and English languages
• Base Model: rwkv-x070-2b9-world-v3-40%trained-20250113-ctx4k.pth
• Architecture: RWKV x070 "Goose"
• Parameters: 2.9B
• Model Dimension: 2560
• Number of Layers: 32

Fine-tuning Details

Training Configuration

• Trainer: RWKV-LM-RLHF (https://github.com/OpenMOSE/RWKV-LM-RLHF)
• PEFT Mode: Hybrid Training combining frozen embeddings and Bone (Block Affine Transformation) + full parameter training
• SFT Method: SmoothingLoss SFT
• Context Window: 5120 (trained with 1024 token overlap)
• Compute Power: AMD Instinct MI100 x 2 60hrs (100％ solar energy)

Dataset Specifications

• Size: 800k pairs
• Content:
  • Mixed data in Japanese, Chinese, and English
  • Conversations
  • Programming code
  • Translation tasks
  • Chain-of-Thought reasoning tasks

How to use

• Install latest RWKV-Infer (Linux,WSL) (https://github.com/OpenMOSE/RWKV-Infer)
• make folder 'models'
• move rwkv-x070-2b9-cje-instruct-1.pth to models folder

• Enjoy with openai compatible api http://127.0.0.1:9000/v1 :)

Important Note

• Set the end token as '\n\n\x17'

User: who are you?\n\n\x17
Assistant: gooday i'm rwkv\n\n\x17

Limitations and Considerations

• This is an experimental model; inference stability is not fully guaranteed
• Unexpected behaviors may occur
• Continuous improvements are being made; feedback is welcome

License

Acknowledgments