This project was made possible through computational resources and technical support provided by Recursal.AI, to whom we extend our deepest gratitude. We are particularly grateful to SmerkyG for his invaluable technical assistance and guidance throughout this research.
RWKV-Seed-OSS-36B-hxa07A is a cost-efficient hybrid architecture model that achieves an optimal balance between model performance and inference efficiency while significantly reducing inference complexity.
Architecture: hxa07A
The hxa07A architecture is based on the RWKV-7 "Goose" Dynamic State Evolution architecture. This hybrid model consists of 64 layers, with 10 layers strategically configured as NoPE-GQA (No Positional Encoding - Grouped Query Attention) to achieve both strong long-context capabilities and RWKV's characteristic inference efficiency.
This model was converted by applying the hxa07A architecture to transform a Transformer model into an RWKV hybrid model, based on the RADLADS (Rapid Attention Distillation to Linear Attention Decoders at Scale) methodology proposed by SmerkyG.
Linear hybrid models represent a promising approach to dramatically reducing computational costs, especially for long-context scenarios. This architecture achieves:
10× or greater reduction in inference costs
Zero-latency reasoning capabilities
Improved AI accessibility
Practical Benefits
With limited GPU resources, this model enables:
Significantly more multi-batch inference compared to pure Transformer models
Substantial reduction in deployment costs
Efficient scaling for production environments
Usage with Hugging Face Transformers
need install flash-linear-attention
pip install flash-linear-attention
python
1from transformers import AutoModelForCausalLM, AutoTokenizer
23model_name ="OpenMOSE/RWKV-Seed-OSS-36B-hxa07A"45model = AutoModelForCausalLM.from_pretrained(6 model_name,7 torch_dtype="auto",8 device_map="auto",9 trust_remote_code=True,10)1112tokenizer = AutoTokenizer.from_pretrained(model_name)1314prompt ="""There is a very famous song that I recall by the singer's surname as Astley.
15 I can't remember the name or the youtube URL that people use to link as an example url.
16 What's song name?"""17messages =[18{"role":"system","content":"You are a helpful assistant."},19{"role":"user","content": prompt},20]21text = tokenizer.apply_chat_template(22 messages, tokenize=False, add_generation_prompt=True23)24model_inputs = tokenizer([text], return_tensors="pt").to(model.device)2526generated_ids = model.generate(**model_inputs, max_new_tokens=512)27generated_ids =[28 output_ids[len(input_ids):]29for input_ids, output_ids inzip(model_inputs.input_ids, generated_ids)30]3132response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]33
Code Repositories
RADLADS Project Code: The main codebase for the RADLADS paper, including conversion scripts and model code, can be found at: https://github.com/recursal/RADLADS
Specific Training Code (OpenMOSE): The training code for this particular model is available at: https://github.com/OpenMOSE/RWKVInside (Note: this repository is still under development and may contain bugs.)
Research Journey
Over the past several months of researching heterogeneous architecture distillation techniques, we encountered numerous challenges when training models exceeding 14B parameters. However, we have now achieved spike-free training even for models exceeding 30B parameters.
Personal note: I estimate I consumed over 100 liters of coffee during this research! (laughs)
We believe that once this conversion technology is fully established, it will enable us to reduce the operational costs of existing models by 1/10th or less while maintaining their capabilities.
Feedback Welcome
We eagerly welcome your feedback and observations on this model!
