OpenMOSE/RWKV-GLM-4.7-Flash-Preview-v0.1
Model Summary
OpenMOSE/RWKV-GLM-4.7-Flash-Preview-v0.1 is a 30B-parameter (Active 3B) experimental hybrid model built from Z.ai's GLM-4.7-Flash, converted into a new RWKV hxa07F + NoPE MLA Hybrid architecture.
The core research question driving this model:
Can DeepSeek-style MLA (Multi-head Latent Attention) architecture be linearly converted to RWKV?
By replacing the vast majority of MLA layers with RWKV, the model retains only 4 out of 47 layers as MLA — achieving a 91% KV-cache reduction and enabling near-linear inference behavior.
⚠️ This is a Preview / Experimental Release. The model currently suffers from training-time spike instability and should be treated as a research artifact, not a production-ready model.
Key Facts
| Item | Value |
|---|
| Repository / Name | OpenMOSE/RWKV-GLM-4.7-Flash-Preview-v0.1 |
| Base (converted from) | Z.ai GLM-4.7-Flash |
| Total parameters | 30B |
| Active parameters | 3B (MoE) |
| Total layers | 47 |
| RWKV layers | 43 |
| NoPE MLA layers | 4 |
| KV-cache ratio | 4/47 (≈8.5%) — nearly full linear operation |
Why This Model Exists
This model represents an important experimental milestone in the RWKV hybrid research line:
- Previous work (hxa07D series) demonstrated that Transformer attention layers can be aggressively replaced with RWKV while preserving instruction-following quality via distillation.
- This model extends that question to MLA, the latent-compressed attention architecture introduced by DeepSeek. MLA differs structurally from standard GQA/MHA — it uses low-rank key-value compression, making the conversion dynamics fundamentally different.
- The goal is to determine whether RWKV can serve as a drop-in linear replacement for MLA layers, just as it has for standard attention layers in prior work.
If successful, this opens a path to converting a broader class of modern architectures (DeepSeek-V2/V3, GLM-4, and similar MLA-based models) into near-linear hybrids with drastically reduced inference costs.
Architecture
Overview: RWKV hxa07F + NoPE MLA Hybrid
The network is a 47-layer stack:
- 43 layers: RWKV hxa07F (linear-time recurrence)
- 4 layers: NoPE MLA (Multi-head Latent Attention without positional embeddings)
With only 4/47 layers retaining MLA-style KV cache, the model operates almost entirely in linear time.
hxa07F RWKV Highlights
hxa07F builds on the RWKV v7 lineage with further refinements:
- Improved RWKV v7 "Goose" Dynamic State Evolution core
- k,v residual connections for stable information flow during distillation
- Big Decay (high-precision forgetting) — retention-focused decay design for long-range stability without relying on KV cache
- Architectural adaptations for compatibility with MLA-based teacher models
NoPE MLA (4 layers)
- DeepSeek-style Multi-head Latent Attention layers with low-rank KV compression
- Retained at strategic positions to preserve selective global interaction capability
- No positional embeddings (NoPE) — position information is handled implicitly
Distillation Method: RADLADS (SmerkyG)
The teacher → student conversion pipeline is based on RADLADS, proposed by SmerkyG.
The key challenge specific to this model:
- The teacher uses MLA (latent-compressed attention), which has different internal representations than standard attention
- The distillation must bridge the gap between MLA's compressed KV space and RWKV's recurrent state space
- This is a qualitatively different conversion problem compared to prior GQA→RWKV work
Known Issues & Experimental Status
⚠️ This is an experimental preview. Known issues include:
- Training Spike Problem: The model exhibits loss spikes during training, indicating instability in the MLA→RWKV conversion or distillation process. This is an active area of investigation.
- Incomplete convergence: As a preview release, the model has not completed full distillation staging.
- Quality may vary significantly across tasks compared to the GLM-4.7-Flash teacher.
This release is published to share progress and enable community investigation into MLA-to-RWKV conversion dynamics.
Performance
Performance benchmarks are pending due to the experimental nature of this preview. Expect degraded quality relative to the teacher model until training stability issues are resolved.
| Category | Benchmark | Score | Notes |
|---|
| Reasoning | MMLU | TBD | Awaiting stable checkpoint |
| Math | GSM8K | TBD | Awaiting stable checkpoint |
| Long Context | PassKey / Needle | TBD | KV cache advantage expected here |
Intended Use
Best for
- Research and experimentation on MLA→RWKV linear conversion feasibility
- Investigating hybrid architecture design with MLA-based source models
- Studying training stability in heterogeneous architecture distillation
Not intended for
- Production or deployment use (experimental preview)
- Safety-critical domains (medical/legal/financial advice)
- Tasks requiring teacher-level quality (distillation is incomplete)
Limitations & Known Considerations
- Spike instability: Training spikes remain unresolved; outputs may be inconsistent or degraded.
- MLA conversion is uncharted territory: Unlike prior GQA→RWKV conversions, MLA's latent compression introduces new challenges for linear replacement.
- Hybrid trade-off: With only 4 MLA layers, some behaviors dependent on deep attention may differ significantly.
- Distillation dependence: Final quality is strongly tied to resolving training stability and the distillation recipe.
Bias, Safety, and Responsible Use
This model inherits typical biases and failure modes from large-scale web-trained models. As an experimental preview, additional caution is warranted:
- Do not rely on this model for any consequential decisions
- Outputs may be unreliable or incoherent due to training instability
- Evaluate thoroughly on your target domains before any downstream use
How to Use
Quick tips
- Treat this as a research checkpoint — expect rough edges.
- If your runtime supports it, prefer settings optimized for RWKV-heavy decoding (kernel-optimized recurrent path).
- KV-cache VRAM usage will be minimal (only 4 MLA layers contribute to cache).
Example
-
- Please check requirements.txt
1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
3
4MODEL_PATH = "OpenMOSE/RWKV-GLM-4.7-Flash-Preview-v0.1"
5
6messages = [{"role": "user", "content": "tell me what is rickroll."}]
7tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH
8 ,torch_dtype="auto",
9 device_map="auto",
10 trust_remote_code=True,
11)
12inputs = tokenizer.apply_chat_template(
13 messages,
14 tokenize=True,
15 add_generation_prompt=True,
16 return_dict=True,
17 enable_thinking=False,
18
19 return_tensors="pt",
20)
21
22print(type(tokenizer))
23print("chat_template is None?", tokenizer.chat_template is None)
24print("chat_template head:\n", (tokenizer.chat_template or "")[:400])
25
26print(inputs)
27
28
29print('---------------------------')
30print(tokenizer.decode(inputs['input_ids']))
31#exit()
32model = AutoModelForCausalLM.from_pretrained(
33 pretrained_model_name_or_path=MODEL_PATH,
34 torch_dtype=torch.bfloat16,
35 device_map="auto",
36 trust_remote_code=True,
37)
38inputs = inputs.to(model.device)
39generated_ids = model.generate(**inputs, max_new_tokens=512,use_cache=True, do_sample=True)
40output_text = tokenizer.decode(generated_ids[0][inputs.input_ids.shape[1]:])
41
42print('--------------------------------------------------------------------------------------')
43print(output_text)
(Adjust to your actual runtime / custom loader depending on how the hybrid is implemented.)
Research Context
This model is part of an ongoing research program exploring the boundaries of linear-attention hybrid architectures:
| Model | Base | Architecture | Attn Layers | RWKV Layers | Total | KV Reduction |
|---|
| hxa07D-L8 | Qwen3-30B-A3B | RWKV hxa07D + NoPEAttn | 8 | 40 | 48 | 83% |
| hxa07F-GLM (this) | GLM-4.7-Flash | RWKV hxa07F + NoPE MLA | 4 | 43 | 47 | 91% |
The progression shows increasingly aggressive attention layer reduction, and the introduction of MLA as the attention variant marks a new research direction.
Acknowledgements
This architecture research and implementation was made possible with computing power and technical support from Recursal AI.
We sincerely thank them for enabling this work.
Distillation methodology is based on RADLADS, proposed by SmerkyG, whose ideas significantly influenced the training pipeline.
Citation
If you use this model in research or products, please cite the model repository and credit the contributors and supporting organizations.
2025 OpenMOSE
