You can deploy this model and start earning money today!

wangyue114514/
rwkv7-g1d-0.4b-hf

Name: rwkv7-g1d-0.4b-hf
Availability: InStock

Views

No views yet

RWKV-7 G1D 0.4B — Hugging Face reference model

This repository is a self-contained Hugging Face conversion of rwkv7-g1d-0.4b-20260210-ctx8192.pth. Release v0.9.0 contains the complete readable, pure-PyTorch reference implementation next to the weights. Normal inference does not require rwkv7-hf, FLA, a custom CUDA wheel, JIT, or CUDA Graphs.

Install and use

python -m pip install torch transformers

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "wangyue114514/rwkv7-g1d-0.4b-hf"
revision = "v0.9.0"
tokenizer = AutoTokenizer.from_pretrained(
    model_id, revision=revision, trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
    model_id, revision=revision, trust_remote_code=True, torch_dtype="auto"
).eval()
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
inputs = tokenizer("The future of recurrent language models is", return_tensors="pt")
inputs = {key: value.to(device) for key, value in inputs.items()}
with torch.inference_mode():
    tokens = model.generate(**inputs, max_new_tokens=32)
print(tokenizer.decode(tokens[0], skip_special_tokens=True))

trust_remote_code=True loads the checked-in files configuration_rwkv7.py, cache_rwkv7.py, ops_rwkv7.py, modeling_rwkv7.py, and tokenization_rwkv7.py. The recurrent state has the canonical [batch, heads, key, value] layout. Cache lifecycle, loss, padding, generation, gradients, and the layer structure are visible in those files.

Fine-tuning and evaluation

The model follows the standard AutoModelForCausalLM contract and supports Trainer, Accelerate, PEFT LoRA, TRL SFT/DPO/GRPO, and lm_eval through the ordinary Transformers path. Set model.config.use_cache = False during training. Reproducible examples and evaluation manifests are in 123123213weqw/hf-adapter.

Reference versus optimized execution

The v0.9.0 model code intentionally favors readability and compatibility. It is the correctness/reference path, not a peak-throughput benchmark backend. Optional CUDA Graph and Triton work is maintained separately on perf/optional-native-backend-v0.10; older CUDA/JIT/quantization and KV-v2 experiments remain on perf/native-kernels-v0.8. Neither branch changes this model repository's public HF contract.

Model and provenance

Architecture: RWKV-7 recurrent causal language model
Checkpoint family: G1D
Parameters: 450,767,872
Layers: 24
Hidden size: 1024
Vocabulary size: 65536
Stored weight dtype: float16
Original checkpoint SHA256: 947cb9b8013224e06b112b72204256bec65096cc935a7767ce63d8e3ddef83bb
Reference source revision: b8438cab0dc7d11238942efca4c05135d53fcaf8
Release tag: v0.9.0

conversion_manifest.json retains the immutable original conversion and weight provenance. v0.9.0 changes the checked-in runtime code and config only; the safetensors bytes are not rewritten or re-uploaded.

License

Apache-2.0. See LICENSE.