Two importance-matrix quantizations of RWKV7-G1f 13.3B,
targeting 8 GB VRAM GPUs:
IQ4_XS (8.00 GB) — The standard Q4_K_M lands at 8.46 GB, just over the limit of common 8 GB GPUs (RTX 3070, 3080 8GB, 4060, 4060 Ti 8GB, etc.). IQ4_XS brings it under that limit, but the fit is extremely tight — down to the megabyte — and may not work reliably on all 8 GB GPUs depending on driver and OS overhead.
IQ3_M (6.72 GB) — A more practical choice for 8 GB GPUs, leaving ~1.3 GB of headroom for overhead. Slightly lower quality but much more comfortable to run.
RWKV is a pure RNN architecture with transformer-level performance. Unlike transformers,
it has constant VRAM usage and constant inference speed regardless of context length —
no KV cache that grows with sequence length.
The G-series data versions rank as follows (newer = better):
G1f > G1e > G1d > G1c > G1b > G1a2 > G1a > G1 > G0a2 > G0
This model supports a reasoning/thinking mode in addition to standard chat.
Quantization details
Both quantizations were produced from FP16 (not from Q4_K_M) to avoid compounding quantization error.
IQ4_XS
Property
Value
Method
IQ4_XS (importance-matrix guided)
Source
FP16 GGUF
Bits per weight
4.82 BPW
Calibration data
calibration_data_v5_rc.txt
Output size
8.00 GB
Tool
llama.cpp
IQ3_M
Property
Value
Method
IQ3_M (importance-matrix guided)
Source
FP16 GGUF
Bits per weight
4.05 BPW
Calibration data
calibration_data_v5_rc.txt
Output size
6.72 GB
Tool
llama.cpp
File listing
Filename
Size
Description
rwkv7-g1f-13.3b-IQ4_XS.gguf
8.00 GB
IQ4_XS quantized model
rwkv7-g1f-13.3b-IQ3_M.gguf
6.72 GB
IQ3_M quantized model
Comparison with other quantizations
Evaluated on a subset of the uncheatable eval dataset (lower BPB = better). Note: results are not directly comparable to the official uncheatable eval leaderboard due to the use of a subset.
Quantization
Size
BPW
ao3_en
arxiv_cs
arxiv_math
arxiv_phys
bbc_news
gh_cpp
gh_js
gh_md
gh_py
wiki_en
avg_bpb
Fits 8 GB GPU
FP16
26.7 GB
16.12
0.8396
0.5305
0.5382
0.5374
0.6276
0.3116
0.2823
0.4271
0.3407
0.6263
0.5061
❌
Q4_K_M
8.46 GB
4.80
0.8453
0.5360
0.5432
0.5433
0.6341
0.3301
0.2867
0.4374
0.3442
0.6339
0.5134
❌ (marginally over)
IQ4_XS (this)
8.00 GB
4.82
0.8430
0.5340
0.5405
0.5405
0.6317
0.3241
0.2858
0.4375
0.3440
0.6326
0.5114
✅
IQ3_M (this)
6.72 GB
4.05
0.8857
0.5598
0.5705
0.5656
0.6689
0.3675
0.3070
0.4695
0.3657
0.6646
0.5425
✅
Prompt formats
⚠️ Strip any trailing spaces from your input — trailing spaces upset the RWKV tokenizer.
Also replace \n\n inside USER_PROMPT with \n since \n\n is used as a chat round separator.
Thinking mode — G1c and newer style (with length control)
User: YOUR_PROMPT (think)
A: <think
User: YOUR_PROMPT (think a bit)
A: <think
User: YOUR_PROMPT (think a lot)
A: <think
FIM (Fill-in-Middle) — G1c and newer
✿prefix✿LEFT_CONTEXT✿suffix✿RIGHT_CONTEXT✿middle✿
Recommended sampling parameters
For use with the RWKV pip package (note: applies temperature after top-p):
Chat:
temp=1, topp=0.5, alpha_presence=2, alpha_frequency=0.1, alpha_decay=0.99
Creative (fiction etc.):
temp=0.6, topp=0.6~0.8, alpha_presence=2, alpha_frequency=0.2, alpha_decay=0.99
Usage
llama.cpp
bash
1./llama-cli \2 -m rwkv7-g1f-13.3b-IQ4_XS.gguf \3 -ngl 99\4 --temp 1.0\5 --top-p 0.5\6 -p "User: What is RWKV?\n\nA:"\7 -n 500
Ollama
ollama run hf.co/A190NUX/RWKV7-G1f-13.3B-GGUF:IQ4_XS
llama-cpp-python
python
1from llama_cpp import Llama
23llm = Llama(4 model_path="rwkv7-g1f-13.3b-IQ4_XS.gguf",5 n_gpu_layers=-1,6)7output = llm(8"User: What is RWKV?\n\nA:",9 max_tokens=500,10 temperature=1.0,11 top_p=0.5,12)13print(output["choices"][0]["text"])
