Two importance-matrix quantizations of RWKV7-G1g 13.3B,
targeting 8 GB VRAM GPUs:
IQ4_XS (8.00 GB) — The standard Q4_K_M lands at 8.46 GB, just over the limit of common 8 GB GPUs (RTX 3070, 3080 8GB, 4060, 4060 Ti 8GB, etc.). IQ4_XS brings it under that limit, but the fit is extremely tight — down to the megabyte — and may not work reliably on all 8 GB GPUs depending on driver and OS overhead.
IQ3_M (6.72 GB) — A more practical choice for 8 GB GPUs, leaving ~1.3 GB of headroom for overhead. Slightly lower quality but much more comfortable to run.
RWKV is a pure RNN architecture with transformer-level performance. Unlike transformers,
it has constant VRAM usage and constant inference speed regardless of context length —
no KV cache that grows with sequence length.
The G-series data versions rank as follows (newer = better):
G1g > G1e > G1d > G1c > G1b > G1a2 > G1a > G1 > G0a2 > G0
This model supports a reasoning/thinking mode in addition to standard chat.
Quantization details
Both quantizations were produced from FP16 (not from Q4_K_M) to avoid compounding quantization error.
IQ4_XS
Property
Value
Method
IQ4_XS (importance-matrix guided)
Source
FP16 GGUF
Bits per weight
4.82 BPW
Calibration data
calibration_data_v5_rc.txt
Output size
8.00 GB
Tool
llama.cpp
IQ3_M
Property
Value
Method
IQ3_M (importance-matrix guided)
Source
FP16 GGUF
Bits per weight
4.05 BPW
Calibration data
calibration_data_v5_rc.txt
Output size
6.72 GB
Tool
llama.cpp
File listing
Filename
Size
Description
rwkv7-G1g-13.3b-IQ4_XS.gguf
8.00 GB
IQ4_XS quantized model
rwkv7-G1g-13.3b-IQ3_M.gguf
6.72 GB
IQ3_M quantized model
Comparison with other quantizations
Evaluated on a subset of the uncheatable eval dataset (lower BPB = better). Note: results are not directly comparable to the official uncheatable eval leaderboard due to the use of a subset.
Quantization
Size
BPW
ao3_en
arxiv_cs
arxiv_math
arxiv_phys
bbc_news
gh_cpp
gh_js
gh_md
gh_py
wiki_en
avg_bpb
Fits 8 GB GPU
FP16
26.7 GB
16.12
0.8395
0.5291
0.5366
0.5356
0.6269
0.3133
0.2818
0.4246
0.3391
0.6262
0.5053
❌
Q4_K_M
8.46 GB
4.80
0.8474
0.5339
0.5418
0.5420
0.6317
0.3306
0.2872
0.4360
0.3448
0.6348
0.5130
❌ (marginally over)
IQ4_XS (this)
8.00 GB
4.82
0.8444
0.5337
0.5407
0.5394
0.6324
0.3280
0.2864
0.4371
0.3439
0.6337
0.5120
✅
IQ3_M (this)
6.72 GB
4.05
0.8716
0.5477
0.5564
0.5561
0.6511
0.3537
0.3012
0.4608
0.3585
0.6534
0.5311
✅
Prompt formats
⚠️ Strip any trailing spaces from your input — trailing spaces upset the RWKV tokenizer.
Also replace `
Thinking mode — G1c and newer style (with length control)
User: YOUR_PROMPT (think)
A: <think
User: YOUR_PROMPT (think a bit)
A: <think
User: YOUR_PROMPT (think a lot)
A: <think
FIM (Fill-in-Middle) — G1c and newer
✿prefix✿LEFT_CONTEXT✿suffix✿RIGHT_CONTEXT✿middle✿
Recommended sampling parameters
For use with the RWKV pip package (note: applies temperature after top-p):
Chat:
temp=1, topp=0.5, alpha_presence=2, alpha_frequency=0.1, alpha_decay=0.99
Creative (fiction etc.):
temp=0.6, topp=0.6~0.8, alpha_presence=2, alpha_frequency=0.2, alpha_decay=0.99
Usage
llama.cpp
bash
1./llama-cli -m rwkv7-G1g-13.3b-IQ4_XS.gguf -ngl 99 --temp 1.0 --top-p 0.5 -p "User: What is RWKV?
23A:" -n 500
Ollama
ollama run hf.co/A190NUX/RWKV7-G1g-13.3B-GGUF:IQ4_XS
llama-cpp-python
python
1from llama_cpp import Llama
23llm = Llama(4 model_path="rwkv7-G1g-13.3b-IQ4_XS.gguf",5 n_gpu_layers=-1,6)7output = llm(8 "User: What is RWKV?
910A:",11 max_tokens=500,12 temperature=1.0,13 top_p=0.5,14)15print(output["choices"][0]["text"])
