You can deploy this model and start earning money today!

RapidMark/
Qwen2.5-1.5B-E4M3-GGUF

Name: Qwen2.5-1.5B-E4M3-GGUF
Availability: InStock

Qwen/Qwen2.5-1.5B-Instruct

Views

No views yet

Qwen2.5-1.5B-Instruct E4M3 (fp8) GGUF

Qwen2.5-1.5B-Instruct quantized to E4M3 (OCP e4m3fn fp8) for llama.cpp. Test model for the CPU e4m3 quant type.

Block layout matches Q8_0: one fp16 scale + 32 one-byte elements per 32-value block. Q8_0 spends the byte on a linear int8; e4m3 spends it on a float (1 sign / 4 exponent / 3 mantissa).

How it was made

Built from the f16 GGUF with llama.cpp (branch cloudhands/fp8-cpu, HEAD 8a02ad204):

llama-quantize qwen2.5-1.5b-f16.gguf qwen2.5-1.5b-e4m3.gguf E4M3

Evaluation

Qwen2.5-1.5B-Instruct, wikitext-2-raw test (full, 584 chunks, n_ctx 512), pure CPU.

type	size	PPL	vs f16	Mean KLD	Median KLD	Max KLD	same-top-token
f16	2.88 GiB	10.134	-	-	-	-	-
q8_0	1.53 GiB	10.168	+0.33%	0.00200	0.00158	0.213	97.33%
e4m3	1.48 GiB	10.248	+1.12%	0.01146	0.00937	0.846	93.64%

llama-bench, pure CPU, 64 threads:

type	pp512 (t/s)	tg128 (t/s)
f16	528.3	19.2
q8_0	499.3	33.1
e4m3	253.2	34.6

Notes

Reference weights for testing, not tuned for downstream use.

RapidMark/Qwen2.5-1.5B-E4M3-GGUF

Qwen2.5-1.5B-Instruct E4M3 (fp8) GGUF

How it was made

Evaluation

Notes

RapidMark/
Qwen2.5-1.5B-E4M3-GGUF