File:Qwen3-8B.Q4_K_M.ggufQuantization:Q4_K_MTask: strict ha-action-v3 Home Assistant JSON planning.
Dataset provenance: V4 combines validated rows from acon96/Home-Assistant-Requests, acon96/Home-Assistant-Requests-V2, and project-curated contract cases. See the V4 dataset card for counts and filtering details.
Benchmark
Qwen3-8B direct Ollama benchmark on 417 held-out V4 cases.
Metric
Result
Exact match
91.85%
Safety OK
93.05%
Entity not invented
100.00%
Mean latency
1.536 s
The evaluation uses the fixed 417-case test split from tuxevil/Home-Assistant-Requests-V4.
Same-quantization model comparison
Quantization
Qwen3-8B exact
Qwen3-4B exact
Delta
Qwen3-8B safety
Qwen3-4B safety
Delta
Q4_K_M
91.85%
87.53%
+4.32 pp
93.05%
88.49%
+4.56 pp
Q5_K_M
97.36%
95.68%
+1.68 pp
98.32%
96.64%
+1.68 pp
Q6_K
97.36%
92.81%
+4.55 pp
98.32%
94.48%
+3.84 pp
Q8_0
Not published
92.57%
—
Not published
93.76%
—
Hugging Face / Transformers reference
Model
Base 4-bit exact
Adapter 4-bit exact
Merged FP16 exact
Base safety
Adapter safety
Merged FP16 safety
Qwen3-8B
0.00%
99.28%
99.20%*
84.17%
100.00%
99.50%*
Qwen3-4B
0.00%
98.08%
88.25%
75.54%
98.80%
89.69%
* Qwen3-8B FP16 published reference retains exact-match and safety metrics; other FP16 fields were not retained in the artifact manifest.
