Views
No views yet
ha-action-v3 JSON contract.unsloth/Qwen3-4B-Instruct-2507tuxevil/Home-Assistant-Requests-V4acon96/Home-Assistant-Requests, acon96/Home-Assistant-Requests-V2, and project-curated contract cases.| Model state | Precision | JSON valid | Exact match | Status exact | Service exact | Safety OK | Mean latency |
|---|---|---|---|---|---|---|---|
| Base model | 4-bit | 99.76% | 0.00% | 75.30% | 99.28% | 75.54% | 5.678 s |
| Trained adapter | 4-bit QLoRA | 100.00% | 98.08% | 98.56% | 100.00% | 98.80% | 3.192 s |
| Merged model | FP16 | 95.44% | 88.25% | 89.45% | 95.44% | 89.69% | 3.242 s |
| Quantization | Qwen3-4B exact | Qwen3-8B exact | Delta | Qwen3-4B safety | Qwen3-8B safety | Delta |
|---|---|---|---|---|---|---|
| Q4_K_M | 87.53% | 91.85% | −4.32 pp | 88.49% | 93.05% | −4.56 pp |
| Q5_K_M | 95.68% | 97.36% | −1.68 pp | 96.64% | 98.32% | −1.68 pp |
| Q6_K | 92.81% | 97.36% | −4.55 pp | 94.48% | 98.32% | −3.84 pp |
| Quantization | Exact match | Safety OK | Mean latency | Deployment note |
|---|---|---|---|---|
| Q4_K_M | 87.53% | 88.49% | 1.096 s | Lowest quality |
| Q5_K_M | 95.68% | 96.64% | 1.162 s | Best observed |
| Q6_K | 92.81% | 94.48% | 1.234 s | No quality gain over Q5 |
| Q8_0 | 92.57% | 93.76% | 1.171 s | Below Q5 in this run |