A Granite 4.1 3B LoRA fine-tune for Home Assistant voice control —
tool calling, state queries, and spoken confirmations — quantized to GGUF
for llama.cpp / Ollama.
Please use this model instead. The older 7B model is deprecated and
no longer maintained. This one is smaller (3.6 GB vs 4.7 GB), ~30%
faster (240 ms vs 341 ms), quantized at higher precision (Q8_0 vs
Q4_K_M), and has an 8× larger context window — while fixing two
behaviours the old model got wrong.
Why replace the 7B?
The v1 7B model executed tool calls correctly but had two production
problems:
It frequently returned no speech after a tool call. Home Assistant
then raises Last content in chat log is not an AssistantContent and
the user hears "Unable to get response" even though the action
succeeded. Deployments worked around it with a synthetic "Done"
fallback patch.
It went silent entirely above ~1,200 prompt tokens — no content, no
tool call. Real Home Assistant prompts are ~2,900 tokens, squarely in
the failing range. This was a capability limit, not a context limit; no
num_ctx value fixed it.
This model was trained and validated specifically against both.
Validation
Measured with a schema-driven battery: 28 utterances × 3 repetitions
against a 17-tool Home Assistant schema, temperature 0.6, at production
prompt size.
Model
Total
State queries
Destructive errors
No speech after tool
this model
81/84 (96%)
15/15
0
0/72
stock granite-4.1-3b
70/84 (83%)
13/15
2
1/81
Per category: state queries 15/15 · cameras 12/12 · noise robustness 9/9 ·
refusals 3/3 · actions 42/45.
"Destructive errors" means a question answered with an action tool
— e.g. replying to "which lights are on?" by turning lights on, or to
"is the front door locked?" by unlocking it. Stock Granite does the
latter 2 times in 3. This model does neither.
Performance
Measured on an RTX 4080 SUPER (16 GB), Ollama, num_ctx 65536:
this model
v1 7B
Latency (production prompt, 16 tools)
240 ms mean
341 ms
Throughput
109.6 tok/s
—
Context
65,536 (fully GPU-resident)
8,192
VRAM
8.54 GB
5.11 GB
Size on disk
3.6 GB
4.7 GB
128k context spills layers to CPU on a 16 GB card; 64k is the practical
ceiling and was verified fully resident.
Files
File
Size
Use
ha-voice-granite-3b-Q8_0.gguf
3.37 GB
Recommended. 8.5 BPW, minimal quantization loss
ha-voice-granite-3b-f16.gguf
6.34 GB
Full precision, for further quantization
A 3–4B model at Q8_0 costs roughly what a 7B costs at Q4_K_M, at much
higher fidelity. Aggressive quants (IQ3/IQ4) were found unusable for this
task in earlier work; do not go below Q4_K_S.
Usage — Ollama
num_ctxmust be set explicitly in the Modelfile; a missing value
silently inherits the server default.
dockerfile
1FROM ./ha-voice-granite-3b-Q8_0.gguf2PARAMETER num_ctx 65536
3# Use the stock Granite 4.1 chat template so <tool_call> tags are parsed4# into OpenAI-format tool_calls:5# ollama show granite4.1:3b --modelfile
ollama create ha-voice -f Modelfile
The template matters. Home Assistant's OpenAI-compatible integrations
read message.tool_calls; without the Granite template the model's
<tool_call> tags arrive as plain text and no tool ever executes.
3 epochs / 1,743 steps, cosine, lr 2e-4, warmup 3%
Masking
Completion-only — loss on assistant turns only
Hardware
NVIDIA DGX Spark (GB10), 24 h 52 m
Final loss
train 0.0338 · eval 0.0253
The dataset lesson worth sharing
An earlier attempt at this model advertised seven tools in thousands of
prompts and never demonstrated a single call to any of them. Three
epochs taught the model to route around those tools toward whatever was
demonstrated. The result: asking "which lights are on in the master
bedroom?" made it call HassTurnOn — it turned the lights on — 0/3,
where the un-tuned base model answered correctly 3/3.
Fine-tuning destroyed a capability the base model already had.
If you fine-tune for tool calling, audit coverage in both directions:
every tool your data calls must exist, and every tool your schema
offers must actually be called. A tool that is advertised but never
demonstrated is worse than one you never offer.
Intended use
Home Assistant voice assistants via an OpenAI-compatible endpoint
(Ollama → LiteLLM → HA). Trained on HA intent schemas —
HassTurnOn/Off, HassGetState, HassLightSet,
HassClimateSetTemperature, HassMediaPause, HassSetVolume,
HassSetPosition, HassCancelAllTimers, list intents, and
parameterized camera scripts.
Limitations
Known issue: with large tool schemas (~17+ tools), "add X to the
shopping list" can select the list-complete intent instead of
list-add. Correct with smaller schemas. A training-data artifact; it
is non-destructive and will be fixed in a future revision.
Entity and area names in training were randomized for generalization;
behaviour on a specific home depends on what Home Assistant exposes.
English only.
The camera intents are specific to a two-script parameterized setup and
will not match other configurations.
License
Apache-2.0, inherited from the Granite 4.1 base model.
