Trusty: Gemma Home Assistant (Q4_K_S, quantised GGUF)
A fine-tuned and quantised build of Gemma 4 E2B IT for
Trusty, a privacy-first local
voice assistant that runs entirely on a Raspberry Pi 5 (or any Apple
Silicon / Linux dev box). The model handles the planner role in
Trusty's voice pipeline: it reads the user's transcript and emits a
strict JSON tool call (one of home.tv, home.vacuum, music,
weather.live, internet.search, memory, local.answer, or none).
This repo ships only the Q4_K_S GGUF (~3.1 GB), produced by
LoRA fine-tuning the base Gemma weights and then quantising the merged
model to 4-bit with an importance matrix (imatrix) calibrated against
the same training distribution. The original f16 / merged checkpoints
are not included: point llama.cpp directly at the GGUF below.
All inference is local via llama.cpp. Only the chosen text query ever
leaves the device, never microphone audio or home sensor logs.
new header
Why this tuned model is different
Stock Gemma 4 E2B needs an ~ 11 KB system prompt to route tool calls reliably. On a Raspberry Pi that long prompt dominates every voice turn (~ 100 s end-to-end) and the planner still emits valid JSON only about 50% of the time.
The Trusty-tuned build was fine-tuned on ~ 14K synthetic planner examples covering every tool in Trusty's catalogue. It does the same routing job with a ~ 1.5 KB prompt (~7× shorter), emits valid JSON on every turn, and brings Pi 5 end-to-end latency from ~ 100 s to ~ 17.6 s (5.7× faster). Filename-driven prompt selection in the orchestrator auto-loads the short planner prompt when the model file contains trusty, so the same code path serves both tuned and un-tuned weights.
The Space sends your text to the model and shows the planner's JSON tool plan; best for testing the home-orchestrator routing behaviour on its own. For the full experience (wake word, voice loop, character Eyes UI, and connected tools for vacuum, weather, music, and web search) set up the complete stack from github.com/Barqawiz/Trusty.
The filename intentionally contains trusty. The Trusty orchestrator
detects that substring in GEMMA_MODEL_PATH and automatically loads the
short 1.5 KB planner prompt (prompts/planner_system.md). Use any other
filename (e.g. plain gemma-4-e2b-it.gguf) and it falls back to the long
11 KB original planner prompt.
What's in this repo
This is a quantised-only release. The full-precision merged checkpoint
isn't published: re-deriving it requires the LoRA adapter from the
Trusty repo plus the original
Gemma base weights.
File
Size
Quant
When to use
trusty-gemma-4-e2b-tuned-q4_k_s.gguf
3.1 GB
Q4_K_S (imatrix-calibrated)
Default: best speed / quality tradeoff on Pi 5 and Mac. This is the only file shipped.
The Q4_K_S quant was built with an importance matrix calibrated against
the same training distribution (4,000 representative planner messages),
so it routes correctly where un-tuned Q4_K_S routes everything to
local.answer.
Benchmarks (Raspberry Pi 5, 8 GB)
End-to-end /chat latency through the Gemma planner. Identical 3
prompts (capital of France, 12 times 7, how does a camera work)
across all rows.
Build
Planner prompt
avg / turn
speedup
JSON-valid
Un-tuned gemma-4-e2b-it-q4_k_s.gguf
long, 11 KB (the un-tuned model needs this much instruction to route correctly)
~100 s
1.0×
50 % (planner fallback fires often)
trusty-gemma-4-e2b-tuned-q4_k_s.gguf
short, 1.5 KB (auto-selected by the orchestrator when the filename contains trusty)
17.6 s
5.7×
100 %
The tuned build uses a ~7× smaller planner prompt and emits valid JSON
on every turn (vs ~50 % for un-tuned), saving roughly 82 s per turn on
Pi 5. Warm voice turns land at 16-20 s for weather/facts and
20-35 s for short stories.
Training summary
Base: google/gemma-3n-E2B-it (4-bit MLX checkpoint stripped to text-only).
Method: LoRA (rank 16, alpha 32, dropout 0), 16 trailing transformer blocks, ~1.05 M trainable params (0.023 % of base).
Optimizer: AdamW, learning rate 2e-4.
Early-stop: probe-based (5-prompt structural validity), triggered at iter 800/10476. Lowest val loss (0.092) re-pinned.
Quantization: convert_hf_to_gguf → f16 → llama-imatrix over 4 K calibration samples → llama-quantize Q4_K_S with imatrix.
Training data
13,971 train + 1,918 val rows, all synthetic JSONL (no user
recordings, no personal data, no scraped chat logs). Each row is a
three-message conversation: a fixed system prompt, a user transcript,
and the exact target JSON the planner should emit. Distribution across
target tools:
Target tool
Train rows
What it teaches
local.answer
3,740 (26.8 %)
math, capitals, conversions, science / history / astronomy facts, language and spelling, cooking and household tips, company origins, general-tech "how does X work", privacy-question answers, persona answers, cheating refusal, capabilities summary.
internet.search
3,171 (22.7 %)
"search / google / look up / find me" verbs, latest X, current officials, live events, brand products, stocks/crypto/news, current-year temporal markers. requires_internet=true enforced.
weather.live
2,653 (19.0 %)
weather/temperature/rain queries with a city slot, including STT mishears (wither, wheather) and "what to wear" variants.
music
1,531 (11.0 %)
play named songs / artists / genres, pause / resume / stop / next / volume; varied STT phrasings.
memory
1,218 (8.7 %)
set_name only on explicit my name is X / call me X / I am X, set_location, clear. STT mishear variants.
privacy violations (audio / microphone / home logs → blocked) and ask_for_location when weather has no city.
home.tv
252 (1.8 %)
TV power, volume, mute, open app (Netflix / YouTube / Disney+ etc.).
Routing examples (local vs internet)
The model keeps static facts on-device and only hits the web when the
answer genuinely changes over time.
User says
Route
what is the capital of France
local.answer
who was president of the US in 1995
local.answer (historical)
who is the current president of the US
internet.search (live)
where is Toyota from
local.answer
latest Toyota cars
internet.search
what is the weather in Dublin
weather.live
play happy birthday
music
stop the vacuum
home.vacuum
Prompt format
Gemma chat template (<start_of_turn>user / <start_of_turn>model). The
output is always a single JSON object with nine fields (tool, action,
arguments, requires_internet, external_payload, privacy_risk,
reason, final_response_required, local_answer). Full prompt at
prompts/planner_system.md
in the Trusty repo.
Privacy
This model only sees text. The host pipeline (Trusty) hard-locks any
tool from sending microphone audio or home-sensor logs. Every turn is
recorded in a privacy ledger at data/privacy_ledger.jsonl on the
local device.
License
This model is released under the Gemma Terms of Use (same as the
base model). Downloading or using these weights constitutes acceptance
of those terms: see https://ai.google.dev/gemma/terms.
Citation
@misc{trusty2026,
title = {Trusty: a privacy-first local voice assistant},
author = {Barqawi, Ahmad},
year = {2026},
url = {https://github.com/Barqawiz/Trusty}
}
