Cipher Nano is a full-parameter fine-tune of h2oai/h2o-danube3-500m-chat, trained on every LLM-backed feature of a local-first email assistant: email triage (importance/summary/category JSON), chat, daily-summary synthesis, draft reply, compose assist, and voice-command intent parsing — not just prompted for these tasks, actually trained on them.
Update: added a 6th task, voice-command intent parsing. Nano's smaller full-fine-tune
capacity means it still sometimes substitutes a plausible-looking wrong address instead of
copying the real one verbatim from the candidate list — 3 rounds of targeted oversampling
narrowed but didn't eliminate this. The actual fix ended up being architectural, not a
bigger model: the calling app never trusts an LLM-supplied email address at all anymore,
re-deriving it deterministically in code from the model's extracted name against the real
candidate list instead. That makes this habit irrelevant regardless of which Cipher tier is
active — see generate_voice_intent.py's training notes and eval_voice_intent.py for the
full detail.
Second update: an incremental continue-training round aimed at voice-intent (done on top
of the update above) turned out to have a side effect elsewhere: manual testing found this
model would occasionally fabricate a specific false detail not present in its given context
(e.g. inventing a person's surname that was never mentioned) on the chat task. Rather than
patch that narrowly on top of an already-long chain of incremental updates, this model was
retrained fresh from the base in one pass on a properly-rebalanced dataset covering all 6
tasks together, with substantially richer prompt-injection coverage across draft-reply and
chat (credential-phishing, wire-transfer, and data-exfiltration attempts, not just one
generic case) and an explicit "the question has nothing to do with your email at all"
scenario category to close the specific hallucination gap that was found. Verified through
repeated testing against the real calling application's code path: no compliance with any
injection attempt, and the specific hallucination case now correctly reports the missing
information instead of inventing it.
It's the smallest of the three Cipher tiers (cipher-nano / cipher-air / cipher-pro), built specifically to run on modest hardware — verified at 29.4 tok/s on a 10-year-old Intel i3-6100T with 8GB RAM, no GPU, with accuracy holding essentially flat versus a high-end GPU. Cipher is the local-model engine for an unreleased larger email-assistant project — that project isn't public yet, but these weights, the training code, the eval script, and all five dataset generators are fully open now, in this repo.
Why this exists
Most email triage today means sending your inbox to a third-party API. Cipher runs entirely on your own hardware via Ollama — nothing about your email ever leaves your machine. Cipher Nano exists so that "your own hardware" can mean literally any hardware, not just a machine with a GPU.
Why Danube3-500M and not something smaller
We tried harder-to-shrink options first and hit real walls:
SmolLM2-135M/360M (49K-token vocab) genuinely shrinks to 100-270MB, but its base pretraining wasn't strong enough to reliably learn category/importance calibration — tried LoRA vs. full fine-tune, epoch sweeps, and reshaped training data, and category accuracy stayed stuck in the 40-65% range regardless.
Qwen2.5-0.5B / Gemma3 have strong pretraining but 152K/256K-token vocabularies that dominate disk size and don't shrink with quantization (confirmed: Q2_K barely shrank vs Q4_K_M).
Danube3-500M is the first candidate with both a small vocabulary (32K tokens) and pretraining strong enough for the task — category accuracy jumped to 70%+ on the first fine-tuning attempt, no extensive tuning needed. Its full-parameter fine-tune (100% of weights retrained, vs. cipher-air/cipher-pro's LoRA) also gives it total plasticity to fully adopt the required output format, no competing base-model habit to fight against.
Benchmark
Evaluated on a 29-fixture triage benchmark across five machines, GPU and CPU-only:
Full retrain (current version, injection/hallucination hardened)
106.7*
100.0%
82.8%
79.3%
* Tok/s measured while the GPU was concurrently running an unrelated training job — not a
like-for-like comparison with the isolated numbers above; accuracy figures are unaffected.
Accuracy holds essentially flat across every machine tested — only throughput changes. Even the weakest machine here, a decade-old office PC with no GPU, does ~2 seconds per email, well within range for a background triage task. Reproduce with:
Danube3's chat template does not support a system role and uses non-ChatML turn markers — the included Modelfile sets an explicit TEMPLATE to match. Don't rely on Ollama's template autodetection with this base model (confirmed live: without an explicit TEMPLATE, Ollama couldn't resolve a working chat format for this GGUF at all).
ollama create cipher-nano -f Modelfile
Query it with grammar-constrained JSON output for reliable parsing:
If you're integrating this into your own app rather than using Ollama: llama-server
(llama.cpp's own server binary) handles this model's real chat template correctly on its
own — verified directly, no override needed there, only for Ollama's simpler
auto-detection.
Training
Base: h2oai/h2o-danube3-500m-chat (plain LlamaForCausalLM, 32,000-token vocab), full fine-tune (100% of parameters, not LoRA), 3 epochs
Data: ~4,800 triage examples + ~1,600-2,000 examples each for chat/daily-summary/draft-reply/compose, all matching production prompts exactly — generated by the five generate_*.py scripts in this repo; training text reformatted to Danube3's native <|prompt|>/<|answer|> turn format since it has no system-role support (see train_cipher_nano.py)
Sequence packing was tried to speed up training (most examples are well under the
2048-token context window) — it crashed outright, an Unsloth/trl version incompatibility,
not a quality tradeoff. Disabled.
Reproduce with train_cipher_nano.py → export_gguf_cipher_nano.py
Voice-intent retrain: added generate_voice_intent.py's data (weighted heavily toward
the no-match case and exact-copy-under-distractor-pressure scenarios) to the mix across 3
rounds of oversampling. Nano's smaller full-fine-tune capacity meant this narrowed but
never fully eliminated one specific habit — substituting a different plausible-looking
address instead of copying the real one verbatim — see the README intro above for how
this ended up getting solved at the calling-app level instead. Reproduce/verify with
eval_voice_intent.py.
Full retrain (current version): one clean pass over the base model on a single
consolidated, properly-balanced dataset covering all 6 tasks, rather than continuing to
layer incremental patches on top of each other — done specifically to fix both the
injection-compliance issue and the hallucination regression noted above without risking
a whack-a-mole cycle where each narrow fix nudges a different, previously-working case.
Followed by one small, targeted continue-train pass (~700 examples, 2 epochs, lr=2e-5)
reinforcing "report missing information plainly rather than inventing a plausible-
sounding detail," specifically targeting the exact hallucination case found in testing.
License
Apache 2.0, inherited from the base model. Weights, training code, and eval harness are fully open.
