All three files are imatrix-calibrated PTQ (importance-matrix post-training quantization),
calibrated on a 534-example travel corpus weighted toward emergency (134 examples, all
available) and food (200 examples) domains. Quantized with llama-quantize --imatrix.
Intended Use & Scope
Target use case: consumer travel translation — ordering food, navigating transportation,
accommodation, shopping, and emergency communication in Japan.
Food and emergency domains are explicitly weighted in both fine-tuning (food is 42.5% of
training pairs) and in the importance matrix calibration (emergency is the heaviest-weighted
domain by proportion). The motivating scenario is a foreigner with no internet access facing
a Japanese-only emergency line — "Please call an ambulance" and "I am having an allergic
reaction" must be correct.
This model is a credible demonstrator with rigorous safety-aware evaluation. It is not a
deployable medical, legal, or defense product. The consumer travel app is the near-term target.
See the safety architecture note below for how life-critical phrases are handled in the intended
deployment.
Not in scope: medical advice, legal documents, regulated translation workflows, or any
context where a confident-wrong output carries high-stakes consequences.
Safety Architecture
The intended deployment is a hybrid phrasebook + LLM system:
Deterministic, human-verified phrasebook handles life-critical phrases: calling an
ambulance, reporting an allergic reaction, "I can't breathe", "I'm lost". These are fixed
strings — no probability, no hallucination risk.
The LLM handles the open-ended long tail: menus, negotiations, descriptions — anything
that formulaic phrases don't cover.
A 4-bit quantized LLM alone must not be relied on for life-critical phrases. Confident-wrong
is the worst failure mode in an emergency; the phrasebook is the safety net.
Evaluation
Measured on a 156-item held-out gold test set (78 EN→JA + 78 JA→EN), food- and
emergency-weighted, with all 44 emergency references human-verified. Evaluated at the desktop
inference stage using llama-server with plain-text prompts on stock llama.cpp b9716.
Label
Size (MB)
COMET (all)
Δ vs bf16
food COMET
emerg COMET
BLEU en→ja
BLEU ja→en
bf16
1,515.2
0.9001
±0.0000
0.8960
0.8913
48.87
39.99
Q8_0
806.2
0.9022
+0.0021
0.8971
0.8925
49.79
40.35
Q5_K_M
569.2
0.9000
−0.0001
0.8933
0.8925
48.92
41.32
Q4_K_M
503.7
0.8988
−0.0013
0.8892
0.8986
47.96
38.97
COMET model:Unbabel/wmt22-comet-da. BLEU tokenizers:char for EN→JA (sacrebleu), 13a for JA→EN (sacrebleu).
Why Q4_K_M: COMET delta −0.0013 vs bf16 is inside the 0.005 negligible threshold, at
67% smaller. Emergency gate clears: Q4 emerg COMET 0.8986 vs bf16 0.8913 — zero emergency
degradation at 4-bit (the small apparent gain is within n=44 noise; the honest claim is no
degradation). The entire quant ladder sits within ~0.003 COMET; quant choice collapses to
smallest-that-fits-the-phone.
Read these caveats before drawing conclusions
COMET ≈ parity is the reliable signal. Do not read small BLEU swings between quant
levels as quality conclusions — the variation is within small-sample noise.
These 156-item numbers are NOT comparable to Phase 3 (fine-tune) numbers. The earlier
40-item eval set contained shorter, more formulaic sentences. The 156-item set includes
longer, more varied sentences; BLEU naturally drops on harder sets. The difference is not
a regression.
EN→JA BLEU uses char tokenizer; JA→EN uses 13a. Mixing tokenizers makes numbers
incomparable. If you re-evaluate, pin these tokenizers explicitly.
Differentiation
The value proposition is pragmatics, not BLEU. Incumbent offline NMT (Google Translate
offline ~47 MB, Apple ~100 MB) wins literal character-BLEU. This model's edge:
Pro-drop resolution — correctly inferring dropped subjects/objects from context in Japanese
Keigo — appropriate politeness register for service interactions (restaurant, hotel, shop)
Cultural and food context — menu descriptions, allergen terms, regional food vocabulary
Do not frame this as a BLEU race against Google or Apple. Frame it on pragmatic correctness
in travel scenarios that require cultural and register awareness.
Critical Tokenizer Note
This GGUF was built with a patched convert_hf_to_gguf.py and tokenizes correctly only
in llama.cpp builds that use the resulting tokenizer.ggml.model = "t5" path.
Background:cyberagent/CAT-Translate-0.8b uses a SentencePiece Unigram tokenizer
(inherited from sbintuitions/sarashina2.2-0.5b). The stock llama.cpp conversion script
hardcodes tokenizer.ggml.model = "llama" for all SPM models, routing the runtime into the
greedy-merge SPM path. SPM greedy-merge ≠ Unigram Viterbi — it produces silently wrong token
splits. All known community GGUFs of this model family (mradermacher, mmnga-o) have this
bug.
The fix (applied at conversion time):llama.cpp/conversion/base.py::_set_vocab_sentencepiece()
was modified to (1) detect tokenizer.json → model.type == "Unigram" and emit
tokenizer.ggml.model = "t5", routing llama.cpp into its LLAMA_VOCAB_TYPE_UGM Viterbi
path; and (2) selectively promote BYTE tokens to NORMAL type only for byte characters that
have no existing NORMAL token — so control characters like \n tokenize correctly without
triggering the duplicate-text assertion crash (GGML_ASSERT(id_to_token.size() == token_to_id.size())).
Validation on stock llama.cpp b9716 (2026-06-19):/tokenize on the template string:
25/25 tokens match HF output exactly, including \n\n = [25, 25]. Broad parity test over
30 diverse strings: all match. COMET ≈ HF parity in both directions on the 40-sentence eval.
Verify your build: run /tokenize on a Japanese string and compare against HF tokenizer
output. If your llama.cpp or PocketPal build uses an unpatched conversion (model string
"llama" instead of "t5"), tokenization will be silently wrong and translation quality
will degrade. PocketPal validation on a real device remains a required gate before any app
deployment — b9716 desktop testing is a strong proxy, not a guarantee.
Usage
llama-cli
bash
1llama-cli \2 -m cat-translate-0.8b-travel-Q4_K_M.gguf \3 --no-warmup \4 -p "Translate the following English text into Japanese.\n\nI'd like a table for two."
llama-server (OpenAI-compatible)
bash
1# Start server2llama-server -m cat-translate-0.8b-travel-Q4_K_M.gguf --port 808034# Query5curl http://localhost:8080/completion \6 -H "Content-Type: application/json"\7 -d '{"prompt": "Translate the following English text into Japanese.\n\nI have a peanut allergy.", "n_predict": 128}'
Use plain-text prompts only — no chat template, no system message. Template:
"Translate the following {src_lang} text into {tgt_lang}.\n\n{source}"
For JA→EN: "Translate the following Japanese text into English.\n\n救急車を呼んでください。"
PocketPal (iOS / Android)
Open PocketPal → Models → Add Model → enter the Hugging Face repo ID
kaiish/capynode → select cat-translate-0.8b-travel-Q4_K_M.gguf.
Enable airplane mode after download to confirm fully offline operation.
Running in a chat app (PocketPal, etc.)
This model is single-turn and uses no system message. Chat UIs that wrap input in a
default system prompt, or that accumulate conversation history across turns, will break it —
the model drops out of translation mode and responds conversationally instead. This is
expected behaviour: CAT-Translate is invoked purely by the instruction in a single user turn,
not by a persistent persona or context.
Recipe validated working fully offline on iPhone 11 in PocketPal:
System prompt: empty — clear any default system message in the model settings.
One translation per conversation — start a fresh chat for each translation; do not
carry history between requests.
Temperature 0 (greedy) for clean, deterministic output.
Jinja chat-template ON — use the embedded template (enabled by default in PocketPal
when the GGUF contains a tokenizer.chat_template key, which this one does).
For app integration, the cleaner path is to call the model with the instruction as a single
user turn (or as a raw text completion) and no history — which is natural for a translation
app since each translation is independent anyway.
Validated offline on iPhone 11 (4 GB) via PocketPal: correct EN→JA and JA→EN including
emergency and food/allergen phrases; UGM tokenizer confirmed intact on the device build.
CC BY-NC-SA 4.0 — excluded (non-commercial clause)
Final training corpus: 6,458 train / 718 validation examples (bidirectional, sarashina2.2
instruct template). Domain distribution: food 42.5%, shopping 17.5%, transportation 15%,
accommodation 15%, emergency 5%, general 5%.
License & Attribution
License: MIT. See LICENSE for this repository's terms.
This GGUF is built on two upstream MIT-licensed components. Their copyright and permission
notices are preserved verbatim in the bundled license files below — these are the authoritative
texts that must be carried forward in any distribution or deployment (e.g., in an app's About
or credits screen):
Cleared for: quantizing, re-hosting on HuggingFace, and shipping in a closed-source paid,
subscription, or ad-supported app, provided both bundled license files above are preserved.
