4-bit llama.cpp build of StreamRevise, a translation model for live subtitles. Every time the ASR
hypothesis updates, you pass the model its own previous translation of the sentence in progress; it decides
whether to keep that text and extend it, or rewrite part of it because the meaning changed. Subtitles grow
smoothly instead of flickering.
1.07 GB on disk, and about 1.4 GB resident at n_ctx 2048. Small enough to ship with a desktop app.
See Footprint for how that scales.
the model. Q4_K_M with imatrix calibration, token embeddings at Q4_K
Hy-MT2-1.8B-StreamRevise.imatrix.gguf
2.3 MB
importance matrix, only needed if you want to re-quantize at another bit width
Only one quantization is published. The alternatives were measured and the differences sit inside the noise
floor, so the smallest one shipped:
variant
size
verdict
Q4_K_M, embeddings Q4_K
1.07 GB
published here
Q4_K_M, embeddings Q8_0
1.19 GB
+0.5pp exact match — noise
Q4_K_M, embeddings F16
1.42 GB
no measurable gain
Q4_K_M, defaults
1.92 GB
llama-quantize promotes embeddings to F32. No quality gain, 470 MB wasted
Embeddings are tied to lm_head in this architecture, which is why their bit width gets its own column.
imatrix calibration used real streaming-translation prompts, which is what makes 4-bit embeddings safe here.
temperature: 0 — greedy, not the base model's 0.7. Consecutive updates hand the model nearly the
same prompt and you want nearly the same output back; sampling introduces changes that have nothing to do
with the source changing, and those land on screen as flicker. All numbers below are greedy.
cache_prompt: true — updates within one utterance share a long prefix. KV reuse is what makes
per-update latency negligible.
{PROMPT} must use the StreamRevise layout, not a plain translation instruction.
A warning you can ignore
llama.cpp prints this on load:
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special_eom_id is not in special_eog_ids - the tokenizer config may be incorrect
It is harmless here. Hy-MT2 declares a single-value eos_token_id, and this build carries an explicit
tokenizer.ggml.eom_token_id = 120020 that llama.cpp does not fold into its special_eog_ids set — hence
the complaint. Generation still terminates correctly: expect stop_type: eos and a predicted_n far below
your n_predict. If instead the model never stops, you are running a GGUF converted without the
end-of-message fix — see below.
Prompt format
Full spec and a copy-paste renderer are in the
LoRA repo. Short version.
First chunk of a new sentence, nothing to revise yet:
text
1Translate the following text into {TARGET_LANGUAGE}. Note that you should only output the translated result without any additional explanation:
23{CURRENT_SOURCE}
Every update after that:
text
1[Background Information]
2Recent source utterances:
3{UP_TO_10_PREVIOUS_SOURCE_SENTENCES}
45Previous version of the current source:
6{PREVIOUS_SOURCE}
78Previous translation of the current source:
9{PREVIOUS_TRANSLATION}
1011When the source meaning has not changed, preserve the still-correct prefix of the previous translation whenever possible. When content is added or corrected, accuracy and completeness take priority.
1213Please translate the following text into {TARGET_LANGUAGE}, taking the provided background information into consideration.
1415[Source Text]
16{CURRENT_SOURCE}
Background blocks are each optional, joined by a blank line, always in that order. {TARGET_LANGUAGE} is a
full English language name. Recent source utterances carries source text only, never the translations.
The model keeps no state between requests — the caller owns the revision chain and sends it in full every
time. That means you can retry, reorder, or drop requests freely.
Speed
short prompt (33 tok)
long prompt (137 tok)
GPU (A40, CUDA, -ngl 99)
0.06 s, 256 tok/s decode
0.06 s, 337 tok/s decode
CPU (56-thread server)
0.53 s, 17 tok/s decode
1.16 s, 17 tok/s decode
Rough expectations elsewhere: 8-core desktop CPU ~10–20 tok/s; Apple Silicon (Metal) ~30–60+ tok/s; any
discrete GPU with ≥2 GB VRAM (CUDA or Vulkan) ~50–300 tok/s.
For live subtitles: under 100 ms per update on a GPU, 0.5–1.2 s on CPU alone — usable, if not instant.
Footprint
Weights are ~1.02 GiB once loaded. The KV cache is the only part that scales with your settings: this is a
32-layer model with 4 KV heads at head_dim 128, so 64 KiB per token at f16.
n_ctx
KV cache
ballpark resident
1024
64 MiB
~1.3 GB
2048
128 MiB
~1.4 GB
4096
256 MiB
~1.5 GB
8192
512 MiB
~1.8 GB
Measured: raising -c from 2048 to 4096 costs exactly 135 MB. --parallel costs nothing extra — -c is the
total KV budget, divided among slots, not multiplied by them.
Two things push the number above these estimates, and neither is the model's doing: the CUDA context and
allocator pool (often 300–500 MB, and nvidia-smi reports it against your process), and compute buffers
sized by -b/-ub. If you are counting VRAM on a small card, lower -c first — that is the part you
control.
What 4-bit costs
843 trajectories / 2,438 states, same prompts and greedy decoding on both sides.
bf16
this build
state exact match
0.287
0.282
mean state similarity
0.814
0.801
final exact match
0.199
0.211
characters erased per append
2.56
2.32
prefix preserved (append transition)
0.798
0.824
prefix preserved (final transition)
0.917
0.925
empty-output rate
0.000
0.000
About 1.3pp of similarity and half a point of exact match. The stability rows come out level or slightly
ahead, which is within noise. Full table in the
LoRA repo.
If you convert Hy-MT2 yourself
llama.cpp's HunYuan converter doesn't write an eog/eom token when the source model has a single-valuedeos_token_id — and Hy-MT2 does. Without a fix, generation never stops and you get endless repetition.
This build already has the fix applied; its metadata carries tokenizer.ggml.eom_token_id = 120020. If
you're converting from scratch, append this to _fix_special_tokens() in conversion/hunyuan.py:
python
1eos = self.hparams.get("eos_token_id")2if eos isnotNone:3 ids = eos ifisinstance(eos,list)else[eos]4 self.gguf_writer.add_eom_token_id(int(ids[0]))
Converted and quantized with llama.cpp b10442.
Limitations
Stability is a tendency, not a guarantee. Nothing enforces prefix preservation; worst case a single
update rewrites the whole line. If your UI can't tolerate that, only display the prefix that has held
steady for N updates.
Prompt format matters a lot — off-format prompts lose quality and stability.
Language coverage is uneven: zh/en/ja are what it was trained on and measured on. Other directions
fall back to base-model behaviour, untested.
Greedy decoding assumed.
Metric definitions are project-internal; don't compare them against numbers from simultaneous-MT papers.
