gemma4-ushistory-apush-v2-GGUF
GGUF quantizations of
ambrosfitz/gemma4-ushistory-apush-v2
(
epoch 3, merged 16-bit), for local inference via llama.cpp / LM Studio.
| File | Size | Notes |
|---|
gemma4-ushistory-apush-v2-e3-Q4_K_M.gguf | 7.38 GB | Structural format adherence weaker than Q5 on this smoke test (see below) |
gemma4-ushistory-apush-v2-e3-Q5_K_M.gguf | 8.55 GB | Recommended — cleaner structural adherence, verified 7-8/8 on smoke test |
Why epoch 3, not epoch 2
Per-epoch intra-set duplicate rate during training (HF Transformers, from the
source repo's TRAIN_REPORT.md):
| Epoch | greedy | temp 0.7 |
|---|
| 1 | 0.133 | 0.100 |
| 2 | 0.100 | 0.067 |
| 3 | 0.150 | 0.067 |
Epoch 2 scored better on greedy duplicate rate (0.100 vs 0.150); temp 0.7 was
tied (0.067). Epoch 3 was the chosen conversion target per this project's
handoff, not because it measured better on this specific metric. The epoch-2
adapter remains available in the source repo (adapter/epoch2/, with full
optimizer/scheduler state) if a re-convert from that checkpoint is ever wanted.
Required generation recipe
This model was trained and evaluated with a specific chat template and stop-token
setup, inherited from the same lineage as
gemma4-ushistory-sft-parityfix-GGUF.
Getting either wrong produces malformed or garbage output.
- Chat template:
gemma-4, non-thinking mode. The correct jinja template
(verified to contain the strip_thinking macro and emit <turn|> correctly) is
embedded in this GGUF's own metadata (tokenizer.chat_template) — confirmed
present and intact via gguf.GGUFReader directly on the conversion pod. Do
not override it with a runtime's built-in "gemma" preset — llama.cpp ships a
built-in template named gemma for the classic Gemma 1/2/3 format
(<start_of_turn>/<end_of_turn>), incompatible with this checkpoint's
Gemma-4 format. Leave the chat template on default (LM Studio: don't override
it; llama.cpp: don't pass --chat-template).
- Stop token:
<turn|> (vocab id 106, confirmed present in the GGUF's vocab
at that id). This is not encoded as an EOS/EOG token in GGUF metadata —
tokenizer.ggml.eos_token_id is [1] (<eos> only) — confirmed directly on
the conversion pod. You must set <turn|> as an explicit stop string in your
runtime (LM Studio: add it under the model's stop-string/sequence setting).
This was verified end-to-end at generation time (see below): once given an
adequate token budget, both quants stopped cleanly at <turn|> on 100%/88% of
smoke-test items — the mechanism works, but only because the stop string is set
explicitly. Without it, generation would run past the intended turn boundary.
Verification
Both quants were served on-pod (RTX 3090) via llama-server (no --chat-template
override) and smoke-tested against 8 real held-out stimuli — the identical
seed=42 validation split used for this model's own epoch-by-epoch training-time
sanity generations — at both greedy and temperature 0.7/top_p 0.95, with
stop=["<turn|>"] explicit in every request.
| Quant | Setting | Parse | Structural | Stop-clean | Dup rate |
|---|
| Q4_K_M | greedy | 8/8 | 3/8 | 8/8 | 0.150 |
| Q4_K_M | temp0.7 | 8/8 | 4/8 | 8/8 | 0.013 |
| Q5_K_M | greedy | 8/8 | 7/8 | 7/8* | 0.171 |
| Q5_K_M | temp0.7 | 8/8 | 7/8 | 7/8* | 0.073 |
* Both Q5_K_M "unclean" cases are the same single stimulus whose natural
response (5-6 fully-explained questions) exceeds even a 2048-token budget —
confirmed well-formed up to the cutoff, i.e. a budget limit, not a stop-token
failure.
The <turn|> stop-token risk this conversion was built around did not
materialize: once given an adequate token budget, Q4_K_M reached 100%
stop-clean and Q5_K_M 88% (with the sole "failure" being budget exhaustion on an
unusually long response, not a metadata problem). Duplicate rates on both quants
(0.013–0.171) are in the same range as this model's own HF Transformers
training-time numbers (0.067–0.150), with Q4_K_M-greedy matching almost exactly
(0.150) — no evidence quantization introduces or worsens repetition.
Q4_K_M does show real, reproducible structural degradation that Q5_K_M mostly
avoids: on specific stimuli, Q4_K_M consistently produces a well-formed
numbered question set with 4 choices each, but omits Correct:/Explanation:
entirely for every question, identically at both greedy and temp 0.7 (ruling out
decoding-randomness noise). Q5_K_M shows almost none of this. This was not
cross-checked against the unquantized fp16 model on these same stimuli, so
whether this is introduced by Q4_K_M's quantization specifically or is a
pre-existing model weak point on this stimulus type is an open question — flagged
rather than overclaimed. Full per-item results and raw malformed outputs in this
repo's gguf_verification.md.
Decision: start local use from Q5_K_M — Q4_K_M showed malformed output on
this smoke test that Q5_K_M did not.
Recommended local config
- Full GPU offload (
-ngl 99 or LM Studio's "GPU offload: max") — both quants fit
comfortably on a 12GB card
- Context length:
-c 4096
- Chat template: leave on default / embedded (see above — do not override)
- Stop string: add
<turn|> explicitly (see above — not in metadata)
- Given the structural-adherence gap above, prefer Q5_K_M for anything beyond
casual testing
Task
Source excerpt (primary or secondary source) in → a set of AP US History-style
multiple-choice questions out (2-8 questions per set observed in training data,
each with a stem, 4 choices A-D, a Correct: key, and an Explanation:).
Training provenance
- Base: Gemma 4 (12B), continued from
ambrosfitz/gemma4-ushistory-sft-parityfix
- SFT: LoRA (r=32, alpha=64) on
ambrosfitz/apush-synthetic-mcq (895 synthetic
train / 47 held-out val, seed=42; a separate 286-set authentic eval split
exists in the dataset for answering-accuracy evaluation, not used in this
generation-focused fine-tune)
- Masking: Unsloth's
train_on_responses_only (marker-string based, independent
of chat-template {% generation %} tags), with a blocking step-0 mask gate
that verified the mask structurally correct before training started —
assistant_only_loss was deliberately left unset, unlike the prior
gemma4-ushistory-apush-continued run whose masking status could not be
determined post-hoc
- Result: intra-set duplicate rate dropped roughly an order of magnitude versus
the prior (mask-unverified) run, across all 3 epochs
- Merged epoch-3 model → this GGUF conversion
Full training report:
TRAIN_REPORT.md
in the source repo.
