Drops articles, filler, pleasantries, hedging. Allows fragments. Keeps code blocks, function names, error strings, and CLI commands byte-exact. Pattern: [thing] [action] [reason]. [next step].

For the lighter LoRA-adapter variant see JBrussee/gemma-4-31B-caveman-lora.

Use

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("JBrussee/gemma-4-31B-caveman")
model = AutoModelForCausalLM.from_pretrained(
    "JBrussee/gemma-4-31B-caveman",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

msgs = [{"role": "user", "content": "Explain database connection pooling."}]
ids = tok.apply_chat_template(msgs, return_tensors="pt", add_generation_prompt=True).to(model.device)
out = model.generate(ids, max_new_tokens=300, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Example

Prompt: "Why does my React component re-render every time the parent updates?"

Output:

Parent re-render → child re-render by default. Props "change" each render
if inline obj/array/fn → new ref → child sees new props.

Fix:
- Wrap child: React.memo(Child)
- Stabilize props: useMemo for obj/array, useCallback for fn
- Lift state so parent re-render not needed

Check: if child memo'd but still re-renders, inline prop is culprit.

Training

Method: QLoRA NF4 + double-quant + bf16 compute (Unsloth), then merged to bf16
LoRA: rank 16, α 32, dropout 0, targets q,k,v,o,gate,up,down
Loss: assistant-only loss masking via TRL completion_only_loss=True
Epochs: 3, lr 2e-4 cosine, batch 2 × grad-accum 8 (effective 16)
Hardware: RunPod RTX PRO 6000 Blackwell 96 GB
Time: ~50 minutes total

Data

1750 train + 193 eval pairs, source-normal → caveman style transfer, drawn from six permissive datasets:

Source	License	Used
OpenAssistant/oasst2	Apache 2.0	dialogue
princeton-nlp/SWE-bench_Verified	research-permissive	debug
ronantakizawa/github-codereview	permissive subset	code review
bigcode/commitpackft (MIT/Apache subset)	MIT/Apache 2.0	refactor
theblackcat102/evol-codealpaca-v1	Apache 2.0	short Q&A
HuggingFaceH4/ultrachat_200k	MIT	short Q&A

Caveman side synthesized via Claude Code (claude -p) and Codex CLI (codex exec with GPT-5.5), routed through the SKILL.md ruleset.

Eval (n=193 holdout)

Category	n	compression	article density	code_fence_match	semantic_sim
dialogue	28	0.59	0.020	1.000	0.91
debug	34	0.92	0.009	0.995	0.98
refactor	27	0.92	0.005	0.963	0.98
qa	104	0.65	0.007	1.000	0.92

Final train_loss 0.024 · eval_loss 0.72 · eval_mean_token_accuracy 0.815.

Strengths: code preservation (96-100% fence-exact), low article density (0.5-2%), strong semantic preservation (91-98%). Weakness: compression ~10-40% rather than the 50-70% of the gold caveman pairs. To compress harder, retrain on a tighter-filtered dataset (filter upper bound 0.7 instead of 1.0).

Reproduce

Full code, data pipeline, and configs: https://github.com/JuliusBrussee/finetune-caveman

License

Inherits the Gemma Prohibited Use Policy. Apache 2.0 base + Gemma terms apply to all outputs. Repository code is MIT. The caveman style ruleset is MIT (https://github.com/JuliusBrussee/caveman).

Citing

@misc{brussee2026cavemanGemma,
  author = {Julius Brussee},
  title  = {Caveman-mode Gemma 4 31B},
  year   = {2026},
  url    = {https://huggingface.co/JBrussee/gemma-4-31B-caveman}
}