QLoRA fine-tune of google/gemma-3-4b-it trained to imitate the structured reasoning style of Claude Opus 4.6. Quantized to GGUF Q4_K_M for efficient local inference.

The model produces explicit <think>...</think> reasoning blocks before answers, distilled from Opus traces in Crownelius/Opus-4.6-Reasoning-3300x.

What This Is (And What It Is Not)

Important context on training method: This is a QLoRA fine-tune, not a full-model fine-tune. Specifically:

The base Gemma 3 4B weights were loaded in 4-bit NF4 and kept frozen
Only a small set of LoRA adapters (~33M parameters, 0.76% of total) were trained
These adapters were then merged back into the base model in FP16
The merged FP16 model was quantized to GGUF Q4_K_M for distribution

This means the model structurally learned to reason like Opus, but it has not absorbed Opus-level knowledge or capability. Claude Opus 4.6 is roughly 1000x larger than this model. Don't expect Opus performance — expect a small Gemma that thinks step-by-step in a useful structured format.

Files

gemma-3-4b-it.Q4_K_M.gguf — Main model (2.4 GB)
gemma-3-4b-it.BF16-mmproj.gguf — Vision projector for multimodal use (812 MB)
Modelfile — Ollama configuration

Training Metrics

Metric	Value
Starting loss (step 1)	2.67
Loss after warmup (step 5)	1.62
Final training loss	1.01
Total training steps	471
Effective batch size	4 (1 × 4 gradient accumulation)
Trainable parameters	32,788,480 / 4,332,867,952 (0.76%)
Training duration	50.6 minutes
Peak VRAM	7.54 GB / 8.0 GB

Loss dropped from 2.67 to 1.01 over a single epoch, indicating the LoRA adapters successfully fit the reasoning pattern.

Capabilities

Structured reasoning with <think>...</think> blocks (Opus-style)
Math problem solving (94% of training data was math)
Vision understanding (multimodal capability preserved from base model — the LoRA only modifies the language portion)
128K context window
140+ languages (from base model)

Reproducing This Training

The full pipeline used to create this model:

1. Environment

Hardware: NVIDIA RTX 3050 8GB + 48GB system RAM
Container: unsloth/unsloth Docker image
Stack: PyTorch 2.10 + CUDA 12.8 + Unsloth 2026.4.8 + Transformers 4.57.6

2. Dataset Filtering

The raw dataset (2,160 rows) was filtered to remove low-quality entries:

python

Used Unsloth's `model.save_pretrained_gguf()` which orchestrates all three steps automatically.

## Usage

### Ollama

```bash
ollama create gemma3-opus-reasoning -f Modelfile
ollama run gemma3-opus-reasoning "Solve: 2x^2 + 5x - 12 = 0"

llama.cpp (text-only)

bash

./llama-cli -m gemma-3-4b-it.Q4_K_M.gguf \
    -p "What is 240 km in 3 hours expressed in m/s?" \
    -n 512 -ngl 999

llama.cpp (with vision)

bash

./llama-mtmd-cli \
    -m gemma-3-4b-it.Q4_K_M.gguf \
    --mmproj gemma-3-4b-it.BF16-mmproj.gguf \
    -p "Describe this image" --image path/to/image.jpg

Example Output

Prompt: If a train travels 240 km in 3 hours, what is its average speed in m/s? Inference speed on RTX 3050: ~98 tok/s prompt processing, ~14 tok/s generation.

Benchmark Results

Evaluated on the full MATH-500 test set (500 competition math problems) using lm-evaluation-harness 0.4.11, task minerva_math500. Both models evaluated under identical conditions: 4-shot in-context examples, batch size 1, 4-bit quantization (load_in_4bit=True), bfloat16 compute dtype, deterministic generation (do_sample=False, temperature=0.0).

Accuracy

Metric	Base Gemma 3 4B	This Fine-tune	Δ
`exact_match`	29.6% ± 2.0%	24.6% ± 1.9%	−5.0pp
`math_verify`	36.4% ± 2.2%	29.6% ± 2.0%	−6.8pp

Standard errors computed via lm-eval-harness bootstrap (100,000 iterations).

Performance (RTX 3050 8GB, single-card)

Metric	Base	Fine-tune	Notes
Total wall-clock	137.7 min	138.0 min	500 problems each
Per-problem time	16.5 s	16.6 s	Identical inference cost
Throughput	3.6 problems/min	3.6 problems/min
Inference speed (Q4_K_M GGUF)	—	~14 tok/s	Measured via `llama-cli`, prompt eval ~98 tok/s
Peak VRAM during eval	~6.2 GB	~6.2 GB	Measured via `nvidia-smi`

Honest Interpretation

The fine-tune underperforms the base on MATH-500. This result is statistically significant (confidence intervals do not overlap) and is reported transparently rather than hidden. Possible explanations, in order of likelihood:

Evaluation format mismatch (most likely). The model was trained on conversational examples with <think>...</think> reasoning blocks. MATH-500 in lm-eval-harness uses 4-shot completion-style prompts that don't include this format. The model's trained instinct to produce thinking blocks may suppress the format the answer extractor expects, causing it to generate correct reasoning that fails extraction.
Catastrophic forgetting on a narrow dataset. Fifty minutes of focused training on math reasoning traces (~1,900 examples after filtering) may have partially overwritten Gemma 3's broader competence — including the format following needed for in-context evaluation.
Distillation from a much larger model. Distilling from Claude Opus 4.6 (~1000× larger) into a 4B model with limited examples is known to produce stylistic improvements without commensurate capability transfer.

The qualitative output (see Example Output above) demonstrates that structured reasoning was learned: the model produces clean <think> blocks, step-by-step decomposition, and self-verification. The trained reasoning style is real even though it does not improve standard benchmark scores.

This is a known pattern in reasoning-distilled small models. Reporting it openly contributes more value than burying it.

Reproducing These Results

bash

# Both runs used identical commands except for the model name:
lm_eval --model hf \
    --model_args "pretrained=<MODEL_ID>,dtype=bfloat16,load_in_4bit=True,trust_remote_code=True" \
    --tasks minerva_math500 \
    --device cuda \
    --batch_size 1 \
    --output_path ./eval_results \
    --log_samples

Where <MODEL_ID> is either Viesar/gemma-3-4b-opus-reasoning-distill or google/gemma-3-4b-it.

Benchmark Results

Evaluated on two standard math reasoning benchmarks using lm-evaluation-harness 0.4.11. Both models evaluated under identical conditions: 4-bit quantization (load_in_4bit=True), bfloat16 compute dtype, batch size 1, deterministic generation (do_sample=False, temperature=0.0).

MATH-500 (full test set, 500 problems, 4-shot)

Metric	Base Gemma 3 4B	This Fine-tune	Δ
`exact_match`	29.6% ± 2.0%	24.6% ± 1.9%	−5.0pp
`math_verify`	36.4% ± 2.2%	29.6% ± 2.0%	−6.8pp

GSM8K (300-problem subset, 5-shot)

Metric	Base Gemma 3 4B	This Fine-tune	Δ
`exact_match` (strict-match)	68.7% ± 2.7%	53.7% ± 2.9%	−15.0pp
`exact_match` (flexible)	69.0% ± 2.7%	55.0% ± 2.9%	−14.0pp

Standard errors computed via lm-eval-harness bootstrap (100,000 iterations). Confidence intervals do not overlap on either benchmark — regressions are statistically significant.

Performance (RTX 3050 8GB, 4-bit)

Metric	Base	Fine-tune
Per-problem time (MATH-500)	16.5 s	16.6 s
Per-problem time (GSM8K)	11.0 s	~11 s
Inference speed (Q4_K_M GGUF)	—	~14 tok/s (gen), ~98 tok/s (prompt eval)
Peak VRAM	~6.2 GB	~6.2 GB

The fine-tune has identical inference cost to the base. Time differences are negligible.

Honest Interpretation

The fine-tune underperforms the base on both benchmarks in standard few-shot completion-style evaluation. We report this transparently rather than hide it. Two effects are likely contributing:

1. Evaluation format mismatch

Both MATH-500 and GSM8K evaluate via few-shot completion prompts:

MATH-500: Problem: ... Solution: ... (4-shot)
GSM8K: Question: ... Answer: ... (5-shot)

Neither anticipates the chat template or <think>...</think> reasoning blocks the model was trained on. When the model's trained instinct (produce thinking blocks first) collides with the in-context format (direct answer), it can fail the regex-based answer extraction even when the underlying reasoning reaches the correct answer.

This effect is well-documented: reasoning-distilled small models tend to perform better on chat-style evaluation and worse on completion-style benchmarks.

2. Capability narrowing from focused training

The GSM8K regression (≈15pp) is larger than MATH-500's (≈6pp). This suggests that beyond format mismatch, training on a narrow distribution (~1,900 reasoning traces, 94% math, single epoch) likely caused some capability narrowing. The model became better at producing structured reasoning in its trained format, at the cost of flexibility on completion-style prompts.

What's still true

The trained <think> reasoning style is real and visible in qualitative outputs (see Example Output).
Inference cost per token is unchanged from the base.
The model is more useful for chat-style reasoning tasks than these benchmarks suggest.

This is a legitimate finding about resource-constrained distillation. Reporting it honestly contributes more value than burying it.

Reproducing These Results

bash

# Common pattern (MODEL_ID is either model)
lm_eval --model hf \
    --model_args "pretrained=<MODEL_ID>,dtype=bfloat16,load_in_4bit=True,trust_remote_code=True" \
    --tasks <TASK> \
    --device cuda \
    --batch_size 1 \
    --output_path ./eval_results \
    --log_samples

For MATH-500 use --tasks minerva_math500 (no --limit). For GSM8K use --tasks gsm8k --limit 300.

<MODEL_ID> is either Viesar/gemma-3-4b-opus-reasoning-distill or google/gemma-3-4b-it.

Limitations

Heavily skewed toward math (94% of training data) — performance on non-math reasoning will be weaker
Modest dataset size (~1,900 filtered examples after quality filtering) — single epoch
4-bit quantization causes minor quality degradation vs the FP16 merged model
QLoRA, not full fine-tune — base model knowledge unchanged, only reasoning structure adjusted
Will not match Claude Opus on capability — Opus is ~1000x larger
May hallucinate on obscure factual questions, like all small models
Vision encoder is unchanged from the base model (LoRA only modified the language portion)

Acknowledgments

Base model by Google
Training framework by Unsloth
Reasoning traces dataset by Crownelius
Trained on consumer hardware (RTX 3050 8GB) in 50 minutes

License

Released under the Gemma License. By using this model, you agree to comply with Google's Gemma usage terms.

Viesar/gemma-3-4b-opus-reasoning-distill-GGUF

Gemma 3 4B — Opus 4.6 Reasoning Distillation (GGUF)

What This Is (And What It Is Not)

Files

Training Metrics

Capabilities

Reproducing This Training

1. Environment

2. Dataset Filtering

llama.cpp (text-only)

llama.cpp (with vision)

Example Output

Benchmark Results

Accuracy

Performance (RTX 3050 8GB, single-card)

Honest Interpretation

Reproducing These Results

Benchmark Results

MATH-500 (full test set, 500 problems, 4-shot)

GSM8K (300-problem subset, 5-shot)

Performance (RTX 3050 8GB, 4-bit)

Honest Interpretation

1. Evaluation format mismatch

2. Capability narrowing from focused training

What's still true

Reproducing These Results

Limitations

Acknowledgments

License

Viesar/
gemma-3-4b-opus-reasoning-distill-GGUF