LoRA adapter that teaches google/gemma-4-E2B-it to emit explicit step-by-step reasoning, supervised-distilled from NovaSky-AI/Sky-T1_data_17k. The Sky-T1 corpus is ~17k reasoning traces generated by QwQ-32B-Preview, hand-curated by NovaSky-AI. Final eval loss: 0.4318.

Why

Prior to this release the confirmed Gemma 4 reasoning LoRAs on the hub were concentrated at 31B (e.g. kai-os/gemma4-31b-Opus-4.6-reasoning, distilled from a different source — Claude Opus traces). This set fills in the smaller sizes (E2B, E4B) with a QwQ-derived distillation, so the hot-swap story works on lighter hardware.

Training recipe

LoRA: rank 32, alpha 64, dropout 0.05, bias=none
Target modules: q/k/v/o/gate/up/down_proj — text tower only (vision + audio projections under Gemma4ClippableLinear are excluded so gradients flow to the layers that actually run during text inference)
Sequence length: 2048, no packing
Effective batch: 16 (micro-batch 8 × grad-accum 2 on E2B; 4 × 4 on E4B)
Optimizer: AdamW, lr 2e-4, cosine schedule with 3% warmup
Epochs: 2
Precision: bf16, gradient checkpointing with use_reentrant=False
Attention: SDPA (flash-attn 2 unavailable on ROCm for Gemma 4's head dim)
Hardware: 1× AMD MI300X (192 GB, ROCm 7.0)

Eval trajectory (held-out 2% split)

epoch	eval_loss
0.199	0.4901
0.398	0.4705
0.597	0.4585
0.796	0.4496
0.996	0.4431
1.194	0.4401
1.393	0.4359
1.592	0.4333
1.791	0.4321
1.991	0.4319
2.000	0.4318

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "google/gemma-4-E2B-it"
adapter_id = "josuediazflores/gemma-4-e2b-skyt1-reasoning-lora"

tokenizer = AutoTokenizer.from_pretrained(base_id)
model = AutoModelForCausalLM.from_pretrained(base_id, dtype="bfloat16")
model = PeftModel.from_pretrained(model, adapter_id)

messages = [{"role": "user", "content": "Prove there are infinitely many primes."}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
out = model.generate(inputs.to(model.device), max_new_tokens=1024)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Caveats

Trained on ~16k reasoning traces — useful for broad reasoning patterns, not a substitute for domain-specific math/code evals.
bf16 adapter; works with any bf16, 8-bit, or mxfp8 quant of the same base. 4-bit quants of the base will still load but some quality drift.
No RLHF step — this is pure supervised distillation from the Sky-T1 corpus, which was itself Qwen-distilled from o1/R1-style outputs.

Citation

@dataset{sky_t1_2025,
  author = {NovaSky-AI},
  title = {Sky-T1 Reasoning Dataset},
  year = {2025},
  url = {https://huggingface.co/datasets/NovaSky-AI/Sky-T1_data_17k}
}