A QLoRA fine-tune of Qwen/Qwen3-0.6B, trained on a mix of math reasoning, chain-of-thought, and general English data. Optimized with Muon rather than AdamW.

This is a LoRA adapter, not a standalone model. You need the base model (Qwen/Qwen3-0.6B) to use it — see the loading instructions below.

Training data

Trained on a mix of three datasets, each mapped into Qwen3's chat format (user question → assistant response with a CoT-style explanation and a final answer):

Dataset	Rows used	Purpose
Gugu8/Math-Dataset	25,000	Math word problems + reasoning
Gugu8/CoT-for-LLM	25,000	General chain-of-thought reasoning
Gugu8/English	12,000	General English fluency (sentences, grammar, narrative)

Training setup

Base model: Qwen/Qwen3-0.6B (606M params)
Method: QLoRA — 4-bit NF4 quantization (double quant, bf16 compute dtype) + LoRA adapters
LoRA config: rank 16, alpha 32, dropout 0.05, applied to all attention and MLP projections (q/k/v/o_proj, gate/up/down_proj)
Trainable params: 10,092,544 (~1.67% of the 606M total)
Optimizer: Muon for all ≥2D LoRA matrices (392 tensors), with a small AdamW fallback group for any 1D params. Muon LR 2e-3, momentum 0.95; AdamW fallback LR 2e-4.
Sequence length: 1024 tokens, packed
Effective batch size: 16 (2× GPUs × batch 2 × grad accumulation 4)
Epochs: 1 (415 steps)
Hardware: 2× Kaggle T4 GPUs, launched via accelerate launch (DDP)
Time ~3 Hours
Final training loss: ~0.156 (down from ~1.32 at step 20)

How to use

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model_id = "Qwen/Qwen3-0.6B"
adapter_id = "Gugu8/<your-repo-name>"  # replace with your actual repo path

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_id)

messages = [{"role": "user", "content": "What is 24% of 350?"}]
inputs = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

with torch.no_grad():
    out = model.generate(inputs, max_new_tokens=200, do_sample=False)

print(tokenizer.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Example output:

Calculate 24/100 * 350 = 0.24*350 = 84. Answer: 84

Notes and limitations

This is a small (0.6B parameter base) model fine-tuned with a lightweight LoRA adapter (rank 16) on a relatively modest amount of data (62K examples, 1 epoch). It's a proof-of-concept / experimental fine-tune, not a production-grade reasoning model.
The Muon optimizer's default learning rate (0.02, tuned for training large weight matrices from scratch) is far too aggressive for LoRA-scale fine-tuning and will diverge — this run used 2e-3 instead. If you fine-tune further from this adapter, keep that in mind.
Training data caps (25K/25K/12K rows) were chosen for a practical training time on 2× T4 GPUs (~2.5 hours), not to exhaust the source datasets.

License

Released under OpenMDW-1.1, a permissive license from the Linux Foundation designed specifically for AI model distributions (covers weights, code, and documentation as a unified package). See the LICENSE file in this repository for the full text.

Gugu8/Molecule-1

Molocule-1 — Math / CoT / English LoRA Adapter

Training data

Training setup

How to use

Notes and limitations

License

Gugu8/
Molecule-1