Base model: Qwen/Qwen3-1.7B
Method: LoRA (r=32, α=64)
Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Dtype: float16

Training

Dataset: Jackrong/GLM-5.1-Reasoning-1M-Cleaned (75,000 examples)
Optimizer: AdamW 8-bit
Learning rate: 2e-4 with cosine scheduler
Batch size: 1 × 16 gradient accumulation (effective batch = 16)
Max sequence length: 4096
Packing: enabled

Notes

May HALLUCINATE

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-1.7B",
    torch_dtype=torch.float16,
    device_map="cuda",
)

tokenizer = AutoTokenizer.from_pretrained("CoNDeNse-AI/GLM-5.1-Qwen3-1.7B-CoNDeNse")
model = PeftModel.from_pretrained(base_model, "CoNDeNse-AI/GLM-5.1-Qwen3-1.7B-CoNDeNse")

prompt = "<|im_start|>user\nYour question here<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7, do_sample=True)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))