Part of our ongoing effort to understand how synthetic post-training affects a large foundation model's reasoning and structured output capabilities — and whether small, targeted SFT datasets can meaningfully shift performance on standard benchmarks.
1from transformers import AutoModelForCausalLM, AutoTokenizer
2from peft import PeftModel
3import torch
4
5base_id = "Qwen/Qwen2.5-32B-Instruct"
6adapter_id = "DuoNeural/qwen32b-all-datasets-sft"
7
8# Load 4-bit base (matches training setup)
9from transformers import BitsAndBytesConfig
10bnb_cfg = BitsAndBytesConfig(
11 load_in_4bit=True,
12 bnb_4bit_quant_type="nf4",
13 bnb_4bit_compute_dtype=torch.bfloat16,
14)
15
16tokenizer = AutoTokenizer.from_pretrained(base_id)
17base = AutoModelForCausalLM.from_pretrained(
18 base_id,
19 quantization_config=bnb_cfg,
20 device_map="auto",
21)
22
23# Load adapter — choose epoch
24model = PeftModel.from_pretrained(base, f"{adapter_id}/epoch_2", is_trainable=False)
25
26# Inference
27messages = [{"role": "user", "content": "Generate a JSON schema for a product catalog."}]
28text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
29inputs = tokenizer(text, return_tensors="pt").to(model.device)
30out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
31print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
Benchmarks (GSM8K, ARC-Challenge, HellaSwag) against the Qwen2.5-32B-Instruct base are in progress. Results will be added here once complete.
If SFT improves benchmark scores, we will release quantized versions (GGUF, GPTQ, AWQ, EXL2) for broader use.
