A 4B parameter model fine-tuned on ~308k mixed samples covering general chat, instruction following, math, coding, and factual QA. This is Stage 1 of a multi-stage training pipeline built on top of Qwen3-4B.

Model Details

Base Model: unsloth/Qwen3-4B
Parameters: 4B
Architecture: Qwen3 (dense, pure text)
Training: LoRA fine-tune → merged
License: Apache 2.0
Language: English (multilingual base)

Files

File	Description
`adapter_model.safetensors`	LoRA adapter weights (unmerged)
`adapter_config.json`	LoRA configuration
`qwen4b-s1-Q4_K_M.gguf`	Quantized GGUF (Q4_K_M, ~2.4GB)
`tokenizer.json`	Tokenizer

Training Details

Dataset Mix (~308k samples)

Dataset	Samples	Purpose
open-thoughts/OpenThoughts-114k	113,957	Reasoning / Chain-of-thought
teknium/OpenHermes-2.5	30,000	General chat
HuggingFaceH4/ultrachat_200k	30,000	Conversation
WizardLM/WizardLM_evol_instruct_V2_196k	20,000	Instruction following
nvidia/OpenMathInstruct-2	50,000	Math
AI-MO/NuminaMath-CoT	20,000	Math competition
hotpot_qa	10,000	Multi-hop QA
squad_v2	8,000	Factual QA
trivia_qa	5,000	Factual QA
google-research-datasets/natural_questions	5,000	Factual QA
google-research-datasets/mbpp	374	Coding
iamtarun/python_code_instructions_18k_alpaca	5,000	Coding
bigcode/self-oss-instruct-sc2-exec-filter-50k	10,000	Coding

Total: ~308k samples, 2 epochs

Hyperparameters

Parameter	Value
LoRA rank	64
LoRA alpha	64
Learning rate	1e-4 cosine
Epochs	2
Sequence length	4096
Batch size	4 (effective 32)
Optimizer	adamw_8bit
Final loss	0.4861

Hardware

Trained on AMD Instinct MI300X (192GB VRAM), ROCm 6.2.4, Unsloth 2026.3.3, PyTorch 2.7.1+rocm6.2.4. Runtime: ~16.3 hours.

Usage

Ollama (GGUF)

ollama run hf.co/lqfdjbf32n/Qwen3-4B-Stage1:Q4_K_M

llama.cpp

bash

llama-cli -m qwen4b-s1-Q4_K_M.gguf \
    -p "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nYour question here<|im_end|>\n<|im_start|>assistant\n" \
    -n 512

Python (LoRA adapter)

python

from unsloth import FastLanguageModel
from peft import PeftModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    "unsloth/Qwen3-4B",
    max_seq_length=4096,
    dtype=torch.bfloat16,
    load_in_4bit=False,
)
model = PeftModel.from_pretrained(model, "lqfdjbf32n/Qwen3-4B-Stage1")
model = model.merge_and_unload()

Limitations

Stage 1 only — general foundation, not reasoning-specialized
English primary (multilingual via base model)
Not suitable for production without validation
Complex multi-step reasoning may still fail

Part of a Series

Model	Description
Qwen2.5-0.5B-ReasonChat	0.5B edge model, reasoning + chat merged
Qwen3-4B-Stage1	4B general foundation (this model)
Qwen3-4B-Stage2	4B + Claude reasoning injection (coming soon)