This model is a supervised fine-tuned variant of Qwen3-1.7B-Base. The original model is a pretrained language model that is not instruction-tuned for conversational tasks. To enable instruction-following and dialogue capabilities, it was fine-tuned on the UltraChat-200k dataset, resulting in a helpful conversational assistant while retaining the base model's general language understanding and knowledge. This model does not support thinking, vision or tool calling yet.

Fine-tuning was performed using LoRA adapters, with additional training applied to the token embedding layer (embed_tokens) and the language modeling head (lm_head). These components were optimized using a lower learning rate than the LoRA parameters, allowing for gradual adaptation of the vocabulary representations and output distribution. This was particularly beneficial for the special tokens, whose embedding vectors in the base model were initially identical, enabling them to learn meaningful and distinct representations during fine-tuning.

View the better direct preference optimized version of this model: Qwen3-1.7B Chat
View the GitHub repo for Post-training pipeline.

Model Overview

Qwen3-1.7B-UltraChat-SFT has following features:

Type: Causal Language Models
Training Stage: Pretraining & Post-training
Number of Parameters 1.7B
Number of Paramaters (Non-Embedding): 1.4B
Number of Layers: 28
Number of Attention Heads (GQA): 16 for Q and 8 for KV
Context Length: 32,768

Training

Method: Supervised Fine-Tuning (SFT)
Framework: Unsloth
Fine-Tuning: LoRA
Precision: BF16
Environment: Windows Subsystem for Linux (WSL)
GPU: RTX 4090 24 GB, using FlashAttention 2, xFormers and Triton
Context Length: 2048 tokens
Dataset: UltraChat 200k. Conversations longer than the context window were trimmed before training.

Training Configuration

Parameter	Value
Epochs	1
Batch Size	4
Gradient Accumulation	8
Effective Batch Size	32
Optimizer	AdamW Fused
Learning rate	2.5e-4
Embedding Learning Rate	5e-5
LoRA Rank	32
LoRA Alpha	64
Scheduler	Cosine
Warmup	3%
Gradient Checkpointing	False
Packing	True

LoRA is applied tp q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj and down_proj.

Evaluation

The evaluation was performed using EleutherAI's Language Model Evaluation Harness.

Benchmark	Score
MMLU-Pro	37.91
GSM8K	71.27
HellaSwag	62.28
TruthfulQA _mc2	51.62
ARC Challenge	44.97
IFEval _{strict prompt}	33.09

Usage

You can either use Unsloth or transformers.

Unsloth (Recommended)

Install Unsloth following these docs. Install Flash Attention if possible. Check if Triton and xFormers are installed.

python

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "ayushshah/Qwen3-1.7B-UltraChat-SFT",
    max_seq_length = 2048,
    dtype = "bfloat16",
    load_in_4bit = False,
)

FastLanguageModel.for_inference(model)

prompt = "Explain gradient descent simply."
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": prompt}
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize = True,
    add_generation_prompt = True,
    return_tensors = "pt",
).to(model.device)

outputs = model.generate(
    input_ids = inputs,
    max_new_tokens = 2048,
)

input_length = inputs.shape[1]
new_tokens = outputs[0][input_length:]

raw_text = tokenizer.decode(new_tokens, skip_special_tokens=True)
print(raw_text)

transformers

python

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("ayushshah/Qwen3-1.7B-UltraChat-SFT")
model = AutoModelForCausalLM.from_pretrained(
    "ayushshah/Qwen3-1.7B-UltraChat-SFT",
    torch_dtype="auto",
    device_map="auto"
)

prompt = "Explain gradient descent simply."
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": prompt}
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize = True,
    add_generation_prompt = True,
    return_tensors = "pt",
).to(model.device)

outputs = model.generate(
    input_ids = inputs["input_ids"],
    max_new_tokens = 2048,
)

input_length = inputs["input_ids"].shape[1]
new_tokens = outputs[0][input_length:]

raw_text = tokenizer.decode(new_tokens, skip_special_tokens=True)
print(raw_text)

Limitations

This model provides the base model with conversational and instruction-following capabilities through supervised fine-tuning. Responses are generally concise and direct, making it a strong foundation for further preference alignment (e.g., DPO, RLHF, PPO) or domain-specific fine-tuning. While suitable for general instruction-following, responses may occasionally lack the depth, natural conversational flow, or helpfulness expected from preference-aligned chat models.

Citation

@misc{qwen3technicalreport,
      title={Qwen3 Technical Report}, 
      author={Qwen Team},
      year={2025},
      eprint={2505.09388},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.09388}, 
}