GGUF models with .gguf suffix can used with llama.cpp framework.
GGUF models with .powerinfer.gguf suffix are integrated with fused sparse FFN operators and sparse LM head operators. These models are only compatible to powerinfer framework.
SmallThinker is a family of on-device native Mixture-of-Experts (MoE) language models specially designed for local deployment,
co-developed by the IPADS and School of AI at Shanghai Jiao Tong University and Zenergize AI.
Designed from the ground up for resource-constrained environments,
SmallThinker brings powerful, private, and low-latency AI directly to your personal devices,
without relying on the cloud.
Performance
Note: The model is trained mainly on English.
Model
MMLU
GPQA-diamond
GSM8K
MATH-500
IFEVAL
LIVEBENCH
HUMANEVAL
Average
SmallThinker-4BA0.6B-Instruct
66.11
31.31
80.02
60.60
69.69
42.20
82.32
61.75
Qwen3-0.6B
43.31
26.77
62.85
45.6
58.41
23.1
31.71
41.67
Qwen3-1.7B
64.19
27.78
81.88
63.6
69.50
35.60
61.59
57.73
Gemma3nE2b-it
63.04
20.2
82.34
58.6
73.2
27.90
64.63
55.70
Llama-3.2-3B-Instruct
64.15
24.24
75.51
40
71.16
15.30
55.49
49.41
Llama-3.2-1B-Instruct
45.66
22.73
1.67
14.4
48.06
13.50
37.20
26.17
For the MMLU evaluation, we use a 0-shot CoT setting.
Note: i9 14900, 1+13 8ge4 use 4 threads, others use the number of threads that can achieve the maximum speed. All models here have been quantized to q4_0.
You can deploy SmallThinker with offloading support using PowerInfer
Model Card
Architecture
Mixture-of-Experts (MoE)
Total Parameters
4B
Activated Parameters
0.6B
Number of Layers
32
Attention Hidden Dimension
1536
MoE Hidden Dimension (per Expert)
768
Number of Attention Heads
12
Number of Experts
32
Selected Experts per Token
4
Vocabulary Size
151,936
Context Length
32K
Attention Mechanism
GQA
Activation Function
ReGLU
How to Run
Transformers
transformers==4.53.3 is required, we are actively working to support the latest version.
The following contains a code snippet illustrating how to use the model generate content based on given inputs.
python
1from transformers import AutoModelForCausalLM, AutoTokenizer
2import torch
34path ="PowerInfer/SmallThinker-4BA0.6B-Instruct"5device ="cuda"67tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)8model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16, device_map=device, trust_remote_code=True)910messages =[11{"role":"user","content":"Give me a short introduction to large language model."},12]13model_inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(device)1415model_outputs = model.generate(16 model_inputs,17 do_sample=True,18 max_new_tokens=102419)2021output_token_ids =[22 model_outputs[i][len(model_inputs[i]):]for i inrange(len(model_inputs))23]2425responses = tokenizer.batch_decode(output_token_ids, skip_special_tokens=True)[0]26print(responses)27
ModelScope
ModelScope adopts Python API similar to (though not entirely identical to) Transformers. For basic usage, simply modify the first line of the above code as follows:
from modelscope import AutoModelForCausalLM, AutoTokenizer
Statement
Due to the constraints of its model size and the limitations of its training data, its responses may contain factual inaccuracies, biases, or outdated information.
Users bear full responsibility for independently evaluating and verifying the accuracy and appropriateness of all generated content.
SmallThinker does not possess genuine comprehension or consciousness and cannot express personal opinions or value judgments.
