Qwen3-4B-Instruct-2507 for RK3588 (RKLLM 1.2.1)

Available Models

Model Selection

Useful links:

Original Model Card for base model, Qwen3-4B-Instruct-2507, below:

Qwen3-4B-Instruct-2507

Highlights

• Significant improvements in general capabilities, including instruction following, logical reasoning, text comprehension, mathematics, science, coding and tool usage.
• Substantial gains in long-tail knowledge coverage across multiple languages.
• Markedly better alignment with user preferences in subjective and open-ended tasks, enabling more helpful responses and higher-quality text generation.
• Enhanced capabilities in 256K long-context understanding.

Model Overview

• Type: Causal Language Models
• Training Stage: Pretraining & Post-training
• Number of Parameters: 4.0B
• Number of Paramaters (Non-Embedding): 3.6B
• Number of Layers: 36
• Number of Attention Heads (GQA): 32 for Q and 8 for KV
• Context Length: 262,144 natively.

Performance

Quickstart

• SGLang:
  python -m sglang.launch_server --model-path Qwen/Qwen3-4B-Instruct-2507 --context-length 262144
• vLLM:
  vllm serve Qwen/Qwen3-4B-Instruct-2507 --max-model-len 262144

Agentic Use

Best Practices

1. Sampling Parameters:
   • We suggest using Temperature=0.7, TopP=0.8, TopK=20, and MinP=0.
   • For supported frameworks, you can adjust the presence_penalty parameter between 0 and 2 to reduce endless repetitions. However, using a higher value may occasionally result in language mixing and a slight decrease in model performance.
2. Adequate Output Length: We recommend using an output length of 16,384 tokens for most queries, which is adequate for instruct models.
3. Standardize Output Format: We recommend using prompts to standardize model outputs when benchmarking.
   • Math Problems: Include "Please reason step by step, and put your final answer within \boxed{}." in the prompt.
   • Multiple-Choice Questions: Add the following JSON structure to the prompt to standardize responses: "Please show your choice in the answer field with only the choice letter, e.g., "answer": "C"."

Citation

@misc{qwen3technicalreport,
      title={Qwen3 Technical Report}, 
      author={Qwen Team},
      year={2025},
      eprint={2505.09388},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.09388}, 
}