GPT-5-Distill-Qwen3-4B-Instruct-2507

• Base Model: Qwen/Qwen3-4B-Instruct-2507
• Parameters: 4B
• Training Method:
  • Supervised Fine-Tuning (SFT) on ShareGPT data
  • Knowledge distillation from LMSYS GPT-5 responses
• Supported Languages: Chinese, English, mixed inputs/outputs
• Max Context Length: Up to 32K tokens (max_seq_length = 32768)

2. Intended Use Cases

✅ Recommended:

• Casual chat in Chinese/English
• General knowledge explanations & reasoning guidance
• Code suggestions and simple debugging tips
• Writing assistance: editing, summarizing, rewriting
• Role-playing conversations (with well-designed prompts)

⚠️ Not Suitable For:

• High-risk decision-making:
  • Medical diagnosis, mental health support
  • Legal advice, financial investment recommendations
• Real-time factual tasks (e.g., news, stock updates)
• Authoritative judgment on sensitive topics

Note: Outputs are for reference only and not intended as the sole basis for critical decisions.

3. Training Data & Distillation Process

Key Datasets:

(1) ds1: ShareGPT-Qwen3 Instruction Dataset

• Source: Jackrong/ShareGPT-Qwen3-235B-A22B-Instuct-2507
• Purpose:
  • Provides diverse instruction-response pairs
  • Supports multi-turn dialogues and context awareness
• Processing:
  • Cleaned for quality and relevance
  • Standardized into instruction, input, output format

(2) ds2: LMSYS GPT-5 Teacher Response Data

• Source: ytz20/LMSYS-Chat-GPT-5-Chat-Response
• Filtering:
  • Only kept samples with flaw == "normal"
  • Removed hallucinations and inconsistent responses
• Purpose:
  • Distillation target for conversational quality
  • Enhances clarity, coherence, and fluency

Training Flow:

1. Prepare unified Chat-formatted dataset
2. Fine-tune base Qwen3-4B-Instruct-2507 via SFT
3. Conduct knowledge distillation using GPT-5's normal responses as teacher outputs
4. Balance style imitation with semantic fidelity to ensure robustness

⚖️ Note: This work is based on publicly available, non-sensitive datasets and uses them responsibly under fair use principles.

4. Key Features Summary

5. Acknowledgements

• LMSYS team for sharing GPT-5 response data
• Jackrong for the ShareGPT-Qwen3 dataset
• Qwen team for releasing Qwen3-4B-Instruct