Qwen2.5-Coder-32B-Python-Specialist is an instruction-tuned version of the standard Qwen2.5-Coder-32B base model. This model has been specifically fine-tuned on a high-quality blend of Python and generalized coding instruction datasets to enhance its proficiency in formatting compliance, multi-turn coding problem solving, and Python-specific tasks.
Note: The model retains the original safety filters and alignment of the Qwen2.5 base model.
The model was fine-tuned using a distilled, high-quality combination of the CodeFeedback-Filtered-Instruction and python_code_instructions_18k_alpaca datasets, running over 20,000 highly diverse programming scenarios.
By aggressively targeting the Attention layers during fine-tuning (while leaving the complex MLP structures frozen), this model achieves state-of-the-art formatting compliance and instruction following without compromising the encyclopedic coding knowledge of the 32B base model.
Model Details
Base Model: unsloth/Qwen2.5-Coder-32B-Instruct-bnb-4bit
Parameters: 32 Billion
Context Length: Up to 32K (optimized at 512 for dense instruction tuning)
Training Strategy: LoRA (Attention Modules Only: q_proj, k_proj, v_proj, o_proj)
The model was fine-tuned utilizing Unsloth for rapid multi-processing data ingestion and memory-efficient LoRA scaling. The dataset consisted of heavily curated coding problems, heavily indexing on Python, converted into standard ShareGPT conversational format.
To enhance instruction following without catastrophic forgetting, we targeted only the Attention matrices. The model was trained with a learning rate of 2e-4, achieving a remarkably low final loss of 0.45 without overfitting.
