Model type: Causal language model with LoRA adapters
Base model: Qwen3-4B
Fine-tuning method: Supervised Fine-Tuning (SFT) with LoRA
Primary task: Mathematical problem solving and step-by-step reasoning
Language: Primarily English
Maximum sequence length during training: 2,048 tokens
Base model loading: 4-bit quantization
Trainable parameters: 33,030,144
Base model parameters: 4,055,498,240
Trainable parameter ratio: 0.81%
MathQwen3-4B is a parameter-efficient fine-tuning of Qwen3-4B for mathematical problem solving. The model was trained with supervised fine-tuning on a 20,000-example subset of MetaMathQA, using LoRA adapters applied to the attention and MLP projection layers.
Only approximately 0.81% of the base model's parameters were updated during fine-tuning, while the underlying Qwen3-4B weights remained frozen.
Important: This repository contains the LoRA adapter and tokenizer rather than a standalone copy of the full base model. The corresponding base model is required to use the adapter.
Training Details
Training Data
Training used the meta-math/MetaMathQA dataset.
A subset of 20,000 examples was selected using random seed 42. The selected examples were then split into:
Training: 18,000 examples
Evaluation: 2,000 examples
Train/evaluation split: 90% / 10%
The dataset was shuffled before subset selection, and the train/evaluation split was performed with the same fixed seed for reproducibility.
Each example was formatted using the Qwen chat template with a system instruction asking the model to solve the mathematical problem step by step and provide a final answer. The original question was used as the user message and the corresponding MetaMathQA response as the assistant message.
Training Procedure
The model was fine-tuned using supervised fine-tuning (SFT) with parameter-efficient Low-Rank Adaptation (LoRA).
LoRA adapters were applied to the following projection modules:
q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj
The base model was loaded in 4-bit quantization during training.
Model Parameters
Parameter
Value
Base model parameters
4,055,498,240
Trainable LoRA parameters
33,030,144
Trainable parameter ratio
0.81%
Quantized loading
4-bit
The relatively small number of trainable parameters makes this a parameter-efficient adaptation of Qwen3-4B.
Training was performed on a single NVIDIA L40S GPU. The L40S supports BF16, and the published training run therefore used BF16 precision.
Training Loss
The training run was monitored using Weights & Biases.
A training-loss curve can be included here to provide a visual overview of optimization and convergence:
Training loss
The training loss should be interpreted only as an optimization diagnostic. It does not directly measure mathematical reasoning accuracy or generalization.
Evaluation
Testing Data
The fine-tuning run reserved 2,000 examples from the selected MetaMathQA subset for evaluation.
In addition to the in-training evaluation set, independent benchmark evaluation is planned on GSM8K and potentially Hendrycks' MATH.
Evaluation Methodology
The planned evaluation will compare the original Qwen3-4B base model against MathQwen3-4B using the same evaluation procedure and datasets.
This comparison is intended to measure whether the mathematical fine-tuning improves mathematical problem-solving performance relative to the base model.
Metrics
The primary planned metric is mathematical answer accuracy on GSM8K and, if included, Hendrycks' MATH.
Additional metrics may be reported depending on the final evaluation setup.
Results
Benchmark results are not yet available and will be added after evaluation.
The planned comparison is:
Model
GSM8K
Hendrycks' MATH
Qwen3-4B
—
—
MathQwen3-4B
—
—
Results should be interpreted as a comparison between the original base model and the fine-tuned adapter under the same evaluation protocol.
Technical Specifications
Model Architecture and Objective
MathQwen3-4B is based on Qwen3-4B and uses Low-Rank Adaptation (LoRA) for parameter-efficient supervised fine-tuning.
The training objective is causal language modeling through supervised fine-tuning on formatted mathematical question-and-answer conversations.
The training data was formatted with a system instruction encouraging step-by-step mathematical problem solving and a final answer.
Environmental Impact
Training was performed using a single NVIDIA L40S GPU on Modal.
Hardware Type: NVIDIA L40S
GPU count: 1
Training precision: BF16
Training time: 1 hour 32 minutes 20 seconds
Cloud Provider: Modal
Compute Region: Not recorded
Carbon Emitted: Not calculated
The total training time corresponds to two training sessions of 35 minutes 19 seconds and 57 minutes 01 second, respectively.
Carbon emissions can be estimated using the Machine Learning Impact calculator described by Lacoste et al. (2019). An exact carbon estimate is not reported because the necessary information about the compute region and associated energy/carbon intensity was not recorded.
Technical Specifications
Compute Infrastructure
Training was performed using Modal with a single NVIDIA L40S GPU.
Hardware
GPU: NVIDIA L40S
Number of GPUs: 1
Training precision: BF16
Total training time: 1 hour 32 minutes 20 seconds
Maximum configured job duration: 6 hours
The training consisted of two sessions:
Session
Duration
Session 1
35 min 19 sec
Session 2
57 min 01 sec
Total
1 h 32 min 20 sec
Software
The training environment included:
Python 3.11
PyTorch
Hugging Face Transformers (version==4.54.0)
Hugging Face TRL (version==0.19.1)
Hugging Face PEFT (version==0.16.0)
Hugging Face Datasets (version==3.6.0)
Unsloth (cuda128-torch270, version==2025.7.8, and unsloth_zoo==2025.7.10)
Weights & Biases (version==0.28.1)
Modal (version==1.5.3)
Training Efficiency
The fine-tuning updated only 33,030,144 parameters, corresponding to 0.81% of the 4,055,498,240 parameters in the base Qwen3-4B model.
This parameter-efficient setup substantially reduces the number of parameters that need to be optimized compared with full-model fine-tuning.
Limitations
The model has not yet been comprehensively benchmarked against the base model. In particular, GSM8K and potentially Hendrycks' MATH evaluations are planned but are not yet reported.
Consequently, the current model card should not make claims that MathQwen3-4B is more accurate than Qwen3-4B on mathematical benchmarks until those evaluations have been completed.
The training loss and held-out SFT evaluation loss, if reported, should likewise not be interpreted as direct evidence of benchmark-level mathematical reasoning performance.
Additional Notes
This model card has been mostly generated by a LLM based on the finetuning informations.
