NxMobileLM-1.5B-SFT is a fine-tuned version of the base model Qwen2.5-1.5B, optimized for mobile and edge applications. This model has been trained on proprietary instruction datasets curated to enhance performance in natural language understanding and generation tasks tailored to specific applications.
Key Features:
Base Model: Qwen2.5-1.5B
Parameter Count: 1.5 billion
Fine-tuning Objective: Supervised fine-tuning (SFT) on instruction datasets.
Specialization: Lightweight and efficient performance for mobile environments.
Multilingual Support: Designed to handle multiple languages effectively, enabling robust cross-lingual capabilities for diverse applications.
Model Details
Training Data
The model was fine-tuned using a proprietary dataset designed for diverse instruction-following tasks, including question answering, summarization, and dialogue. The dataset emphasizes:
Multi-domain generalization
Task-specific instruction understanding
Multilingual Coverage: Training data includes samples from several major languages to enhance cross-lingual understanding.
Training Configuration
Framework: PyTorch
Optimizer: AdamW
Learning Rate: 5e-5
Batch Size: 128
Epochs: 3
Mixed Precision: FP16
Evaluation
The model was evaluated on a variety of benchmarks, demonstrating:
High Accuracy: Achieves strong performance across general natural language tasks.
Efficiency: Optimized for low-latency inference on edge devices.
Multilingual Competence: Strong performance across multiple languages, making it suitable for global applications.
Performance Comparison
Open-LLM Leaderboard
On January 15, 2025, NxMobileLM-1.5B-SFT was ranked among the top 10 edge device models with fewer than 3 billion parameters and achieved the first rank for models with under 2 billion parameters, according to the OpenLLM leaderboard.
image/png
P-MMEval
To evaluate the multilingual capabilities of the model, We conducted evaluations on several benchmarks across three languages: English (en), Japanese (ja), and Vietnamese (vi). For detailed benchmark information, refer to P-MMEval.
Benchmark
Llama-3.2-1B-Instruct
SmolLM2-1.7B-Instruct
Qwen2.5-1.5B-Instruct
NxMobileLM-1.5B-SFT
mifeval-en
44.79
43.75
50
57.29
mifeval-ja
22.92
23.96
29.17
30.21
mifeval-vi
30.21
25
28.12
46.88
mmmlu-EN-US
35.25
42.5
45.5
45.25
mmmlu-JA-JP
31.5
26.25
36.00
41.00
mmmlu-VI-VT
22.75
22.25
39.00
38.00
xnli-en
35.83
35.83
59.17
66.67
xnli-ja
34.17
35.83
52.5
57.5
xnli-vi
37.5
34.17
45.83
55.83
Average
32.21
31.61
42.93
48.07
LightEval
The table below compares NxMobileLM-1.5B-SFT with other instruction-tuned models using various benchmarks. Results were obtained using the lighteval evaluation framework and are referenced from Hugging Face TB:
Metric
SmolLM2-1.7B-Instruct
Llama-1B-Instruct
Qwen2.5-1.5B-Instruct
SmolLM1-1.7B-Instruct
NxMobileLM-1.5B-SFT
IFEval (Average prompt/inst)
56.7
53.5
47.4
23.1
64.2
HellaSwag
66.1
56.1
60.9
55.5
63.57
ARC (Average)
51.7
41.6
46.2
43.7
45.21
PIQA
74.4
72.3
73.2
71.6
72.91
MMLU-Pro (MCF)
19.3
12.7
24.2
11.7
15.43
BBH (3-shot)
32.2
27.6
35.3
25.7
31.44
GSM8K (5-shot)
48.2
26.8
42.8
4.62
59.51
Average
49.8
41.5
47.1
33.7
50.3
Limitations
While NxMobileLM-1.5B-SFT excels in many areas, it may not perform well on tasks outside the scope of the fine-tuned dataset. Biases inherent in the training data may also affect outcomes.
Intended Use
NxMobileLM-1.5B-SFT is designed for use in:
Mobile virtual assistants
Real-time language-based applications
Compact edge AI solutions
Multilingual Scenarios: Supporting applications that require cross-lingual communication and understanding.
Misuse Warning: The model is not intended for use in generating harmful, biased, or illegal content.
How to Use
Here is a sample code snippet to load and use the model:
python
1from transformers import AutoModelForCausalLM, AutoTokenizer
23# Load the model and tokenizer4model_name ="NTQAI/NxMobileLM-1.5B-SFT"5model = AutoModelForCausalLM.from_pretrained(model_name)6tokenizer = AutoTokenizer.from_pretrained(model_name)78# Example usage9inputs = tokenizer("What is the capital of Vietnam?", return_tensors="pt")10outputs = model.generate(**inputs, max_length=50)11print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Citation
If you use this model in your research, please cite it as:
