This model is a pruned version of the Llama-3.2 architecture, with a parameter reduction of 40% in the MLP Layers.
The pruning process aims to enhance computational efficiency while maintaining acceptable performance across specific tasks.
This model is not intended to be used directly, but rather to be fine-tuned for specific tasks where it can achieve equal or superior performance compared to fine-tuning the base model for the same task.
Model Details
Model Type: Pruned version of LLaMA-1.2B using structured pruning
Original Model: meta-llama/Llama-3.2-1B
Pruning Method: Structured pruning of MLP layers using importance scores based on absolute maximum weights
Size Reduction: 26.3% (from 1.24B to 914M parameters)
Architecture: Same as original LLaMA but with reduced MLP layer sizes
These models are part of the study "Exploring GLU Expansion Ratios: Structured Pruning in Llama-3.2 Models". They explore structured pruning in GLU-based architectures using Llama-3.2 (1B and 3B variants). The pruning experiments target optimal expansion ratios to balance performance, computational efficiency, and environmental sustainability. The models were evaluated across multiple benchmarks, including BoolQ, ARC-Easy, and MUSR, and demonstrate significant efficiency gains while maintaining robust task performance.
Performance on Standard Benchmarks
Benchmark
Original Model
Pruned Model
Relative Change
ARC-Easy
65.19%
40.19%
-38.7%
BoolQ
64.16%
62.11%
-3.2%
LAMBADA-OpenAI
62.20%
29.85%
-52.0%
LAMBADA-Standard
53.46%
24.78%
-53.6%
Key Findings
Remarkably maintains strong performance on binary classification tasks (BoolQ)
Significant degradation on reasoning tasks (ARC-Easy)
Substantial impact on long-range comprehension (LAMBADA)
Notable increase in perplexity for language modeling tasks
Limitations
Considerable reduction in performance on complex language understanding tasks
Significant degradation in long-range dependency handling
May not be suitable for applications requiring high accuracy on language completion tasks
