Model Card for oopere/pruned60-llama-1b

Model Details

• Model Type: Pruned version of LLaMA-1.2B using structured pruning
• Original Model: meta-llama/Llama-3.2-1B
• Pruning Method: Structured pruning of MLP layers using importance scores based on absolute maximum weights
• Size Reduction: 39.3% (from 1.24B to 753M parameters)
• Architecture: Same as original LLaMA but with reduced MLP layer sizes
• Language(s): Same as original model
• License: Same as original model
• Developed by: Pere Martra

Performance on Standard Benchmarks

Key Findings

• Still maintains reasonable performance on binary classification tasks (BoolQ)
• Severe degradation on reasoning tasks (ARC-Easy)
• Critical impact on long-range comprehension (LAMBADA)
• Extreme increase in perplexity for language modeling tasks

Limitations

• Severe reduction in performance on complex language understanding tasks
• Critical degradation in long-range dependency handling
• Not suitable for language completion or generation tasks
• Only recommended for simple classification tasks where memory constraints are critical

Implementation Details

• Pruning Notebook: Detailed implementation and methodology
• GitHub Repository: LLM Course

Pruning Method

• Technique: Structured pruning targeting MLP layers
• Pruning Ratio: 60% of neurons removed from MLP layers
• Selection Criteria: Importance scoring based on absolute maximum weights
• Architecture Specifics: Maintained GLU structure during pruning

Hardware Requirements

Memory Requirements

• Base Model:
  • Parameters: ~2.48 GB (FP16)
  • Total Runtime Memory: ~3.1 GB
• Pruned Model (60%):
  • Parameters: ~1.51 GB (FP16)
  • Total Runtime Memory: ~1.9 GB
• Memory Reduction:
  • Parameter Memory: 39.3%
  • Total Runtime Memory: ~38.7%

Notes:

• Memory requirements assume FP16 precision
• Actual memory usage may vary depending on:
  • Batch size
  • Sequence length
  • Implementation details
  • Runtime environment

Minimum Requirements

• GPU Memory: 3GB for base model, 2GB for pruned model
• CPU Memory: 8GB recommended for both models

Acknowledgments

• Thanks to Mariusz Kurman for creating llama-pruning, a library that implements and extends this pruning methodology.
• This model was created following the pruning method described in the paper: The Width Pruning Dichotomy in Llama-3.2

@misc{martra2025fragileknowledgerobustinstructionfollowing,
      title={Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2}, 
      author={Pere Martra},
      year={2025},
      eprint={2512.22671},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2512.22671}, 
}