You can deploy this model and start earning money today!

ra225/
Qwen2.5-14B-pruned-90p-rmsnorm-finetuned

Name: Qwen2.5-14B-pruned-90p-rmsnorm-finetuned
Availability: InStock

Views

No views yet

Qwen2.5-14B Pruned + RMSNorm Finetuned (90% Parameters)

This model is a pruned and finetuned version of Qwen/Qwen2.5-14B, retaining approximately 90% of parameters while maintaining strong performance through genetic algorithm pruning and RMSNorm fine-tuning.

Model Details

Base Model: Qwen/Qwen2.5-14B
Parameter Retention: ~90%
Pruning Method: Genetic Algorithm
Fine-tuning Method: RMSNorm calibration

Performance

Metric	Value
PPL (Before Fine-tuning)	7.01
PPL (After Fine-tuning)	6.15
Improvement	12.26%

Performance Comparison

This 90% model is part of a family of pruned models with different compression ratios:

Model	PPL (After FT)
50% params	16.94
70% params	9.60
80% params	7.11
90% params	6.15

Usage

python

import torch
from transformers import AutoTokenizer

# Load tokenizer (standard Qwen2.5-14B tokenizer)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B")

# Note: This model uses a custom pruned architecture
# Custom loading code is required to use this model

Files Included

model_weights.pt: Full model state dict
README.md: This documentation

License

Apache 2.0 (inherited from base model)

Related Models

Qwen/Qwen2.5-14B - Base model
Other compression ratios: 50%, 70%, 80%, 90% versions available in this account

ra225/Qwen2.5-14B-pruned-90p-rmsnorm-finetuned