You can deploy this model and start earning money today!

ra225/
Llama-2-13b-best_ratio50_rmsnorm_finetuned

Name: Llama-2-13b-best_ratio50_rmsnorm_finetuned
Availability: InStock

Views

No views yet

Qwen2.5-7B Pruned + RMSNorm Finetuned (50% Parameters)

This model is a pruned and finetuned version of Qwen/Qwen2.5-7B, retaining approximately 50% of parameters while maintaining strong performance through genetic algorithm pruning and RMSNorm fine-tuning.

Model Details

Base Model: Qwen/Qwen2.5-7B
Parameter Retention: ~50%
Pruning Method: Genetic Algorithm
Fine-tuning Method: RMSNorm calibration

Performance

Metric	Value
PPL (Before Fine-tuning)	52.93
PPL (After Fine-tuning)	25.04
Improvement	52.68%

Performance Comparison

This 50% model is part of a family of pruned models with different compression ratios:

Model	PPL (After FT)
50% params	25.04
70% params	11.12
80% params	8.71
90% params	7.64

Usage

python

import torch
from transformers import AutoTokenizer

# Load tokenizer (standard Qwen2.5-7B tokenizer)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B")

# Note: This model uses a custom pruned architecture
# Custom loading code is required to use this model

Files Included

model_weights.pt: Full model state dict
README.md: This documentation

License

Apache 2.0 (inherited from base model)

Related Models

Qwen/Qwen2.5-7B - Base model
Other compression ratios: 50%, 70%, 80%, 90% versions available in this account

ra225/Llama-2-13b-best_ratio50_rmsnorm_finetuned