This model applies mixed-precision quantization directly on a 30% Taylor-pruned Llama 3 8B model.
Base Model: meta-llama/Meta-Llama-3-8B
Pruning: 30% Taylor pruning
Quantization: Mixed precision via llama.cpp --tensor-type
This variant explores mixed-precision quantization without outlier suppression, serving as an ablation baseline in a structured LLM compression pipeline.
Model Details
Metric
Value
Average Bitwidth
~5–6 bpw
File Size
~4.0 GB
Compression vs FP16
~73%
Mixed-Precision Strategy
Critical tensors retained at higher precision
Less sensitive tensors aggressively quantized
No activation or weight smoothing applied
Method
Start from pruned FP16 model
Convert to GGUF
Generate importance matrix (imatrix) for sensitivity estimation
Apply mixed-precision quantization via tensor-type rules
Usage
bash
1./llama-cli -m llama-3.1-8b-pruned-taylor30-padded-mixed-precision-quantization.gguf -p "Hello, I am"2
