Qwen3-8B-LaCo-Pruned
This model is a
layer-pruned version of
Qwen3-8B-Base using the
LaCo (Layer Collapse) structured pruning method.
Model Summary
| Attribute | Value |
|---|
| Base Model | Qwen/Qwen3-8B-Base |
| Pruning Method | LaCo (Layer Collapse) |
| Original Layers | 36 |
| Pruned Layers | 30 |
| Layers Removed | 6 |
| Compression | 16.7% |
Key Results
This model achieves 16.7% compression while retaining:
- ~90% of physical reasoning (PIQA)
- ~94% of commonsense reasoning (WinoGrande)
- ~79% of common sense completion (HellaSwag)
- ~41% of factual knowledge (MMLU)
This is a raw pruned model without post-training. Fine-tuning can further recover lost capabilities.
Benchmark Results (Pre-Training)
Note: All benchmarks below are evaluated on the pruned model without any post-training or fine-tuning. These results represent the raw performance after pruning only. Post-training is expected to improve these scores, particularly on knowledge-intensive tasks like MMLU.
Comparison with Original Qwen3-8B-Base
| Benchmark | Original | Pruned | Retention |
|---|
| PIQA (acc_norm) | 79.54% | 71.38% | 89.7% |
| WinoGrande | 67.0% | 62.83% | 93.8% |
| ARC-Challenge (acc_norm) | 42.0% | 36.09% | 85.9% |
| ARC-Easy (acc_norm) | 72.0% | 58.04% | 80.6% |
| HellaSwag (acc_norm) | 78.55% | 61.98% | 78.9% |
| BoolQ | 83.09% | 64.95% | 78.2% |
| MMLU (5-shot) | 76.89% | 31.30% | 40.7% |
Benchmark Interpretation
| Capability | Benchmarks | Retention | Status |
|---|
| Physical Reasoning | PIQA | 89.7% | Excellent |
| Commonsense Reasoning | WinoGrande | 93.8% | Excellent |
| Basic Reasoning | ARC-Challenge | 85.9% | Good |
| Reading Comprehension | BoolQ | 78.2% | Good |
| Common Sense | HellaSwag | 78.9% | Good |
| Factual Knowledge | MMLU | 40.7% | Degraded |
The "Knowledge Cliff"
Our experiments reveal a critical finding: factual knowledge collapses catastrophically between 16-22% compression.
| Compression | Layers | MMLU | Status |
|---|
| 16.7% | 30 | 31.30% | Partial retention |
| 22.2% | 28 | 25.89% | Random chance |
| 27.8% | 26 | 25.12% | Random chance |
While reasoning capabilities degrade gradually with compression, factual knowledge encoded in specific layers is lost abruptly when those layers are removed.
Intended Use
This model is suitable for:
- Research on model compression and efficiency
- Fine-tuning base for domain-specific applications
- Inference optimization where speed/memory matters
- Applications prioritizing reasoning over factual recall
Limitations
Important: This is a raw pruned model without post-training.
| Use Case | Recommendation |
|---|
| Physical/commonsense reasoning | Recommended |
| Reading comprehension | Recommended |
| General text understanding | Recommended |
| Factual question answering | Fine-tune first |
| Knowledge-intensive tasks | Fine-tune first |
Pruning Details
LaCo Hyperparameters
| Parameter | Value | Description |
|---|
| MERGE_LAYERS (C) | 3 | Layers merged per operation |
| LOWEST_LAY (L) | 4 | Minimum layer index for merging |
| HIGHEST_LAY (H) | 28 | Maximum layer index for merging |
| INTERVAL (I) | 2 | Minimum gap between merge points |
| THRESHOLD (T) | 0.85 | Cosine similarity threshold |
| MAX_COMPRESSION | 20% | Maximum allowed compression |
Pruning Statistics
| Metric | Value |
|---|
| Successful Merges | 3 |
| Rejected Merges | 0 |
| Total Iterations | 4 |
| Final Compression | 16.7% |
Usage
Basic Inference
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model_name = "Mercity/Qwen3-8B-LaCo-Pruned"
4
5tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
6model = AutoModelForCausalLM.from_pretrained(
7 model_name,
8 torch_dtype="auto",
9 device_map="auto",
10 trust_remote_code=True
11)
12
13# Text generation
14prompt = "The process of photosynthesis"
15inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
16outputs = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.7)
17print(tokenizer.decode(outputs[0], skip_special_tokens=True))
With 4-bit Quantization (Further Compression)
1from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
2
3quantization_config = BitsAndBytesConfig(
4 load_in_4bit=True,
5 bnb_4bit_compute_dtype="float16",
6 bnb_4bit_quant_type="nf4",
7)
8
9model = AutoModelForCausalLM.from_pretrained(
10 "Mercity/Qwen3-8B-LaCo-Pruned",
11 quantization_config=quantization_config,
12 device_map="auto",
13 trust_remote_code=True
14)
Recovery Recommendations
To improve factual knowledge after pruning:
LoRA Fine-tuning (Recommended)
1from peft import LoraConfig, get_peft_model
2
3lora_config = LoraConfig(
4 r=32,
5 lora_alpha=64,
6 target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
7 "gate_proj", "up_proj", "down_proj"],
8 lora_dropout=0.05,
9)
10model = get_peft_model(model, lora_config)
11# Fine-tune on OpenOrca, Alpaca, or domain-specific data
Expected recovery: MMLU could reach 45-55% with fine-tuning.
Technical Specifications
| Attribute | Value |
|---|
| Architecture | Transformer decoder-only |
| Layers | 30 |
| Hidden Size | 4096 |
| Attention Heads (Q) | 32 |
| Attention Heads (KV) | 8 (GQA) |
| Intermediate Size | 12288 |
| Vocabulary Size | 151,669 |
| Max Context Length | 32,768 tokens |
| Precision | bfloat16 |
Citation
If you use this model, please cite the original LaCo paper and Qwen3:
1@article{yang2024laco,
2 title={LaCo: Large Language Model Pruning via Layer Collapse},
3 author={Yang, Yifei and Cao, Zouying and Zhao, Hai},
4 journal={arXiv preprint arXiv:2402.11187},
5 year={2024}
6}
7
8@misc{qwen3technicalreport,
9 title={Qwen3 Technical Report},
10 author={Qwen Team},
11 year={2025},
12 eprint={2505.09388},
13 archivePrefix={arXiv},
14 primaryClass={cs.CL},
15 url={https://arxiv.org/abs/2505.09388}
16}
References
License
Apache 2.0 (same as base Qwen3 model)
Acknowledgments
- Qwen Team for the excellent Qwen3-8B-Base model
- LaCo authors for the pruning methodology
- Hugging Face for model hosting
