Views
No views yet
Important — non-strict parameter budget. This checkpoint follows the original BlockPruner block-count setting. The historicalratio0.2label does not enforce an exact 20% end-to-end parameter reduction.
重要说明：参数量未被严格约束。 这是原始 BlockPruner 按 MHA/MLP 子块数量进行剪枝的 checkpoint。历史名称ratio0.2不代表全模型参数量被严格减少 20%。
meta-llama/Llama-2-7b-hf with the
original BlockPruner mix search setting. Each of the 32 Transformer layers is
split into one MHA residual sub-block and one MLP residual sub-block, giving 64
candidate sub-blocks. The final deletion set is stored in
blockpruner_config.json and in config.json.blockpruner.| Item | Value |
|---|---|
| Historical folder label | ratio0.2 |
| Removed sub-blocks | 12 / 64 (10 MHA + 2 MLP) |
| Sub-block-count reduction | 18.75% |
| Dense base parameters | 6,738,415,616 |
| Remaining checkpoint parameters | 5,796,745,216 |
| Actual total-parameter reduction | 13.974655968% |
| Weight dtype | BF16 |
LlamaForCausalLM config. Calling
AutoModelForCausalLM.from_pretrained(...) by itself can recreate the missing
sub-blocks with newly initialized weights and does not correctly restore the
pruned model.blockpruner_config.json (the authoritative final deletion set).MaskedLlamaDecoderLayer.transformers==4.39.3; the inference widget is disabled because a
mask-aware loader is required.meta-llama/Llama-2-7b-hf01c7f73d771dfac7d292323805ebc428287df4f9mix (MHA and MLP residual sub-blocks)ppl_search_file value in the config records the historical search artifact
name. It is not required for loading because the final deletion set is embedded
directly in the checkpoint config.1@article{zhong2024blockpruner,
2 title = {BlockPruner: Fine-grained Pruning for Large Language Models},
3 author = {Zhong, Longguang and Wan, Fanqi and Chen, Ruijun and Quan, Xiaojun and Li, Liangzhi},
4 journal = {arXiv preprint arXiv:2406.10594},
5 year = {2024}
6}