This model repository is part of the ACIP Project and provides a compressible version of meta-llama/Llama-2-13b-hf. For more details, please visit our code repo.

Just load the ACIP model via from_pretrained:

This will download and create a fully parameterized ACIP model that can be pruned to any compression rate you wish. For example,

will prune model to 40% if its original size measured in number of parameters, i.e., 60% compression rate. A unique feature of ACIP is that this operation is revertible in the sense that you can rerun model.prune_model_by_score as often as you like to evaluate your model at different sizes. Finally, you can "commit" to a certain ratio and run

which will discard all pruned mask values of compressible linear layers. Now the model is actually compressed and you should observe a significant decrease of memory usage (this step is not revertible without reloading the ACIP model). If you like, you can also run

to save even more memory (we have only tested 4bit quantization with bitsandbytes, but you could also customize this).

Note: The parameter size_ratio ranges from 1.0 to 0.0, indicating the model size after compression. For example, 0.4 means that the model has only 40% of the original number of parameters and 1.0 means no compression at all. Alternatively, you can also set compression_rate in prune_model_by_score, which is equivalent to size_ratio = 1.0 - compression_rate.

To run an ACIP model from our hub, you only need minimal dependencies, namely torch, transformers, peft, and optionally, bitsandbytes in case you want to quantize your model. See requirements.txt for pip-installable dependencies with exact version pins (newer version should work as well).

This model is released under the llama2 license.

When using or referring to this model, please cite our paper: