Due to the license regulation of LLaMA, we are not allowed to release the accelerated parameters directly. We hope to discuss with you guys to figure out a legal way to share lyraLLaMA. If you have any suggestions, please feel free to drop us a line at benbinwu@tencent.com.
Model Card for lyraLLaMA
lyraLLaMA is currently the fastest LLaMA-13b available. The inference speed of lyraLLaMA has achieved 3000+ tokens/s on A100, up to 6x acceleration upon the torch version.
Among its main features are:
device: Nvidia GPU with Amperer architecture or Volta architecture (A100 or higher, V100).
batch_size: compiled with dynamic batch size, maximum depends on device.
We use the LLaMA.13B model for measurement, but this optimized inference is applicable to LLaMA models of different sizes.
Speed
Evaluated at tokens/s
test on A100 40G
fp16 and MEMOPT precision
LLaMA-Ziya-13B
Version
Batch Size 1
Batch Size 8
Batch Size 16
Batch Size 32
Batch Size 64
Torch LLaMA
31.74
289.2
521.37
775.69
OOM
lyraLLaMA fp16
73.2
565.6
1179.59
1795.63
3061.27
lyraLLaMA MEMOPT
104
770.5
1389.9
2390.4
3782.1
LLaMA-Vicuna-13B
Version
Batch Size 1
Batch Size 8
Batch Size 16
Batch Size 32
Batch Size 64
Torch LLaMA
24.65
167.3
322.97
407.99
OOM
lyraLLaMA fp16
53.67
421.38
804.31
1519.28
2679.82
lyraLLaMA MEMOPT
79.81
603.15
1117.27
1966.52
3200.32
Docker Environment Recommendation
For Cuda 11.X: we recommend nvcr.io/nvidia/pytorch:22.12-py3
For Cuda 12.0: we recommend nvcr.io/nvidia/pytorch:23.02-py3
bash
1docker pull nvcr.io/nvidia/pytorch:23.02-py3
2docker run --rm -it --gpus all -v ./:/lyraLLaMA nvcr.io/nvidia/pytorch:23.02-py3
34pip install -r requirements.txt
5python demo.py
#### input
List 3 different machine learning algorithms and explain where they are applicable.
#### output
Outputs:
Linear Regression: Linear regression is a simple, yet powerful algorithm that is used to predict the value of a continuous variable based on one or more input features. It is commonly used in prediction and forecasting tasks, such as predicting the price of a stock or the sales of a product.
Decision Trees: Decision Trees are a type of supervised learning algorithm that can be used for both classification and regression tasks. They work by partitioning the feature space into smaller subspaces, with each subspace corresponding to a leaf node in the tree. Decision Trees are commonly used in applications such as credit risk assessment and customer segmentation.
Support Vector Machines (SVMs): SVMs are a type of supervised learning algorithm that can be used for both classification and regression tasks. They work by finding the best hyperplane that separates the data into different classes. SVMs are commonly used in applications such as image classification and natural language processing.
TODO
Support for int4
Inference for longer context situations
Streaming inference mode.
Citation
@Misc{lyraLLaMA2023,
author = {Kangjian Wu, Zhengtao Wang, Yibo Lu, Bin Wu},
title = {lyraLLaMA: Accelerating LLaMA-13b(fp16) to 3000+ tokens/s},
howpublished = {\url{https://huggingface.co/TMElyralab/lyraLLaMA}},
year = {2023}
}
