Bringing SOTA quantization to mobile LLM deployment: A practical Executorch integration guide

Usage

• Download and install the .apk file on your Android phone. (llama3-aqlm.apk for ~1.1 tok/s at low power consumption. llama3-aqlm-4cores.apk for ~2.7 tok/s at high loads)
• Download the .pte and .model files and put them into the /data/local/tmp/llama folder on your Android phone.
• Running the app you will see the option to load the .pte and .model files. After loading them, you'll be able to chat with the model.

Requirements

Limitations

• Although the app looks like chat, generation requests are independent.
• Llama-3 chat template is hard-coded into the app.