Pre-quantization: The model was pre-quantized externally using the LLM-Compressor library (compressed-tensors / safetensors format) prior to compilation.
Performance
Model
Precision
Device
Token Length
Response Rate (tokens/sec)
Time To First Token (sec)
Llama-3.2-1B-Instruct
A16W4
Modalix
128
68.84 tokens/sec
0.04 sec
Llama-3.2-1B-Instruct
A16W4
Modalix
256
67.81 tokens/sec
0.08 sec
Llama-3.2-1B-Instruct
A16W4
Modalix
512
65.56 tokens/sec
0.17 sec
Llama-3.2-1B-Instruct
A16W4
Modalix
1024
61.52 tokens/sec
0.35 sec
Benchmark Results
These models were evaluated against a large collection of different datasets and
metrics to cover different aspects of text generation:
Reasoning and factuality
Benchmark
N-Shot
Score
winogrande
0-shot
0.6085
Prerequisites
To run this model, you need:
SiMa.ai Modalix Device
SiMa.ai CLI: Installed on your Modalix device.
SiMa.ai Neat Runtime: Install or update the Neat Library on Modalix. The LLiMa runtime is installed as part of the Neat runtime.
Hugging Face CLI: Optional, for downloading the model on a host before copying it to Modalix.
Installation & Deployment
Follow these steps to deploy the model to your Modalix device.
1. Install or Update Neat Runtime
Note: This is a one-time setup. If the Neat Library is already installed on your Modalix device, you can skip this step and continue with model download.
The llima CLI is available on Modalix after the Neat runtime is installed. It manages precompiled GenAI models under /media/nvme/llima/models by default. Set LLIMA_MODELS_PATH to use a different model directory.
2. Download the Model
Download the compiled model assets from this repository directly to your device.
bash
1# Download the model to a local directory2llima pull Llama-3.2-1B-Instruct-GPTQ-a16w4
Alternatively, you can download the compiled model to a Host and copy it to the Modalix device:
Replace <modalix-ip> with the IP address of your Modalix device.
Expected Directory Structure:
text
1/media/nvme/llima/
2└── models/
3 └── Llama-3.2-1B-Instruct-GPTQ-a16w4/ # The compiled model
Usage
Validate with LLiMa CLI
Run the model directly on Modalix:
llima run Llama-3.2-1B-Instruct-GPTQ-a16w4
For all runtime options, run:
llima run -h
GenAI Demo Application
The GenAI demo application is separate from LLiMa installation. Use the GenAI Multimodal Assistant page to install and run the demo app. Once installed, the demo app can use precompiled models such as this one.
API Usage
To serve this model with OpenAI- or Ollama-compatible APIs and send requests to it, use the GenAI server workflow in Serve GenAI Models.
For direct LLM calls without setting up a server, see Run an LLM.
Limitations
Quantization: This model is quantized (A16W4) for optimal performance on embedded devices. While this maintains high accuracy, minor deviations from the full-precision model may occur.
Troubleshooting
sima-cli not found: Ensure that sima-cli is installed on your Modalix device.
llima not found: Install or update the Neat Library. See Getting Started.
Model can't be run: Verify the model directory is exactly inside /media/nvme/llima/models/ and not nested (e.g., /media/nvme/llima/models/Llama-3.2-1B-Instruct-GPTQ-a16w4/Llama-3.2-1B-Instruct-GPTQ-a16w4).
Permission Denied: Ensure you have read/write permissions for the /media/nvme directory.
