You can deploy this model and start earning money today!

simaai/
Llama-2-7b-chat-hf-a16w4_return_logits

Name: Llama-2-7b-chat-hf-a16w4_return_logits
Availability: InStock

Llama-2-7b-chat-hf-a16w4_return_logits: Optimized for SiMa.ai Modalix Accuracy Benchmarking

Overview

This repository contains the Llama-2-7b-chat-hf-a16w4_return_logits model, optimized and compiled for the SiMa.ai Modalix platform.

This model is compiled with --return_logits so that it can be used for Modalix accuracy benchmarking without recompiling the source model.

Model Architecture: Llama 2 (7B parameters)
Quantization: A16W4 (16-bit activations, 4-bit weights)
Maximum context length: 2048
Source Model: meta-llama/Llama-2-7b-chat-hf
Return logits: Enabled

Accuracy Benchmark Results

The table below compares accuracy results produced by SiMa.ai on the Modalix backend against Hugging Face backend reference results.

Task	Version	Filter	Metric	Direction	HF Backend Value	HF Stderr	Modalix Backend Value	Modalix Stderr	Abs. Diff
hellaswag	1.0	none	acc	higher	0.5798	0.0049	TBD	TBD	TBD
hellaswag	1.0	none	acc_norm	higher	0.7603	0.0043	TBD	TBD	TBD
piqa	1.0	none	acc	higher	0.7622	0.0099	TBD	TBD	TBD
piqa	1.0	none	acc_norm	higher	0.7699	0.0098	TBD	TBD	TBD
triviaqa	3.0	remove_whitespace	exact_match	higher	0.0800	0.0020	TBD	TBD	TBD
wikitext	2.0	none	bits_per_byte	lower	0.6756	N/A	TBD	N/A	TBD
wikitext	2.0	none	byte_perplexity	lower	1.5972	N/A	TBD	N/A	TBD
wikitext	2.0	none	word_perplexity	lower	12.2326	N/A	TBD	N/A	TBD
winogrande	1.0	none	acc	higher	0.6843	0.0131	TBD	TBD	TBD

Evaluation Sample Counts

Task	sample_len
hellaswag	10042
piqa	1838
triviaqa	17944
wikitext	62
winogrande	1267

Prerequisites

To benchmark accuracy with this model, you need:

SiMa.ai Modalix Device
SiMa.ai CLI: Installed on your Modalix device.
SiMa.ai Neat Runtime: Install or update the Neat Library on Modalix. The LLiMa runtime is installed as part of the Neat runtime.
LLiMa benchmark CLI: Installed on the host machine used to launch accuracy benchmarking.
Hugging Face CLI: Optional, for downloading the model on a host before copying it to Modalix.

Installation & Deployment

Follow these steps to deploy the model to your Modalix device.

1. Install or Update Neat Runtime

Note: This is a one-time setup. If the Neat Library is already installed on your Modalix device, you can skip this step and continue with model download.

Follow the SiMa.ai Neat getting started guide to install or update the Neat Library on your Modalix device.

The llima CLI is available on Modalix after the Neat runtime is installed. It manages precompiled GenAI models under /media/nvme/llima/models by default. Set LLIMA_MODELS_PATH to use a different model directory.

2. Download the Model

Download the compiled model assets from this repository directly to your device.

bash

# Download the model to a local directory
llima pull Llama-2-7b-chat-hf-a16w4_return_logits

Alternatively, you can download the compiled model to a Host and copy it to the Modalix device:

bash

hf download simaai/Llama-2-7b-chat-hf-a16w4_return_logits --local-dir Llama-2-7b-chat-hf-a16w4_return_logits
scp -r Llama-2-7b-chat-hf-a16w4_return_logits sima@<modalix-ip>:/media/nvme/llima/models/

Replace <modalix-ip> with the IP address of your Modalix device.

Expected Directory Structure:

text

/media/nvme/llima/
`-- models/
    `-- Llama-2-7b-chat-hf-a16w4_return_logits/   # The compiled model

Usage

Modalix Backend Accuracy

Run the benchmark from a host machine. The positional model_id must be the source Hugging Face model because MOLE uses its tokenizer and configuration; --board_model selects this compiled artifact on Modalix.

bash

MODALIX_IP="<modalix-ip>"
llima-benchmark accuracy meta-llama/Llama-2-7b-chat-hf \
    --backend modalix \
    --output ./results/modalix \
    --board_ip "$MODALIX_IP" \
    --board_model Llama-2-7b-chat-hf-a16w4_return_logits \
    --task hellaswag piqa triviaqa wikitext winogrande

The compiled --board_model must already exist on Modalix and contain both devkit/ and elf_files/. The CLI starts the benchmark server by default. Add --board_venv_path /path/to/venv when llima is installed in a non-default virtual environment, or --no-board_start_server when connecting to a server that is already running.

Hugging Face Backend Accuracy

Run the same task suite against the source model on the host:

bash

llima-benchmark accuracy meta-llama/Llama-2-7b-chat-hf \
    --backend hf \
    --output ./results/hf \
    --task hellaswag piqa triviaqa wikitext winogrande

See MOLE accuracy benchmarking for the full accuracy benchmarking workflow.

Limitations

Quantization: This model is quantized (A16W4/A16W8) for Modalix execution. Minor deviations from the full-precision source model may occur.
Return logits: This model is compiled with --return_logits for accuracy benchmarking workflows. For standard text generation deployments, use the corresponding SiMa.ai model from the Large Language Models collection.

Troubleshooting

sima-cli not found: Ensure that sima-cli is installed on your Modalix device.
llima-benchmark not found: Ensure that the LLiMa benchmark CLI is installed on the host machine.
llima not found: Install or update the Neat Library. See Getting Started.
Modalix accuracy benchmark fails: Verify that the model was compiled with --return_logits and that the model directory contains both devkit/ and elf_files/.
Model can't be found on Modalix: Verify the model directory is exactly inside /media/nvme/llima/models/ and not nested (e.g., /media/nvme/llima/models/Llama-2-7b-chat-hf-a16w4_return_logits/Llama-2-7b-chat-hf-a16w4_return_logits).
Permission Denied: Ensure you have read/write permissions for the /media/nvme directory.

simaai/Llama-2-7b-chat-hf-a16w4_return_logits

Llama-2-7b-chat-hf-a16w4_return_logits: Optimized for SiMa.ai Modalix Accuracy Benchmarking

Overview

Accuracy Benchmark Results

Evaluation Sample Counts

Prerequisites

Installation & Deployment

1. Install or Update Neat Runtime

2. Download the Model

Usage

Modalix Backend Accuracy

Hugging Face Backend Accuracy

Limitations

Troubleshooting

Resources

simaai/
Llama-2-7b-chat-hf-a16w4_return_logits