You can deploy this model and start earning money today!

anemll/
anemll-DeepHermes-3-Llama-3-8B-Preview-ctx1024_0.1.1

Name: anemll-DeepHermes-3-Llama-3-8B-Preview-ctx1024_0.1.1
Availability: InStock

Views

No views yet

NousResearch/DeepHermes-3-Llama-3-8B-Preview converted to Apple Neural Engine via ANEMLL

ANEMLL

ANEMLL (pronounced like "animal") is an open-source project focused on accelerating the porting of Large Language Models (LLMs) to tensor processors, starting with the Apple Neural Engine (ANE).

The goal is to provide a fully open-source pipeline from model conversion to inference for common LLM architectures running on ANE.

This enables seamless integration and on-device inference for low-power applications on edge devices, ensuring maximum privacy and security.

This is critical for autonomous applications, where models run directly on the device without requiring an internet connection.

License

ANEMLL is licensed under the MIT License.
The model is based on Meta's LLaMA 3.2 and may require a separate license.

This test model is exclusively for the Meta's LLaMA 3.2 1B (1024 context) model converted for CoreML, released before the official launch of the ANEMLL repository and minimal documentation. It is intended for early adopters only who requested an early release.

Requirements

macOS Sequoia with Apple Neural Engine and 16GB RAM
CoreML Tools and HuggingFace Transformers libraries
Python 3.9

chat.py provides a sample inference script.
chat_full.py provides a sample inference script with history and conversation management.

Installation

Download the model from Hugging Face:

bash

# Install required tools
pip install huggingface_hub

# Install Git LFS (Large File Support)
# macOS with Homebrew:
brew install git-lfs
# Or Ubuntu/Debian:
# sudo apt-get install git-lfs

# Initialize Git LFS
git lfs install

# Clone the repository with model files
git clone https://huggingface.co/anemll/anemll-DeepHermes-3-Llama-3-8B-Preview-ctx1024_0.1.1

Extract model files:

bash

# Navigate to cloned directory
cd anemll-DeepHermes-3-Llama-3-8B-Preview-ctx1024_0.1.1

# Pull LFS files (model weights)
git lfs pull

# Extract CoreML model files
find . -type f -name "*.zip" -exec unzip {} \;

Install dependencies:

pip install coremltools transformers

Coremltools:

See coremltools installation guide at https://coremltools.readme.io/v4.0/docs/installation

How to Run

Basic chat interface:

python chat.py --meta ./meta.yaml

Full conversation mode with history: (use /t to switch thinking mode for HERMES-3)

python chat_full.py --meta ./meta.yaml

Note: The first time the model loads, macOS will take some time to place it on the device. Subsequent loads will be instantaneous. Use Ctrl-D to exit, Ctrl-C to interrupt inference.

More Info Please check following links for later updates:

realanemll@gmail.com