ANEMLL (pronounced like "animal") is an open-source project focused on accelerating the porting of Large Language Models (LLMs) to tensor processors, starting with the Apple Neural Engine (ANE).

The goal is to provide a fully open-source pipeline from model conversion to inference for common LLM architectures running on ANE.

This enables seamless integration and on-device inference for low-power applications on edge devices, ensuring maximum privacy and security.

This is critical for autonomous applications, where models run directly on the device without requiring an internet connection.

For more information, visit the ANEMLL GitHub repository.

License

ANEMLL is licensed under the MIT License. The original model may require a separate license depending on the architecture:

LLaMA models: Based on Meta's LLaMA and may require Meta's license
Qwen models: Based on Alibaba's Qwen and may require Alibaba's license
Gemma models: Based on Google's Gemma and subject to Gemma Terms of Use
Other models: Check respective original model licenses

This model is converted for CoreML using ANEMLL's open-source conversion pipeline. It supports multiple LLM architectures including LLaMA, Qwen, Gemma, and DeepSeek variants.

Requirements

macOS 15 (Sequoia) or later with Apple Neural Engine and 8GB RAM or more
CoreML Tools 8.x+ and HuggingFace Transformers libraries
Python 3.9+

chat.py provides a sample inference script.
chat_full.py provides a sample inference script with history and conversation management.

Installation

Download the model from Hugging Face:

bash

# Install required tools
pip install huggingface_hub

# Install Git LFS (Large File Support)
# macOS with Homebrew:
brew install git-lfs
# Or Ubuntu/Debian:
# sudo apt-get install git-lfs

# Initialize Git LFS
git lfs install

# Clone the repository with model files
git clone https://huggingface.co/anemll/anemll-google-gemma-3-4b-it-qat-int4-unquantized-ctx4096_0.3.5

Extract model files:

bash

# Navigate to cloned directory
cd anemll-google-gemma-3-4b-it-qat-int4-unquantized-ctx4096_0.3.5

# Pull LFS files (model weights)
git lfs pull

# Extract CoreML model files
find . -type f -name "*.zip" -exec unzip {} \;

Install dependencies:

pip install coremltools transformers

Coremltools:

See coremltools installation guide at https://apple.github.io/coremltools/docs-guides/source/installing-coremltools.html

How to Run

Basic chat interface:

python chat.py --meta ./meta.yaml

Full conversation mode with history:

python chat_full.py --meta ./meta.yaml

Note: The first time the model loads, macOS will take some time to place it on the device. Subsequent loads will be instantaneous. Use Ctrl-D to exit, Ctrl-C to interrupt inference.

More Info Please check following links for later updates:

realanemll@gmail.com

anemll-google-gemma-3-4b-it-qat-int4-unquantized-ctx4096_0.3.5

This is a CoreML model converted using ANEMLL for Apple Neural Engine inference.

Available Distributions

Standard Distribution

Contains zipped MLMODELC files
Suitable for macOS and development

iOS Distribution

Contains unzipped MLMODELC files
Ready for iOS deployment
Includes offline tokenizer support

Model Information

Context Length: 4096
Batch Size: 64
Number of Chunks: 3
LUT Quantization: 6

Quick Start

Test in iOS/macOS App

Try our sample Chat-Bot app on TestFlight:

Install TestFlight from App Store
Join beta test: TestFlight Link
App includes a small demo model pre-installed
You can add custom models via HuggingFace URLs

[!Note]

The TestFlight app works on both iOS and macOS

Demonstrates proper model integration and provides a reference implementation

iOS requires unzipped MLMODELC files and config.json for offline tokenizer

macOS supports both zipped and unzipped model formats

anemll/anemll-google-gemma-3-4b-it-qat-int4-ctx4096_0.3.5

ANEMLL