This repository contains a ready-to-run .litertlm bundle converted from the original Hugging Face BF16 Safetensors checkpoint for use with Google LiteRT-LM.

This repository is not a new fine-tune and does not claim authorship of the original model weights. It is a runtime-format conversion of the Huihui model.

Model lineage

text

google/gemma-4-E4B
        ↓
google/gemma-4-E4B-it
        ↓
huihui-ai/Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated
        ↓
BF16 Safetensors
        ↓
litert-torch export_hf
        ↓
Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm

Property	Value
Base model	`huihui-ai/Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated`
Format	LiteRT-LM (`.litertlm`)
Architecture	Gemma 4 E4B
Quantization	`dynamic_wi4_afp32`
Source model	QAT Q4
Context length	4096 tokens
Prefill lengths	128, 256, 512
Vision	✅ Included
Audio	❌ Audio encoder is not present in the resulting bundle
Tool calling metadata	✅
Thinking channel metadata	✅
Target	CPU / GPU edge inference

About the source model

The source model was published by huihui-ai as an abliterated variant of google/gemma-4-E4B-it.

According to the original model card, abliteration significantly reduces the model's refusal behavior and safety filtering. This conversion does not restore or add safety alignment.

Please read the source model card before use:

https://huggingface.co/huihui-ai/Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated

Conversion

The model was converted from the original Hugging Face Safetensors checkpoint using the LiteRT Torch Hugging Face exporter.

The conversion command was:

bash

litert-torch export_hf \
  --model="$HOME/Models/huggingface/huihui-gemma-4-e4b-qat" \
  --output_dir="$HOME/Models/litert/huihui-gemma-4-e4b-qat-full" \
  --task=image_text_to_text \
  --cache_length=4096 \
  --prefill_lengths=128,256,512 \
  --externalize_embedder=true \
  --moe_exports_implementation=eager \
  --export_vision_encoder=true \
  --export_audio_encoder=true \
  --quantization_recipe=dynamic_wi4_afp32 \
  --jinja_chat_template_override=litert-community/gemma-4-E4B-it-litert-lm \
  --bundle_litert_lm=true

The LiteRT-compatible Gemma 4 E4B chat template was taken from:

litert-community/gemma-4-E4B-it-litert-lm

The resulting bundle is:

Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm

No GGUF or Ollama model was used as an intermediate format.

Download

Install the Hugging Face CLI if needed:

pip install -U huggingface_hub

Download only the LiteRT-LM bundle:

bash

hf download \
  vokash3/Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated-LiteRT-LM \
  Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm \
  --local-dir .

This produces:

./Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm

Run with LiteRT-LM

Install the current LiteRT-LM CLI:

uv tool install litert-lm

Run the model:

bash

litert-lm run \
  ./Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm \
  --prompt="Hello! Tell me briefly who you are."

For an interactive run:

litert-lm run ./Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm

Available backends and CLI options can change as LiteRT-LM evolves, so check the installed version before using backend-specific flags:

litert-lm run --help

Benchmarks

The benchmark figures from the earlier non-QAT LiteRT-LM conversion are not reused here because this artifact is produced from a different upstream checkpoint and uses the dynamic_wi4_afp32 LiteRT export recipe.

This bundle should be benchmarked independently. For example:

bash

litert-lm benchmark \
  Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm \
  --backend=gpu

bash

litert-lm benchmark \
  Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm \
  --backend=cpu

The intended reference workload is 256 prefill tokens, 256 decode tokens and a 4096-token maximum context.

QAT and LiteRT quantization

The direct source checkpoint is the Huihui QAT/Q4_0 unquantized model. QAT is part of the upstream model lineage; the LiteRT conversion then applies its own runtime-oriented dynamic_wi4_afp32 recipe.

These are separate stages and should not be interpreted as the same quantization format.

Conversion environment

text

ai-edge-litert             2.2.0
ai-edge-litert-nightly     2.3.0.dev20260822
aiosqlite                  0.22.1
litert-cli-nightly         0.2.0.dev20260824
litert-converter           0.5.0.dev20260822
litert-lm                  0.16.1
litert-lm-api              0.16.1
litert-lm-builder          0.16.1
litert-torch-nightly       0.10.0.dev20260824

Several components are nightly builds, so conversion behavior and compatibility may change between releases.

Platforms

LiteRT-LM is designed for on-device inference across supported desktop and edge platforms. Consult the upstream project for the current platform/backend support matrix:

https://github.com/google-ai-edge/LiteRT-LM

The .litertlm file in this repository was created as a portable LiteRT-LM bundle rather than a GGUF model for llama.cpp/Ollama.

Google AI Edge Gallery / Edge Studio compatibility

The refreshed bundle is exported as an image_text_to_text model with the vision encoder and vision adapter included.

The model has been successfully imported and initialized in Google AI Edge Gallery / Edge Studio, including AI Chat mode.

The working export uses:

text

--task=image_text_to_text
--export_vision_encoder=true
--export_audio_encoder=true
--prefill_lengths=128,256,512

Inspection of the generated bundle confirms the presence of the text inference components plus the Gemma 4 vision encoder and vision adapter.

Audio status

The export was requested with --export_audio_encoder=true, but inspection of the resulting .litertlm bundle does not show a separate audio encoder. Therefore this release should currently be treated as a text + vision bundle; audio support is not claimed.

Benchmarks

Benchmark configuration:

text

Prefill tokens : 256
Decode tokens  : 256
Context        : 4096

Device	Backend	Prefill, tok/s	Decode, tok/s	TTFT
Radeon RX 5700 XT	GPU	926.31	47.82	0.297 s
Intel Core i5-11400	CPU	177.40	7.18	1.582 s
Mac mini M4	GPU	531.24	25.16	0.522 s
Mac mini M4	CPU	405.78	13.71	0.704 s
MacBook Air M1	GPU	273.03	13.80	1.010 s
MacBook Air M1	CPU	129.90	7.93	2.097 s
MacBook Neo A18 Pro	GPU	239.40	14.51	1.138 s
MacBook Neo A18 Pro	CPU	285.49	9.41	1.003 s
iPhone 16 Pro Max A18 Pro	GPU	212.51	18.35	1.34 s
iPhone 16 Pro Max A18 Pro	CPU	221.49	6.21	1.43 s
Tecno Camon 40 Pro 5G	GPU	100.78	2.96	2.90 s
Tecno Camon 40 Pro 5G	CPU	34.54	2.27	7.87 s

Linux desktop benchmark

Test system:

text

OS:     Linux Mint 22.1
CPU:    Intel Core i5-11400
RAM:    32 GB
GPU:    AMD Radeon RX 5700 XT
VRAM:   8 GB
Driver: radeonsi / Navi 10
Kernel: Linux 6.8

GPU

text

Prefill speed:        926.31 tokens/s
Decode speed:          47.82 tokens/s
Init time:              4.4219 s
Time to first token:    0.2973 s

CPU

text

Prefill speed:        177.40 tokens/s
Decode speed:           7.18 tokens/s
Init time:              1.5860 s
Time to first token:    1.5824 s

Apple Silicon

Mac mini M4 16 GB

Backend	Prefill	Decode	TTFT
GPU	531.24 tok/s	25.16 tok/s	0.522 s
CPU	405.78 tok/s	13.71 tok/s	0.704 s

MacBook Air M1 16 GB

Backend	Prefill	Decode	TTFT
GPU	273.03 tok/s	13.80 tok/s	1.010 s
CPU	129.90 tok/s	7.93 tok/s	2.097 s

MacBook Neo A18 Pro 8 GB

Backend	Prefill	Decode	TTFT
GPU	239.40 tok/s	14.51 tok/s	1.138 s
CPU	285.49 tok/s	9.41 tok/s	1.003 s

Mobile devices

Mobile benchmarks were performed using Google AI Edge Gallery.

iPhone 16 Pro Max — A18 Pro

Backend	Prefill	Decode	TTFT
GPU	212.51 tok/s	18.35 tok/s	1.34 s
CPU	221.49 tok/s	6.21 tok/s	1.43 s

Tecno Camon 40 Pro 5G — Dimensity 7300 Ultimate

Backend	Prefill	Decode	TTFT
GPU	100.78 tok/s	2.96 tok/s	2.90 s
CPU	34.54 tok/s	2.27 tok/s	7.87 s

Notes

The Radeon RX 5700 XT provides the highest measured performance among the tested devices:

text

Prefill: 926.31 tokens/s
Decode:   47.82 tokens/s

The model also runs successfully on mobile devices through Google AI Edge Gallery.

Performance depends heavily on the LiteRT backend, GPU drivers, available memory, operating system, and LiteRT-LM version. The benchmark numbers above should therefore be treated as practical reference measurements rather than universally reproducible performance figures.

Important notes and limitations

This is a conversion, not a separately trained model.
Behavioral characteristics of the source Huihui model remain relevant.
The source model has substantially reduced refusal/safety behavior.
Output can include sensitive, controversial, unsafe, inaccurate or otherwise undesirable content.
Users should review outputs before using them in production or public-facing applications.
Runtime behavior may differ from Transformers or GGUF builds because of differences in quantization, execution backend, sampling, prompt formatting and runtime implementation.
Exact token-for-token parity with the original Safetensors checkpoint is not expected.
This repository does not provide independent benchmark or quality claims.
The conversion is intended for LiteRT-LM multimodal chat / text-and-vision use.

License and upstream terms

The source Huihui repository is published on Hugging Face with the Apache-2.0 license metadata.

Users should also review the terms, notices and usage requirements associated with the upstream Gemma model family and the original Huihui model before redistribution or deployment.

Upstream resources:

Huihui source model: https://huggingface.co/huihui-ai/Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated
Google Gemma 4 E4B IT: https://huggingface.co/google/gemma-4-E4B-it
LiteRT-LM: https://github.com/google-ai-edge/LiteRT-LM
LiteRT Torch: https://github.com/google-ai-edge/litert-torch
LiteRT Community Gemma 4 E4B: https://huggingface.co/litert-community/gemma-4-E4B-it-litert-lm

Credits

This repository is a runtime-format conversion and packaging project built on top of several upstream models, tools and community resources.

Google / Google DeepMind

Google / Google DeepMind created the Gemma model family and the Gemma 4 architecture underlying this model. The instruction-tuned upstream family is represented by google/gemma-4-E4B-it.

Google also develops the LiteRT ecosystem used for on-device execution.

huihui-ai

huihui-ai created and published the direct source checkpoint:

huihui-ai/Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated

The QAT/Q4_0 and abliterated characteristics originate from the upstream Huihui model. This repository does not claim authorship of those weights, training changes or behavioral modifications.

LiteRT-LM

Google AI Edge LiteRT-LM provides the runtime and .litertlm bundle format used by this repository. It provides the infrastructure for packaging model components, tokenizer data, metadata, KV-cache configuration and executable LiteRT/TFLite components for local inference.

LiteRT Torch

LiteRT Torch provides the Hugging Face export pipeline used to create this artifact. The conversion uses litert-torch export_hf with the dynamic_wi4_afp32 recipe, externalized embeddings, Gemma 4 MoE export settings, multiple prefill lengths and multimodal export options.

LiteRT Community

LiteRT Community published the Gemma 4 E4B LiteRT-LM reference repository used for the compatible Jinja chat template:

litert-community/gemma-4-E4B-it-litert-lm

vokash3

vokash3 performed and published the LiteRT-LM conversion in this repository, including source-checkpoint preparation, export configuration experiments, selection of the LiteRT quantization recipe, multimodal packaging, bundle inspection, LiteRT-LM compatibility testing, Google AI Edge Gallery testing, documentation of the current audio limitation, and publication of the resulting artifact.

No authorship of the original Gemma or Huihui weights is claimed.

Upstream resources

Google Gemma 4 E4B IT: https://huggingface.co/google/gemma-4-E4B-it
Huihui source model: https://huggingface.co/huihui-ai/Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated
LiteRT-LM: https://github.com/google-ai-edge/LiteRT-LM
LiteRT Torch: https://github.com/google-ai-edge/litert-torch
LiteRT Community Gemma 4 E4B: https://huggingface.co/litert-community/gemma-4-E4B-it-litert-lm

Repository contents

text

.
├── README.md
└── Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm

Reproducibility summary

text

Source:
huihui-ai/Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated

Source format:
QAT Q4_0 unquantized Hugging Face checkpoint

Exporter:
litert-torch export_hf

Options:
--task=image_text_to_text
--cache_length=4096
--prefill_lengths=128,256,512
--externalize_embedder=true
--moe_exports_implementation=eager
--export_vision_encoder=true
--export_audio_encoder=true
--quantization_recipe=dynamic_wi4_afp32
--jinja_chat_template_override=litert-community/gemma-4-E4B-it-litert-lm
--bundle_litert_lm=true

Output:
Huihui-gemma-4-E4B-it-qat-q4_0-unquantized-abliterated.litertlm