On-device default MLX 4-bit quantised build of lemer — Gemma 4 E2B with the Lethean Ethical Kernel (LEK) merged into the text attention weights, quantised to 4 bits per weight via mlx-vlm's native quantisation (affine mode, group size 64). Full multimodal support preserved (text, image, audio). Effective rate: 6.851 bits per weight average (embeddings and sensitive layers kept at higher precision). This is the default on-device variant — smallest footprint, fastest inference, best for consumer Apple Silicon.
The Lethean Ethical Kernel (LEK) has been merged directly into the text attention projections (100 q/k/v/o_proj layers) of Gemma 4 E2B via LoRA finetune, then folded into the base weights. The vision tower and audio tower are preserved unmodified from Google's upstream — LEK only shifts text reasoning.
This variant is MLX Q4 quantised from the merged model — the smallest, fastest multimodal Lemma variant suitable for on-device inference on consumer Apple Silicon. Single safetensor file, ~4.1 GB. Quantisation is 4 bits for attention/MLP weights, with embeddings and selected layers kept at higher precision (hence the 6.851 bits/weight average). Verified on M3 Ultra at 145+ tokens/sec generation via mlx-lm; vision inference tested against COCO sample images via mlx-vlm with accurate descriptions.
Use this variant when:
You want the default on-device Lemma experience
You're running on consumer Apple Silicon (M1/M2/M3 base, Air, Pro, Studio)
You need the fastest inference with acceptable quality
Then any OpenAI-compatible client can hit http://localhost:8080/v1/chat/completions. Works with LM Studio, pi-coding-agent, OpenWebUI, and any other OpenAI-API-compatible client.
Note: use mlx_vlm.server (not mlx_lm.server) because lemer is multimodal. The text-only mlx_lm.server does not correctly route the vision/audio tensors for Gemma 4.
Recommended Sampling
Per Google's Gemma 4 model card, use these across all use cases. Gemma 4 is calibrated for temperature=1.0 — greedy / temperature=0 is NOT recommended and will measurably underperform.
Parameter
Value
temperature
1.0
top_p
0.95
top_k
64
Already set in generation_config.json.
Model Details
Property
Value
Architecture
Gemma 4 E2B
Format
MLX Q4 (affine quantisation)
Quantisation bits
4 (6.851 bits/weight average including full-precision layers)
Quantisation group size
64
Parameters
5.1B total, 2.3B effective (Per-Layer Embeddings)
Layers
35 text decoder layers
Context Length
128K tokens
Vocabulary
262K tokens
Modalities
Text, Image, Audio
Vision Encoder
~150M params (preserved unmodified from Google)
Audio Encoder
~300M params (preserved unmodified from Google)
Weight file
Single model.safetensors (~4.1 GB)
LEK delta
LoRA rank 8 merged into 100 text attention projections, then quantised
Quantisation source
lthn/lemer-mlx-bf16 via mlx_vlm.convert(quantize=True, q_bits=4, q_group_size=64)
mlx-lm generation: ~145 tokens/sec on text-only inference
Peak runtime memory: ~3.4 GB (ample headroom for context growth)
Vision inference: correct multi-object scene description on COCO test images
Should run comfortably on M1/M2/M3/M4 Air (8 GB RAM) for text inference, and on Pro/Max/Ultra variants for full multimodal workloads.
Full Model Card
Detailed documentation — Lemma family overview, GGUF variants, capability map, benchmarks, the "why EUPL-1.2" framing, and the Roadmap — lives on the main repo:
Lethean is a social enterprise building ethical AI infrastructure. The Lemma model family is part of the LEM (Lethean Ethical Model) project — training protocol and tooling for intrinsic ethical alignment of language models via consent-based LoRA finetunes, shipped EUPL-1.2 so the ethical layer stays in the open.
