The model has been quantized to 8-bit to dramatically reduce memory requirements while retaining near-lossless reasoning and coding capabilities. It is optimized for local inference on Apple Silicon Macs using the mlx-lm library.

How to Use with MLX

Install the required dependency:

pip install --upgrade mlx-lm

Run inference from Python:

python

from mlx_lm import load, generate

# Load the 8-bit quantized MLX model.
model, tokenizer = load("mlx-community/gemma-4-12b-coder-fable5-composer2.5-8bit")

prompt = "Write a Python script to sort a dictionary by its values."
messages = [{"role": "user", "content": prompt}]

formatted_prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

response = generate(
    model,
    tokenizer,
    prompt=formatted_prompt,
    verbose=True,
    max_tokens=1024,
)

python

response = generate(
    model,
    tokenizer,
    prompt=formatted_prompt,
    verbose=True,
    max_tokens=1024,
    temp=0.0,
)

Base and License

Base model: google/gemma-4-12B-it
Original fine-tune: yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1
License: Apache 2.0

Free to use, modify, and redistribute under the Apache 2.0 license.