Method: Mixed quantization (mixed_3_6)
Bits per weight: ~4.5
Details: 3-bit for most layers, 6-bit for attention value projections, down projections, and lm_head in select layers
Converted with: mlx-lm

Usage

pip install mlx-lm

python

from mlx_lm import load, generate

model, tokenizer = load("alankessler/Mistral-Small-3.2-24B-Instruct-2506-MLX-mixed_3_6")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Hello!"}],
    add_generation_prompt=True,
    tokenize=False,
)
response = generate(model, tokenizer, prompt=prompt, max_tokens=512)
print(response)

Base Model

Model: Mistral Small 3.2 24B Instruct 2506
Parameters: 24B
Architecture: Mistral Small 3.2
License: Apache 2.0