TurboQuant 8-bit-equivalent (TQ8) quantization of Qwen/Qwen2.5-0.5B, produced with the production defaults of turboquant-mlx-core's tq-convert tool (--sensitive-layers 4 --per-layer-codebooks --bits 4 --residual-bits 4) and --target-world-size 1.

Topology Capability

max_supported_world_size = 1. This snapshot was converted with per-layer block sizes that satisfy the row-parallel sharding constraint for tensor-parallel clusters of up to 1 rank. Loading it into a larger cluster is not supported and will be rejected at SwiftLM's cluster bring-up.

Quality

Metric	Value
PPL delta vs fp16 (default eval corpus)	1.33%

Measured with turboquant-mlx-models/scripts/eval_ppl.py (default ~350-token corpus, max_tokens=512).

Loading

bash

# Dequant to fp16 for use with mlx_lm
tq-dequant ./Qwen2.5-1-TQ8-TP1 ./dequanted

Native shard-aware loading via SwiftLM's distributed inference path is the intended runtime; see turboquant-mlx-core and SwiftLM.

Files

model*.safetensors — TQ-encoded quantizable weight tensors plus codebooks
model*_passthrough.safetensors — fp16 1D and sensitive-layer tensors
tq_shard_metadata.json — sidecar with shard plan, dtype tags, and max_supported_world_size = 1
config.json — base model config plus injected quantization_config block
Tokenizer files copied verbatim from the source

Reproducibility

tq-convert --model /path/to/Qwen2.5-0.5B --target-world-size 1

The default flags produce this snapshot. See the turboquant-mlx-models/benchmarks/tp_target_world_size_results.md table for cross-model comparisons and methodology notes.

ekovshilovsky/Qwen2.5-0.5B-TQ8-TP1

Qwen2.5-0.5B-TQ8-TP1

Topology Capability

Quality

Loading

Files

Reproducibility

ekovshilovsky/
Qwen2.5-0.5B-TQ8-TP1