You can deploy this model and start earning money today!

kcnpxcp/
Mistral-Medium-3.5-128B-TQ3

Name: Mistral-Medium-3.5-128B-TQ3
Availability: InStock

Views

No views yet

Mistral-Medium-3.5-128B-TQ3

Mistral-Medium-3.5-128B quantized to 3-bit using TurboQuant TQ3.

⚠️ EXPERIMENTAL / RESEARCH PURPOSE ONLY

This model uses the TurboQuant TQ3 format — a community-driven quantization scheme. There is no official loader from the TurboQuant team yet. The current loading path relies on a community fork and may break with future updates.

What this means for you:

The quantized weights are mathematically sound and will be loadable by future official tooling

Current inference requires the turboquant_vllm fork and --enforce-eager mode

Expect Triton kernel fallbacks on some architectures (e.g., sm_120 Blackwell)

This checkpoint is provided for research and tinkering — production use at your own risk

Once official TQ3 loader support is released, this model should load directly without re-quantization.

Model Card

Property	Value
Original Model	`mistralai/Mistral-Medium-3.5-128B`
Architecture	`Mistral3ForConditionalGeneration`
Parameters	128B (dense, non-MoE)
Hidden Size	12,288
Num Layers	88
Attention Heads	96
KV Heads	8 (GQA)
Head Dim	128
Vocab Size	131,072
Context Length	262,144 tokens
RoPE	YaRN (theta=1,000,000, factor=64, β=4)
Vision Encoder	Pixtral (48 layers, hidden=1664, 16 heads, 1540×1540)
Multimodal	✅ Text + Vision
Quantization	TQ3 3-bit, group-size 128
Compressed Layers	955
Disk Size	~54 GB (11 shards)
License	Modified MIT License (see original)

Quantization Details

Property	Value
Format	`tq3_native`
Bits	3
Group Size	128
Quantizer Seed	42
Compressed Layers	955
Method	Random projection + codebook quantization (TurboQuant)

Quantization was performed using a streaming converter — one shard at a time, with minimal RAM usage.

Memory Requirements

Metric	Value
Original BF16	~245 GB
TQ3 3-bit	~54 GB
Reduction	~78%

Loading & Inference

vLLM (recommended, experimental)

bash

vllm serve kcnpxcp/Mistral-Medium-3.5-128B-TQ3 \
  --quantization turboquant \
  --enforce-eager \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

Python (turboquant_vllm fork)

python

import torch
from turboquant_vllm.checkpoint import load_tq3_model
from transformers import AutoProcessor

# Note: load_tq3_model uses AutoModelForCausalLM internally —
# the Mistral3 multimodal architecture may require manual handling
# of the vision tower and processor.
processor = AutoProcessor.from_pretrained("kcnpxcp/Mistral-Medium-3.5-128B-TQ3")
model = load_tq3_model("kcnpxcp/Mistral-Medium-3.5-128B-TQ3", device="cuda")

# Text-only inference
messages = [
    {"role": "user", "content": "Explain quantum computing in simple terms."}
]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to("cuda")

outputs = model.generate(inputs, max_new_tokens=256, do_sample=True, temperature=0.7)
print(processor.decode(outputs[0], skip_special_tokens=True))

⚠️ Multimodal Architecture Notes

This model uses Mistral3ForConditionalGeneration (Pixtral-style vision + text). The standard load_tq3_model() function is designed for AutoModelForCausalLM and may not handle the vision tower correctly. For full multimodal support, custom loading code may be needed.

Hardware Requirements

Component	Minimum	Recommended
GPU VRAM	72 GB	80 GB+
System RAM	64 GB	128 GB
Disk Space	60 GB	100 GB
GPU	NVIDIA RTX PRO 5000 / H100	A100 80GB / H100

Known Limitations

Triton kernels on sm_120 (Blackwell): May produce RuntimeError: illegal memory access — fused GEMM falls back to CUDA
MoE models: TQ3 currently only supports dense models; MoE backends (triton & flashinfer_cutlass) crash on sm_120
--enforce-eager required: Graph mode is not yet compatible with TQ3 dequantization
Throughput: ~12 tok/s on RTX PRO 5000 (eager mode, Blackwell sm_120)
Context length: The --max-model-len 8192 setting is a practical limit on 72 GB VRAM. The original model supports 262K tokens.
Multimodal loading: The load_tq3_model() function may need adaptation for the Pixtral vision architecture

File Structure

Mistral-Medium-3.5-128B-TQ3/
├── config.json                          # Model config
├── generation_config.json               # Generation parameters
├── params.json                          # Architecture parameters
├── params_extra.json                    # Additional parameters
├── tq_config.json                       # TQ3 quantization config
├── processor_config.json                # Pixtral processor config
├── chat_template.jinja                  # Chat template
├── tokenizer.json                       # Tokenizer (17 MB)
├── tokenizer_config.json                # Tokenizer config
├── tekken.json                          # Tekken tokenizer (16 MB)
├── model.safetensors.index.json         # Shard index
├── consolidated.safetensors.index.json  # Consolidated index
├── model-00001-of-00011.safetensors     # Shard 1 (~5 GB)
├── ...                                  # Shards 2-10
└── model-00011-of-00011.safetensors     # Shard 11 (~5 GB)

Citation

If you use this model in your work, please cite:

bibtex

@software{turboquant_tq3,
  title = {TurboQuant: Near-optimal weight quantization with on-the-fly dequantization},
  author = {TurboQuant Team},
  url = {https://github.com/varjoranta/turboquant-vllm},
  year = {2025}
}

Troubleshooting

Problem	Solution
`illegal memory access` with Triton kernels	Use `--enforce-eager` and verify CUDA kernels compiled correctly
Out of Memory	Reduce `--max-model-len` to 4096 or less; set `--gpu-memory-utilization 0.75`
Slow startup	First inference compiles Triton kernels (~30s); subsequent runs are normal
`cannot pickle '_Ops' object`	Use vLLM 0.20.0 with the `config.py` cloudpickle+JSON patch
Multimodal loading errors	The vision tower may need manual handling — see Architecture Notes above

Quantized with TurboQuant TQ3 on NVIDIA RTX PRO 5000 (72 GB, sm_120, Blackwell).