You can deploy this model and start earning money today!

Felladrin/
gguf-Q2_K_S-Mixed-AutoRound-MiniMax-M2.5

Name: gguf-Q2_K_S-Mixed-AutoRound-MiniMax-M2.5
Availability: InStock

gguf

MiniMaxAI/MiniMax-M2.5

Views

No views yet

A Q2_K_S-Mixed GGUF version of MiniMaxAI/MiniMax-M2.5 generated with intel/auto-round, where the embedding layer and lm-head layer have 8-bit precision and the non-specialized layers have 4-bit precision.

Script for reproducing this model.

pip install transformers==4.56.0 torch==2.9.1 auto_round==0.9.4

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_round import AutoRound

model_name = "MiniMaxAI/MiniMax-M2.5"

model = AutoModelForCausalLM.from_pretrained(model_name, device_map="cpu", trust_remote_code=True, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)
layer_config = {}
for n, m in model.named_modules():
    if n == "lm_head" or isinstance(m,torch.nn.Embedding):
        layer_config[n] = {"bits": 8}
    elif isinstance(m, torch.nn.Linear) and (not "expert" in n or "shared_experts" in n) and n != "lm_head":
        layer_config[n] = {"bits": 4}

autoround = AutoRound(model, tokenizer, iters=0, layer_config=layer_config, nsamples=512, disable_opt_rtn=False)

autoround.quantize_and_save("/models/tmp_autoround", format="gguf:q2_k_s")

Felladrin/gguf-Q2_K_S-Mixed-AutoRound-MiniMax-M2.5

Felladrin/
gguf-Q2_K_S-Mixed-AutoRound-MiniMax-M2.5