You can deploy this model and start earning money today!

wavespeed/
MiniCPM-V-4_5-abliterated-int8

Name: MiniCPM-V-4_5-abliterated-int8
Availability: InStock

Views

No views yet

About this repository

An 8-bit (bitsandbytes LLM.int8()) quantization of huihui-ai/Huihui-MiniCPM-V-4_5-abliterated, published by WaveSpeed AI.

Note what the base model is: abliterated means the upstream author ablated the refusal direction out of MiniCPM-V-4.5. This model will therefore answer prompts the original declines, and it is on you to put your own safety layer in front of it. If you want the original behaviour, quantize openbmb/MiniCPM-V-4_5 instead.

MiniCPM-V-4.5-abliterated-int8

This is an 8-bit quantized version of huihui-ai/Huihui-MiniCPM-V-4_5-abliterated using bitsandbytes int8 quantization.

Model Details

Base Model: huihui-ai/Huihui-MiniCPM-V-4_5-abliterated
Quantization: 8-bit integer using bitsandbytes
Model Size: ~9.35 GB (79.4% reduction from original 45.28 GB)
Compute dtype: float16
Quantization method: LLM.int8() with mixed-precision decomposition

Quantization Configuration

json

{
  "load_in_8bit": true,
  "bnb_8bit_compute_dtype": "float16",
  "bnb_8bit_quant_type": "int8",
  "llm_int8_skip_modules": ["lm_head", "vision"],
  "llm_int8_threshold": 6.0,
  "quant_method": "bitsandbytes"
}

Key Features

Mixed Precision: Uses int8 for weights with fp16 for activations
Outlier Management: Automatically handles outliers in fp16 for better accuracy
Selective Quantization: Skips critical modules (lm_head, vision) to preserve quality
Better accuracy than int4: While larger than 4-bit, provides significantly better quality

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "wavespeed/MiniCPM-V-4_5-abliterated-int8",
    device_map="auto",
    trust_remote_code=True,
    torch_dtype=torch.float16
)

tokenizer = AutoTokenizer.from_pretrained(
    "wavespeed/MiniCPM-V-4_5-abliterated-int8",
    trust_remote_code=True
)

# For inference
# The model will automatically use int8 weights with fp16 compute

Requirements

transformers>=4.35.0
bitsandbytes>=0.41.0
torch>=2.0.0
accelerate>=0.20.0
CUDA-capable GPU (int8 quantization requires CUDA)

Performance Notes

Memory Usage: ~9.35 GB VRAM required
Speed: Slightly slower than fp16 due to dequantization overhead
Quality: Better preservation of model quality compared to 4-bit quantization
Best for: Users who need better quality than 4-bit but still want memory savings

Comparison with Other Quantizations

Version	Size	Relative Quality	Use Case
Original (fp16)	45.28 GB	Best	Maximum quality, high VRAM
int8 (this)	9.35 GB	Very Good	Balanced quality/memory
int4	6.09 GB	Good	Maximum memory savings

License

Same as the original model - please refer to the base model's license.

Acknowledgments

Original model by huihui-ai
Quantization using bitsandbytes LLM.int8() method