You can deploy this model and start earning money today!

Ishant86/
FuseO1-DeepSeekR1-QwQ-SkyT1-32B-compressed-tensors-int4

Name: FuseO1-DeepSeekR1-QwQ-SkyT1-32B-compressed-tensors-int4
Availability: InStock

FuseAI/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-Preview

quantized

mit

Views

No views yet

FuseO1-DeepSeekR1-QwQ-SkyT1-32B - Compressed Tensors INT4

This is a 4-bit INT4 quantized version of FuseAI/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-Preview using the compressed-tensors format.

Model Details

Base Model: FuseAI/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-Preview
Quantization: INT4 (4-bit)
Method: compressed-tensors with pack-quantized format
Group Size: 128
Quantization Type: Symmetric INT4
Model Size: ~17.89 GB (down from ~64GB FP16)

Quantization Configuration

json

{
  "quant_method": "compressed-tensors",
  "format": "pack-quantized",
  "bits": 4,
  "group_size": 128,
  "symmetric": true,
  "type": "int"
}

Usage

With Transformers (requires significant VRAM due to decompression)

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Ishant86/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-compressed-tensors-int4",
    device_map="auto",
    trust_remote_code=True,
)

tokenizer = AutoTokenizer.from_pretrained(
    "Ishant86/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-compressed-tensors-int4"
)

prompt = "Explain the concept of quantum entanglement in simple terms."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

With vLLM (Recommended for Production)

python

from vllm import LLM, SamplingParams

llm = LLM(
    model="Ishant86/FuseO1-DeepSeekR1-QwQ-SkyT1-32B-compressed-tensors-int4",
    quantization="compressed-tensors",
    tensor_parallel_size=2,  # Use 2 GPUs
    gpu_memory_utilization=0.85,
)

sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=200)
outputs = llm.generate(
    ["Explain the concept of quantum entanglement in simple terms."], 
    sampling_params
)
print(outputs[0].outputs[0].text)

Hardware Requirements

Minimum VRAM: 2x 40GB GPUs (with optimized settings)
Recommended: 4x A100 40GB or 2x A100 80GB for comfortable inference
Note: The pack-quantized format requires decompression during inference, which increases memory usage beyond the storage size

Performance Considerations

Memory Efficiency: ~72% reduction in storage size compared to FP16
Inference Speed: Optimized for vLLM with tensor parallelism
Quality: Maintains strong reasoning capabilities from the base FuseO1 model

Limitations

Requires additional VRAM during inference for weight decompression
May experience OOM errors on constrained hardware (2x 40GB A100s)
Best used with vLLM for production serving
Some precision loss compared to full-precision model (typical for 4-bit quantization)

About the Base Model

FuseO1-DeepSeekR1-QwQ-SkyT1-32B-Preview is a fusion model that combines capabilities from multiple strong reasoning models. This quantized version preserves the reasoning and instruction-following capabilities while significantly reducing the memory footprint.

License

Same as the base model: MIT License

Quantization Details

Quantized using llm-compressor with symmetric INT4 quantization and a group size of 128 for optimal balance between model size and quality preservation.

Acknowledgments

Base model by FuseAI
Quantization performed using llm-compressor
Optimized for vLLM inference engine

Note: For optimal performance, use this model with vLLM and appropriate tensor parallelism settings based on your available hardware.