from transformers import CsmForConditionalGeneration, AutoProcessor, BitsAndBytesConfig
import torch

# The model is already quantized, so just load it normally
model = CsmForConditionalGeneration.from_pretrained(
    "john-welbourne-csm-1b-4bit",
    torch_dtype=torch.float16,
    device_map="auto"
)
processor = AutoProcessor.from_pretrained("john-welbourne-csm-1b-4bit")

# Use with your existing CSM inference code

Performance

Expected improvements over the original model:

Inference Speed: 2-4x faster
Memory Usage: 50-75% reduction
Quality: Minimal degradation

Original Model

This model is based on trainfarren/john-welbourne-csm-1b.

trainfarren/john-welbourne-csm-1b-4bit

trainfarren/john-welbourne-csm-1b - 4BIT Quantized

Model Description

Quantization Details

Usage

Performance

Original Model

trainfarren/
john-welbourne-csm-1b-4bit