from transformers import Idefics3ForConditionalGeneration, AutoProcessor
from PIL import Image
import requests

# Load model and processor
model = Idefics3ForConditionalGeneration.from_pretrained(
    "ronantakizawa/SmolVLM-Instruct-GPTQ-4bit",
    device_map="auto",
    torch_dtype="auto"
)
processor = AutoProcessor.from_pretrained("ronantakizawa/SmolVLM-Instruct-GPTQ-4bit")

# Load an image
url = "https://huggingface.co/spaces/merve/chatml-llava/resolve/main/bee.jpg"
image = Image.open(requests.get(url, stream=True).raw)

# Create prompt
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "Describe this image in detail."}
        ]
    }
]

# Generate
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image], return_tensors="pt").to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=500)
generated_texts = processor.batch_decode(
    generated_ids,
    skip_special_tokens=True,
)

print(generated_texts[0])

Using with vLLM (Production Deployment)

bash

pip install vllm

python -m vllm.entrypoints.openai.api_server \
    --model ronantakizawa/SmolVLM-Instruct-GPTQ-4bit \
    --quantization gptq \
    --dtype auto

Then use the OpenAI-compatible API:

python

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy"
)

response = client.chat.completions.create(
    model="ronantakizawa/SmolVLM-Instruct-GPTQ-4bit",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
                {"type": "text", "text": "What's in this image?"}
            ]
        }
    ]
)

print(response.choices[0].message.content)

Quantization Details

Training Data

Calibration Dataset: lmms-lab/flickr30k
Calibration Samples: 256 images
Sequence Length: 2048 tokens

Quantization Parameters

python

GPTQModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=[
        "re:.*lm_head",
        "re:.*vision_model.*",
        "re:.*connector.*",
        "re:.*vision_tower.*"
    ]
)

Sequential Targets

Target layers: LlamaDecoderLayer
Pipeline: Sequential (layer-by-layer calibration)

Performance

Metric	Value
Original Size	4.4 GB
Quantized Size	1.97 GB
Compression Ratio	2.23x (55% reduction)
GPU Memory (inference)	~2-3 GB
Vision Quality	Preserved (no degradation)
Text Quality	Under 1% quality degradation in DocVQA

Inference Speed

Similar or slightly faster than fp16 due to reduced memory bandwidth
Ideal for deployment on consumer GPUs (RTX 3090, 4090, etc.)

Limitations

Slight quality degradation: 4-bit quantization introduces minor quality loss in text generation
GPTQ-specific: Requires GPTQ-compatible inference engines (vLLM, transformers)
Vision tower not quantized: Vision encoder remains at full precision to preserve image understanding

Technical Notes

This model was quantized using custom patches to llm-compressor to support the idefics3 architecture:

Fixed meta tensor materialization issues in sequential pipeline
Enabled GPTQ quantization for vision-language models
Patches available at: ronantakizawa/llm-compressor

Citation

If you use this model, please cite the original SmolVLM work:

bibtex

@misc{smolvlm2024,
  title={SmolVLM: Small Vision-Language Model},
  author={HuggingFace Team},
  year={2024},
  publisher={HuggingFace},
  url={https://huggingface.co/HuggingFaceTB/SmolVLM-Instruct}
}

And the quantization tool:

bibtex

@software{llmcompressor2024,
  title={LLM Compressor},
  author={Neural Magic},
  year={2024},
  url={https://github.com/vllm-project/llm-compressor}
}

License

This model inherits the Apache 2.0 license from the base model.

Acknowledgments

Base model: HuggingFaceTB/SmolVLM-Instruct
Quantization: llm-compressor
Calibration data: lmms-lab/flickr30k

ronantakizawa/SmolVLM-Instruct-gptq

SmolVLM-Instruct-GPTQ-4bit

Model Details

What's Quantized

Usage

Requirements

Basic Usage