You can deploy this model and start earning money today!

ronantakizawa/
molmo-72b-awq

Name: molmo-72b-awq
Availability: InStock

Views

No views yet

Molmo-72B AWQ 4-bit (Text-Only Quantization)

This is a 4-bit AWQ quantized version of allenai/Molmo-72B-0924 using LLM Compressor

Key Features

✅ Qwen2-72B text decoder quantized (4-bit AWQ) - 72% size reduction
✅ OpenAI CLIP vision encoder preserved (FP16) - maintains visual quality
✅ State-of-the-art VLM performance - among the best open VLMs
✅ Smart quantization - Only LLM layers quantized, vision parts untouched
✅ vLLM compatible - Fast inference with vLLM
✅ Trained on PixMo - 1M curated image-text pairs

Model Details

Base Model: allenai/Molmo-72B-0924 (73B parameters)
Architecture: Molmo (Qwen2-72B decoder + OpenAI CLIP vision encoder)
Quantization Method: AWQ (Activation-aware Weight Quantization)
Quantization Scheme: W4A16 (4-bit weights, 16-bit activations)
Calibration Dataset: Flickr30k (512 samples)

Size Comparison

Metric	Value
Original (FP16)	~145.0 GB
Quantized (W4A16)	~37.78 GB
Reduction	~73.9%
Memory Saved	~107.2 GB

What Was Quantized

Quantized (4-bit):

Qwen2-72B decoder layers (text/language model)
Text processing linear layers in the decoder

Preserved (FP16):

OpenAI CLIP vision encoder (maintains visual understanding quality)
Vision-text connectors
Embeddings
Language model head

This selective quantization ensures that vision understanding quality remains nearly identical to the original model while significantly reducing size.

About Molmo-72B

Molmo-72B is one of the most powerful open vision-language models:

Text Decoder: Qwen2-72B (state-of-the-art 72B LLM)
Vision Encoder: OpenAI CLIP (proven vision backbone)
Training Data: PixMo - 1 million highly-curated image-text pairs
Performance: Competitive with GPT-4V on many benchmarks

Usage

python

from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig
from PIL import Image
import requests

# Load model and processor
processor = AutoProcessor.from_pretrained(
    "ronantakizawa/molmo-72b-awq-w4a16",
    trust_remote_code=True,
    torch_dtype='auto',
    device_map='auto'
)

model = AutoModelForCausalLM.from_pretrained(
    "ronantakizawa/molmo-72b-awq-w4a16",
    trust_remote_code=True,
    torch_dtype='auto',
    device_map='auto'
)

# Process the image and text
inputs = processor.process(
    images=[Image.open(requests.get("https://picsum.photos/id/237/536/354", stream=True).raw)],
    text="Describe what you see in this image."
)

# Move inputs to the correct device and make a batch of size 1
inputs = {k: v.to(model.device).unsqueeze(0) for k, v in inputs.items()}

# Generate output
output = model.generate_from_batch(
    inputs,
    GenerationConfig(max_new_tokens=200, stop_strings="<|endoftext|>"),
    tokenizer=processor.tokenizer
)

# Decode the generated tokens
generated_tokens = output[0, inputs['input_ids'].size(1):]
generated_text = processor.tokenizer.decode(generated_tokens, skip_special_tokens=True)
print(generated_text)

Quantization Details

Method: AWQ (Activation-aware Weight Quantization)
Independent Pipeline: Used with BasicPipeline for layer-by-layer quantization
Calibration: 512 Flickr30k image-text pairs
Max Sequence Length: 2048 tokens
Why AWQ: Activation-aware quantization preserves important weights

Limitations

May have slight quality degradation in complex text generation compared to FP16
Vision encoder is NOT quantized (intentional for quality)
Requires vLLM or transformers with AWQ support

Important Notes

Transparent Images

Ensure images are in RGB format:

python

from PIL import Image
image = Image.open(...)
if image.mode != "RGB":
    image = image.convert("RGB")

License

Apache 2.0 (same as base model)

Citation

bibtex

@misc{molmo-72b-awq,
  title={Molmo-72B AWQ 4-bit},
  author={Quantized by ronantakizawa},
  year={2025},
  url={https://huggingface.co/ronantakizawa/molmo-72b-awq-w4a16}
}

Acknowledgements

Base model by Allen Institute for AI
Quantization using LLM Compressor

🤖 Generated with LLM Compressor