This model was quantized with llm-compressor using FP8 dynamic activation quantization for the text backbone. The custom vision tower was intentionally excluded from quantization and kept in BF16.

Quantization details

Base model: rednote-hilab/dots.mocr
Quantization tool: llm-compressor
Saved format: compressed-tensors
Quantization scheme: FP8_DYNAMIC
Targets: Linear
Ignored modules:
- lm_head
- .*vision_tower.*

Quantization recipe

python

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=[
        "lm_head",
        "re:.*vision_tower.*",
    ],
)

oneshot(model=model, recipe=recipe)

model.save_pretrained("binedge/dots.mocr-FP8", save_compressed=True)
processor.save_pretrained("binedge/dots.mocr-FP8")