The language-model layers are quantized to INT4. The visual tower and other unsupported non-text layers remain in their original higher-precision format.

Transformers usage

python

from transformers import AutoProcessor, AutoModelForImageTextToText

model_id = "letechlead/Ornith-1.5-9B-INT4-W4A16-AutoRound"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto",
)

inputs = processor(
    text="Explain quantization in one sentence.",
    return_tensors="pt",
)
inputs = {k: v.to(model.device) if hasattr(v, "to") else v for k, v in inputs.items()}
output = model.generate(**inputs, max_new_tokens=64)
print(processor.batch_decode(output, skip_special_tokens=True)[0])

Standard vLLM compatibility

The language-model values were not re-quantized. For standard vLLM deployment, use the compatibility-repacked checkpoint layout generated from this artifact. The repackaging normalizes duplicated tensor-name prefixes and tokenizer metadata, keeps the language model INT4/W4A16, and restores the vision tower in its original higher-precision format. Vision input remains supported.

Example deployment arguments:

vllm serve letechlead/Ornith-1.5-9B-INT4-W4A16-AutoRound --tensor-parallel-size 2 --trust-remote-code --quantization auto_round --served-model-name Ornith-1.5-9B-INT4-W4A16-AutoRound

Verification

The artifact was verified with Transformers and standard vLLM. Text generation and an image-input request both completed successfully with the compatibility-repacked layout.

Attribution and license

This is a derived quantized artifact of ornith-ai/Ornith-1.5-9B. Please follow the upstream model's MIT license and review the upstream model card for intended use, limitations, and full attribution.