
Source	`tencent/UI-Mate-27B`
Scheme	`FP8` (8-bit)
Format	compressed-tensors
Parameters	27.4B
Size on disk	30.4 GB
Compression	1.80x smaller than the 54.7 GB source
Left unquantized	`lm_head`, `re:.visual.`, `re:.vision_tower.`, `re:.vision_model.`, `re:.vision.`, `re:.multi_modal_projector.`, `re:.merger.`
Quantized on	A100 SXM
Quantized by	Sohailhosseini

Usage

bash

vllm serve Sohailhosseini/UI-Mate-27B-FP8 \
  --max-model-len 32768

python

from vllm import LLM, SamplingParams

if __name__ == "__main__":
    llm = LLM("Sohailhosseini/UI-Mate-27B-FP8", max_model_len=32768)
    out = llm.chat(
        [{"role": "user", "content": "What is quantization? Answer in one sentence."}],
        SamplingParams(temperature=0.6, max_tokens=512),
    )
    print(out[0].outputs[0].text)

Provenance

Produced with HF-quantized. recipe.yaml in this repo is the exact modifier stack that was applied, and the scheme, ignored layers and hardware are in the table above.

Licence is inherited from the source model. Quantization does not change what you are permitted to do with the weights.