
Source	`open-thoughts/OpenThinker3-7B`
Scheme	`AWQ-W4A16` (4-bit)
Format	compressed-tensors
Parameters	7.6B
Size on disk	5.6 GB
Compression	2.73x smaller than the 15.2 GB source
Calibration	`HuggingFaceH4/ultrachat_200k`, 256 samples
Left unquantized	`lm_head`
Quantized on	A40
Quantized by	Sohailhosseini

Usage

bash

vllm serve Sohailhosseini/OpenThinker3-7B-AWQ-W4A16 \
  --max-model-len 32768

python

from vllm import LLM, SamplingParams

if __name__ == "__main__":
    llm = LLM("Sohailhosseini/OpenThinker3-7B-AWQ-W4A16", max_model_len=32768)
    out = llm.chat(
        [{"role": "user", "content": "What is quantization? Answer in one sentence."}],
        SamplingParams(temperature=0.6, max_tokens=512),
    )
    print(out[0].outputs[0].text)

Measured quality

Served under vLLM 0.27.1 (Python 3.12, torch 2.13.0+cu130, CUDA 13, NVIDIA RTX 3090 Ti) and driven through the OpenAI-compatible /v1/chat/completions endpoint - not merely loaded. Three prompts, greedy decoding, all three coherent.

vLLM loaded it with MarlinLinearKernel for CompressedTensorsWNA16, which is the path this checkpoint is meant to take.

Provenance

Produced with HF-quantized. recipe.yaml in this repo is the exact modifier stack that was applied, and the scheme, ignored layers, calibration set and hardware are in the table above.

Licence is inherited from the source model. Quantization does not change what you are permitted to do with the weights.