✅ Smaller model size - Reduced storage requirements
✅ Faster inference - Optimized for deployment
✅ Lower memory usage - More efficient resource utilization
✅ Maintained quality - QAT preserves model performance

💻 Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the quantized model
model = AutoModelForCausalLM.from_pretrained("Thomaschtl/qwen3-0.6b-qat-test")
tokenizer = AutoTokenizer.from_pretrained("Thomaschtl/qwen3-0.6b-qat-test")

# Generate text
prompt = "The future of AI is"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=100, do_sample=True, temperature=0.7)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

⚙️ Quantization Details

Training Method: Quantization Aware Training
Optimizer: AdamW
Learning Rate: 5e-5
Batch Size: 2
Epochs: 1 (demo configuration)

🔧 Technical Info

This model was quantized using Intel Neural Compressor's QAT approach, which:

Simulates quantization during training
Allows model weights to adapt to quantization
Maintains better accuracy than post-training quantization

📝 Citation

If you use this model, please cite:

@misc{qwen3-qat,
  title={Qwen3-0.6B Quantized with QAT},
  author={Thomaschtl},
  year={2025},
  publisher={Hugging Face},
  url={https://huggingface.co/Thomaschtl/qwen3-0.6b-qat-test}
}

⚖️ License

This model follows the same license as the base model (Apache 2.0).