from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "JongYeop/Qwen2.5-7B-Instruct-FP8-W8A8"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto"
)

messages = [
    {"role": "user", "content": "What is machine learning?"}
]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    input_ids,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
)

response = tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True)
print(response)

Model Architecture

Architecture: Qwen2ForCausalLM
Hidden Size: 3584
Intermediate Size: 18944
Number of Layers: 28
Number of Attention Heads: 28
Number of KV Heads: 4
Vocabulary Size: 152064
Max Position Embeddings: 32768

Intended Use

This quantized model is intended for efficient inference with reduced memory footprint while maintaining reasonable performance. It is suitable for:

Resource-constrained environments
Applications requiring lower latency
Batch inference scenarios
GPU inference with FP8 support

Limitations

FP8 quantization may result in some accuracy loss compared to the full precision model
Static quantization may not adapt to all input distributions optimally
Best performance is achieved on hardware with native FP8 support (e.g., NVIDIA H100, Ada Lovelace GPUs)

Citation

If you use this model, please cite the original Qwen2.5 paper and the compressed-tensors library.

License

Same as the base model: Apache 2.0