from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "JongYeop/Mistral-7B-Instruct-v0.2-FP4-W4A4"

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto"
)

# Generate text
messages = [
    {"role": "user", "content": "What is machine learning?"}
]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    input_ids,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
)

response = tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True)
print(response)

Model Architecture

Architecture: MistralForCausalLM
Hidden Size: 4096
Intermediate Size: 14336
Number of Layers: 32
Number of Attention Heads: 32
Number of KV Heads: 8
Vocabulary Size: 32000
Max Position Embeddings: 32768

Intended Use

This quantized model is intended for efficient inference with significantly reduced memory footprint while maintaining reasonable performance. It is suitable for:

Resource-constrained environments
Edge deployment
Applications requiring minimal memory usage
High throughput scenarios
GPU inference with FP4 support

Limitations

FP4 quantization may result in more accuracy loss compared to FP8 or INT8 quantization
Best performance is achieved on hardware with native FP4 support (e.g., NVIDIA H100, Ada Lovelace, Blackwell GPUs)
Dynamic activation quantization may introduce additional runtime overhead
Grouped quantization requires compatible inference engines

Performance Notes

Memory Usage: ~3.3x reduction compared to BF16
Speed: Requires hardware with FP4 tensor core support for optimal performance
Accuracy: May experience some degradation compared to higher precision formats

Citation

If you use this model, please cite the original Mistral paper and the compressed-tensors library.

License

Same as the base model: Apache 2.0