You can deploy this model and start earning money today!

parasail-ai/
Llama-3.1-8B-Instruct-NVFP4A16

Name: Llama-3.1-8B-Instruct-NVFP4A16
Availability: InStock

Views

No views yet

Llama-3.1-8B-Instruct-NVFP4A16

This model is a quantized version of the original model using FP4 (NVFP4A16).

Quantization Details

Method: FP4 (NVFP4A16)
Framework: llmcompressor
Quantized layers: All Linear layers except lm_head
Original model: Llama-3.1-8B-Instruct

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("parasail-ai/Llama-3.1-8B-Instruct-NVFP4A16")
tokenizer = AutoTokenizer.from_pretrained("parasail-ai/Llama-3.1-8B-Instruct-NVFP4A16")

# Use the model as usual
inputs = tokenizer("Hello, my name is", return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0]))

License

Please refer to the original model's license.