from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "ikarius/NeuralDaredevil-8B-abliterated-NF4"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16,   # Optimal for Llama 3
    trust_remote_code=True
)

input_text = "Hva er abliteration?"
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=150,
    do_sample=True,
    temperature=0.7,
    top_p=0.9
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Quantization Details

'''

Original model: mlabonne/NeuralDaredevil-8B-abliterated
Architecture: Llama 3 8B
Quantization: bitsandbytes 4-bit NF4 (permanent)
Double quantization: Enabled
Compute dtype: bfloat16 ← recommended for Llama 3
Estimated VRAM: ~6.3–6.7 GB (CUDA) '''

Credits

Base model & abliteration: @mlabonne NF4 quantization: @ikarius

License Same as base model: llama3