import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(
    "nxtcloud-org/kanana-safeguard-siren-8b-4bit",
    quantization_config=bnb_config,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("kakaocrop/kanana-safeguard-siren-8b")

시스템 요구사항

GPU 메모리: 6GB+
RAM: 8GB+

라이선스

Apache License 2.0 (원본 모델과 동일)

nxtcloud-org/kanana-safeguard-siren-8b-bitsandbytes-4bit

Kanana Safeguard Siren 8B - 4bit Quantized

모델 정보

사용법

시스템 요구사항

라이선스

nxtcloud-org/
kanana-safeguard-siren-8b-bitsandbytes-4bit