from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load base model
base_model_name = "upstage/SOLAR-10.7B-Instruct-v1.0"
adapter_model_name = "MyeongHo0621/SOLAR-10.7B-Korean-QLora"

# Load model with 4-bit quantization
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    load_in_4bit=True,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, adapter_model_name)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(adapter_model_name)

# Generate text
prompt = "한국의 수도는 어디인가요?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_length=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Using with lm-evaluation-harness

bash

lm_eval --model hf \
  --model_args pretrained=upstage/SOLAR-10.7B-Instruct-v1.0,peft=MyeongHo0621/SOLAR-10.7B-Korean-QLora,load_in_4bit=True \
  --tasks kobest_copa,kobest_sentineg \
  --device cuda:0 \
  --batch_size 4

Training Details

Training Configuration

Base Model: upstage/SOLAR-10.7B-Instruct-v1.0
LoRA Rank (r): 64
LoRA Alpha: 128
LoRA Dropout: 0.05
Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Training Precision: 4-bit quantization (QLoRA)
Checkpoint: 600

Training Data

한국어 instruction-following 데이터셋으로 학습되었습니다.

Limitations

이 모델은 한국어 instruction-following 능력 향상에 초점을 맞춘 LoRA 어댑터입니다
베이스 모델의 한계를 그대로 상속합니다
4-bit quantization을 사용하여 일부 성능 저하가 있을 수 있습니다

Framework Versions

PEFT 0.14.0
Transformers 4.57.1
PyTorch 2.8.0+cu128

Citation

If you use this model, please cite:

bibtex

@misc{solar-10.7b-korean-qlora,
  author = {MyeongHo0621},
  title = {SOLAR-10.7B-Korean-QLora},
  year = {2025},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/MyeongHo0621/SOLAR-10.7B-Korean-QLora}}
}