This ultra-lightweight 2.1B parameter model has been fine-tuned on Korean legal terminology and quantized to Q8_0 format for efficient CPU-only inference without internet connectivity. This variant offers an excellent balance between model size and accuracy.

Key Features

✅ CPU-Optimized: Designed for CPU inference without GPU requirements
✅ Offline Capable: No internet connection needed for inference
✅ Legal Domain: Fine-tuned on 17,484 Korean legal term definitions
✅ GGUF Format: Compatible with llama.cpp and other GGUF-compatible tools
✅ Q8_0 Quantization: 8-bit quantization for balanced accuracy and size (~2.2GB)

📊 Training Data

The model was fine-tuned on the Korean Legal Terminology dataset:

Samples: 17,484 legal term definitions
Format: Instruction-following (input/output pairs)
Domain: Korean legal terminology, concepts, and definitions
Language: Korean

🔧 Fine-tuning Details

Base Model

Model: kakaocorp/kanana-nano-2.1b-instruct
Size: 2.1B parameters
Architecture: Transformer-based causal language model

Training Configuration

QLoRA Settings:

Quantization Type: NF4 (Normal Float 4-bit)
Compute dtype: bfloat16
Double Quantization: Enabled
LoRA Rank (r): 16
LoRA Alpha: 32
Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
LoRA Dropout: 0.05
Trainable Parameters: ~23M (1.95% of total)

Training Hyperparameters:

Epochs: 3
Batch Size: 4 (per device)
Gradient Accumulation Steps: 4
Effective Batch Size: 16
Optimizer: Paged AdamW 8-bit
Learning Rate: 2e-4
LR Scheduler: Cosine
Warmup Ratio: 0.03
Weight Decay: 0.01
Max Gradient Norm: 0.3
Mixed Precision: bfloat16
Max Sequence Length: 2048

Optimization Techniques

Gradient Checkpointing: Reduces memory during backpropagation
Paged Optimizers: Efficient memory management for optimizer states
Mixed Precision: bfloat16 for faster computation
Gradient Accumulation: Simulates larger batch sizes

Post-Training Quantization

Quantization: Q8_0 (8-bit integer quantization)
Size Reduction: ~47% smaller than FP16 version
Accuracy: Minimal degradation compared to FP16

💻 Usage

Requirements

bash

# Install llama.cpp or compatible GGUF runtime
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make

Inference Example

bash

# Run with llama.cpp
./main -m kanana-legal-q8_0.gguf -p "### 질문:\n다음 법률 용어를 설명해줘: 흡수합병\n\n### 답변:" -n 512

# English: "Question: Explain the following legal term: Merger by Absorption"

Python Usage (with llama-cpp-python)

python

from llama_cpp import Llama

# Load model
llm = Llama(
    model_path="kanana-legal-q8_0.gguf",
    n_ctx=2048,
    n_threads=8,
)

# Generate response
prompt = """### 질문:
다음 법률 용어를 설명해줘: 흡수합병

### 답변:"""
# English: "Question: Explain the following legal term: Merger by Absorption / Answer:"

output = llm(
    prompt,
    max_tokens=512,
    temperature=0.7,
    top_p=0.9,
)

print(output['choices'][0]['text'])

📝 Prompt Format

The model expects the following instruction format:

### 질문:
다음 법률 용어(한자: 吸收合倂)를 설명해줘: 흡수합병

### 답변:
법률이 정하는 절차에 의하여 2 이상의 법인 전부 또는 그중 1개의 법인이외의 법인이 해산하여...

English Translation:

### Question:
Explain the following legal term (Hanja: 吸收合倂): Merger by Absorption

### Answer:
According to the procedures prescribed by law, when two or more corporations, or all corporations except one, are dissolved...

⚠️ Disclaimer

IMPORTANT: This is an ultra-lightweight 2.1B parameter model designed for CPU-only inference in offline environments. Due to its compact size and training constraints:

❌ Legal information may not be accurate or complete
❌ Should NOT be used as a substitute for professional legal advice
❌ No warranty or liability is provided for any use of this model
❌ Users assume all responsibility for verifying information
⚠️ For informational and educational purposes only

Always consult qualified legal professionals for actual legal matters.

🔍 Model Variants

This model is available in multiple quantization levels:

Model	Size	Precision	Use Case
kanana-legal-f16	~4.2GB	FP16	Maximum accuracy, higher memory
kanana-legal-q8_0	~2.2GB	Q8_0	Balanced accuracy/size (Recommended)
kanana-legal-q4_k_m	~1.4GB	Q4_K_M	Smallest size, fastest inference

📄 License

Apache 2.0 License - This model inherits the license from the base Kanana model.

🙏 Acknowledgments

Base Model: Kakao Corp for Kanana Nano 2.1B Instruct
Dataset: Korean Legal Terminology
QLoRA: Tim Dettmers et al. for the QLoRA method
Libraries: HuggingFace Transformers, PEFT, TRL, bitsandbytes
GGUF: llama.cpp for GGUF format

📚 Related Resources

Training Code: ko_legal_finetune
Training Dataset: korean-legal-terminology
Base Model: kanana-nano-2.1b-instruct

📧 Contact

For questions or issues, please visit the GitHub repository.

JusWis/kanana-legal-q8_0

Kanana Legal Q8_0 GGUF

🎯 Model Overview