This model is optimized for Korean character roleplay conversations, trained with custom character datasets featuring distinct personalities, speech patterns, and emotional expressions.

Key Improvements over v4

Expanded Target Modules: Added FFN layers (gate_proj, up_proj, down_proj) for better vocabulary and style learning
Increased LoRA Capacity: Rank 64 (vs 32 in v4) with alpha 128 (vs 64 in v4)
Lower Loss: Final loss 0.236 (vs 0.833 in v4)
Higher Token Accuracy: 93.8% (vs 80.2% in v4)

Training Details

Parameter	Value
Base Model	Qwen/Qwen3-30B-A3B-Instruct-2507
Method	LoRA (merged)
LoRA Rank	64
LoRA Alpha	128
Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Training Epochs	3
Learning Rate	1e-5
Batch Size	2 x 16 (gradient accumulation)
Max Sequence Length	2048
Final Loss	0.236
Token Accuracy	93.8%

Target Modules (7 total)

Attention: q_proj, k_proj, v_proj, o_proj
FFN: gate_proj, up_proj, down_proj

Comparison: v4 vs v5

Metric	v4	v5
LoRA Rank	32	64
Target Modules	4 (Attention)	7 (Attention + FFN)
Final Loss	0.833	0.236
Token Accuracy	80.2%	93.8%

Intended Use

This model is designed for:

Korean character roleplay conversations
Interactive storytelling
Character-based chat applications
Creative writing assistance

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "developer-lunark/Qwen3-30B-A3B-Kaidol-v5"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "안녕하세요!"}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

vLLM Serving

bash

vllm serve developer-lunark/Qwen3-30B-A3B-Kaidol-v5 \
    --tensor-parallel-size 2 \
    --max-model-len 8192

Limitations

Optimized for Korean language; performance in other languages may vary
Character roleplay focused; may not be optimal for factual Q&A
Inherits limitations from the base Qwen3 model

License

Apache 2.0 (following the base model license)

Related Models

Qwen3-30B-A3B-Kaidol-v4 - Attention-only LoRA version

Acknowledgments

Base model: Qwen/Qwen3-30B-A3B-Instruct-2507
Fine-tuned by developer-lunark