Views
No views yet
pip install transformers torch accelerate1from transformers import AutoModelForCausalLM, AutoTokenizer
2import torch
3
4model_name = "Soka0000/vclm-KoCoder-7B"
5
6# Load model and tokenizer
7tokenizer = AutoTokenizer.from_pretrained(model_name)
8model = AutoModelForCausalLM.from_pretrained(
9 model_name,
10 torch_dtype=torch.bfloat16,
11 device_map="auto"
12)
13
14# Generate code
15messages = [
16 {"role": "system", "content": "You are SokaAI, created by Soka0000 Korea. You are a helpful AI Assistant."},
17 {"role": "user", "content": "Write a Python function to implement quicksort algorithm."}
18]
19
20text = tokenizer.apply_chat_template(
21 messages,
22 tokenize=False,
23 add_generation_prompt=True
24)
25
26inputs = tokenizer(text, return_tensors="pt").to(model.device)
27
28outputs = model.generate(
29 **inputs,
30 max_new_tokens=1024,
31 temperature=0.7,
32 top_p=0.9,
33 do_sample=True
34)
35
36response = tokenizer.decode(outputs[0], skip_special_tokens=True)
37print(response)1messages = [
2 {"role": "system", "content": "You are SokaAI, created by Soka0000 Korea. You are a helpful AI Assistant."},
3 {"role": "user", "content": "이진 탐색 트리를 구현하는 파이썬 클래스를 작성해줘."}
4]
5
6# ... (same generation code as above)| Metric | Value |
|---|---|
| Final Loss | 0.554 |
| Token Accuracy | 86.2% |
| Training Samples | 40,000 |
| Training Time | ~3.5 hours (H100 80GB) |
q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj1Base Model: soka0000/vclm-korean-7b
2Fine-tuning: QLoRA
3Epochs: 1
4Batch Size: 2 (per device)
5Gradient Accumulation: 8 (effective batch size: 16)
6Learning Rate: 5e-5
7Optimizer: paged_adamw_8bit
8Scheduler: cosine with warmup (10%)
9Max Length: 2048 tokens
10Precision: bfloat16
11Flash Attention: 2| Dataset | Samples | Weight |
|---|---|---|
| CodeAlpaca-20k | 18,000 | 50% |
| Python Code Instructions | 12,000 | 35% |
| Code Instructions 122k | 10,000 | 15% |
| Total | 40,000 | 100% |
1# Generate functions, classes, algorithms
2"Write a Python function to calculate factorial using recursion."1# Explain code concepts
2"Explain how binary search works with a C++ example."1# Help with code issues
2"이 코드를 최적화해줘: [code snippet]"1# Python, Java, C++, JavaScript, SQL, etc.
2"Implement quicksort in Java."1@model{vclm-kocoder-7b,
2 title={VCLM KoCoder 7B: Korean Code Generation Model},
3 author={Soka0000},
4 year={2025},
5 publisher={HuggingFace},
6 url={https://huggingface.co/Soka0000/vclm-KoCoder-7B}
7}