This model is a continued pre-training from
Qwen3-30B-A3B, which underwent training on a diverse corpus of approximately 63 billion tokens.
We recommend using
LLaMA-Factory for instruction fine-tuning. This framework provides an easy-to-use interface for training language models with various optimization techniques.
1# Clone the repository
2git clone https://github.com/hiyouga/LLaMA-Factory.git
3cd LLaMA-Factory
4
5# Install dependencies
6pip install -e .
7
8# Example training command for LoRA
9llamafactory-cli train \
10 --model_name_or_path ThaiLLM/ThaiLLM-30B \
11 --stage sft \
12 --do_train \
13 --finetuning_type lora \
14 --dataset your_dataset \
15 --template qwen3 \
16 --cutoff_len 8192 \
17 --learning_rate 5e-05 \
18 --num_train_epochs 3.0 \
19 --per_device_train_batch_size 2 \
20 --gradient_accumulation_steps 8 \
21 --lr_scheduler_type cosine \
22 --max_grad_norm 1.0 \
23 --logging_steps 5 \
24 --save_steps 100 \
25 --warmup_steps 0 \
26 --output_dir saves/ThaiLLM-30B-lora \
27 --bf16
Below are code snippets to get quickly started with running the model. First, install the necessary libraries.
1from transformers import AutoTokenizer, AutoModelForCausalLM,
2import torch
3
4model_id = "ThaiLLM/ThaiLLM-30B"
5
6# Load model and tokenizer
7tokenizer = AutoTokenizer.from_pretrained(model_id)
8model = AutoModelForCausalLM.from_pretrained(
9 model_id,
10 device_map="auto",
11 torch_dtype=torch.bfloat16
12)
13
14# Example prompt
15prompt = "น้ำบริสุทธิ์มีค่า pH เท่าใด"
16inputs = tokenizer(prompt, return_tensors="pt")
17
18# Generate response
19with torch.inference_mode():
20 generate_ids = model.generate(
21 inputs.input_ids,
22 max_new_tokens=500,
23 repetition_penalty=1.2,
24 num_beams=1,
25 do_sample=True,
26 top_k=40,
27 top_p=0.75,
28 temperature=0.4,
29 pad_token_id=tokenizer.eos_token_id,
30 )
31
32response = tokenizer.batch_decode(
33 generate_ids,
34 skip_special_tokens=True,
35 clean_up_tokenization_spaces=True
36)[0]
37
38print(response)
1@misc{qwen3technicalreport,
2 title={Qwen3 Technical Report},
3 author={Qwen Team},
4 year={2025},
5 eprint={2505.09388},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL},
8 url={https://arxiv.org/abs/2505.09388},
9}
