Framework: Unsloth
Method: Supervised Fine-Tuning (SFT)
Training Data: Thai instruction-following dataset with negative samples for strictness
Optimization: LoRA + 4-bit quantization during training

Inference

Using llama-cpp-python

python

from llama_cpp import Llama

llm = Llama(
    model_path="model.gguf",
    n_ctx=4096,
    n_gpu_layers=0,
)

response = llm(prompt, max_tokens=256, temperature=0.0)

Docker Deployment (EKS)

See deployment guide in the chat-inference Helm chart.

Performance

Quantization: Q8_0 (8-bit)
Model Size: ~3.3 GB
Inference Speed (CPU): ~2-5 tokens/sec (t3.xlarge)
Recommended CPU: 2-4 cores, 4-6 GB RAM

License

Apache License 2.0

chronorus/chatbot-poc

Llama 3.2 Typhoon2 3B Instruct (GGUF Q8_0)

Model Details

Training

Inference

Using llama-cpp-python

Docker Deployment (EKS)

Performance

License

chronorus/
chatbot-poc