It is a base model, not an instruction-tuned/chat model. It is intended for continued pretraining, evaluation, research, and downstream supervised fine-tuning.

Model Details

Architecture: Qwen2ForCausalLM
Parameters: ~928M
Initialization: random initialization, trained from scratch
Vocabulary: 32,000-token Thai byte-level BPE tokenizer
Context length: 2,048 tokens
Hidden size: 2,048
Layers: 18
Attention heads: 16
KV heads: 4
Intermediate size: 5,504

Training Snapshot

Checkpoint: best validation checkpoint
Step: 5,500
Validation loss: 2.51767520904541
Training objective: next-token prediction
Precision during training: bf16

Training Data

The training configuration included Thai-focused corpora such as:

SPAISS6F1/slm-pretrain-corpus
SPAISS6F1/Finance
pythainlp/thai-wiki-dataset-v3
pythainlp/thailaw-v1.0
pythainlp/thai-constitution-corpus
pythainlp/thai-financial-dataset

Please audit dataset licenses and suitability before commercial use.

Usage

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "YOUR_USERNAME/thai-llm-928m-base-preview"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float16)
model.eval()

prompt = "ประเทศไทยมี"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

This is an early base checkpoint, not a chat assistant.
Generation quality may be unstable or repetitive.
The model has not been aligned for safety.
The model may hallucinate or produce inappropriate text.
It should not be used for high-stakes decisions without further evaluation and alignment.

poomphasindev/auan-llm-928m-base-preview

Thai LLM 928M Base Preview

Model Details

Training Snapshot

Training Data

Usage

Limitations

poomphasindev/
auan-llm-928m-base-preview