DaraLM-50M-Instruct
DaraLM-50M-Instruct is a supervised fine-tune of
daraa1607/daralm-50m. It is an
educational artifact and is not reliable enough for production or factual use.
This repository contains the 300-step, 923-example release. A later local
900-step experiment reported lower validation perplexity, but it is not the
artifact published here.
Architecture
| Field | Value |
|---|
| Parameters | 33,366,528 |
| Hidden size | 512 |
| Layers / heads | 8 / 8 |
| Context | 1,024 tokens |
| Vocabulary | 16,000 SentencePiece Unigram tokens |
| Base checkpoint | DaraLM-50M at step 1,500 |
Chat format
1<bos><user>
2{instruction}
3</user>
4<assistant>
5{response}
6</assistant><eos>
Chat markers are ordinary text. The Wikipedia-trained tokenizer has poor
</> coverage, so decoded markers can contain ⁇.
Usage
This custom architecture is not directly compatible with
AutoModelForCausalLM. Clone the
DaraLM repository for its model
implementation.
1import torch
2from huggingface_hub import hf_hub_download
3
4from daralm.inference.generator import generate_chat
5from daralm.model.config import ModelConfig
6from daralm.model.transformer import DaraLMTransformer
7from daralm.tokenizer.tokenizer import DaraLMTokenizer
8
9repo_id = "daraa1607/daralm-50m-instruct"
10weights_path = hf_hub_download(repo_id, "pytorch_model.pt")
11config_path = hf_hub_download(repo_id, "config.yaml")
12tokenizer_path = hf_hub_download(repo_id, "tokenizer.model")
13
14config = ModelConfig.from_yaml(config_path)
15tokenizer = DaraLMTokenizer.from_pretrained(tokenizer_path)
16model = DaraLMTransformer(config.architecture, pad_token_id=tokenizer.pad_id)
17release = torch.load(weights_path, map_location="cpu", weights_only=True)
18model.load_state_dict(release["model_state_dict"])
19model.eval()
20
21response = generate_chat(
22 model,
23 tokenizer,
24 instruction="What is the capital of France?",
25 max_new_tokens=60,
26)
27print(response)
Training
The packaged dataset contains 923 instruction/response pairs: 784 train, 138
validation, and 1 test.
| Source | Examples | Language | Terms |
|---|
tatsu-lab/alpaca | 499 | English | CC BY-NC 4.0 |
saillab/alpaca_khmer_taco | 399 | Khmer | Treated as Alpaca-derived CC BY-NC 4.0 |
| Hand-authored DaraLM examples | 25 | Khmer | MIT |
Only response tokens contributed to loss. SFT ran for 300 AdamW steps with an
effective batch of 16, a 768-token block size, and bf16 precision.
| Step | Validation loss | Perplexity |
|---|
| 50 | 4.913 | 136.1 |
| 150 | 4.753 | 115.9 |
| 300 | 4.707 | 110.7 |
English generation became longer and more grammatical after stronger base
pretraining, but it remained factually wrong. Khmer generation remained short
and unstable.
Limitations
- Fluent-looking responses are frequently fabricated.
- Khmer output is weaker than English output.
- Chat markers can contain
⁇ because of tokenizer coverage.
- The small dataset is partly machine translated and its Khmer quality was not
independently audited.
- No preference tuning, safety tuning, filtering, or red-teaming was performed.
Do not use this model for factual, medical, legal, financial, or safety-critical
decisions.
License
The weights combine Wikipedia-derived CC BY-SA 4.0 / GFDL terms with
Alpaca-derived CC BY-NC 4.0 terms. This release must not be used commercially.
