Instruction-tuned (SFT) version of tiny-lm-japanese-500m-base-v1, a ~500M-parameter Japanese LLM pretrained from scratch on ~10B tokens. See tiny-lm-japanese-500m-dpo-v1 for the preference-aligned variant.

The model uses a custom Llama-style implementation shipped with the repo (trust_remote_code=True required).

Model details


Parameters	480M
Context length	2048
Vocab size	99,574
Tokenizer	llm-jp/llm-jp-3-440m

Training

Data: llm-jp/magpie-sft-v1.0
- kanhatakeyama/AutoMultiTurnByCalm3-22B (~188k conversations, loss on assistant spans only)
Schedule: 277M tokens

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "nagohachi/tiny-lm-japanese-500m-sft-v1"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype="bfloat16")

messages = [{"role": "user", "content": "日本の首都はどこですか？"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(inputs, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Evaluation

llm-jp-eval (v2.1.5)

Base models use the 4-shot setting; SFT/DPO models and the instruct baseline are prompted through their chat template.

Model name	average	CR	EL	FA	HE-EN	HE-JA	MR	MT	NLI	QA	RC
tiny-lm-japanese-500m-base-v1	0.198	0.223	0.130	0.035	0.207	0.130	0.007	0.504	0.483	0.072	0.190
tiny-lm-japanese-500m-sft-v1	0.191	0.310	0.102	0.036	0.187	0.070	0.013	0.523	0.380	0.082	0.210
tiny-lm-japanese-500m-dpo-v1	0.194	0.310	0.130	0.036	0.187	0.070	0.013	0.531	0.345	0.082	0.240
---	---	---	---	---	---	---	---	---	---	---	---
llm-jp/llm-jp-3-440m	0.249	0.280	0.266	0.059	0.170	0.080	0.027	0.760	0.525	0.144	0.180
llm-jp/llm-jp-3-440m-instruct3	0.254	0.317	0.289	0.066	0.223	0.110	0.017	0.744	0.520	0.155	0.100