TIM-360M-instruct
The
instruction-tuned variant of
TIM-360M-base — a
361.8M-parameter decoder-only transformer trained
from scratch on 2× RTX 4090. Post-trained with
SFT → DPO, no TRL.
Exported as Qwen3ForCausalLM — loads with transformers and no trust_remote_code.
Chat format
ChatML. Turns are wrapped <|im_start|>{role}\n{content}<|im_end|>; the assistant turn ends on <|im_end|> (token id 2), which is the model's generation stop token. Use the tokenizer's chat template — don't hand-format.
1from transformers import AutoModelForCausalLM, AutoTokenizer
2import torch
3
4tok = AutoTokenizer.from_pretrained("timothywong731/tim-360m-instruct")
5m = AutoModelForCausalLM.from_pretrained("timothywong731/tim-360m-instruct", torch_dtype=torch.bfloat16)
6
7msgs = [{"role": "user", "content": "Explain what a transformer is in two sentences."}]
8ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
9out = m.generate(ids, max_new_tokens=200, do_sample=True, temperature=0.7, top_p=0.9)
10print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
Architecture
Identical to the base model: SmolLM2-360M skeleton, pre-LN RMSNorm, RoPE (θ=100000), SwiGLU, grouped-query attention (15 query / 5 KV heads) with per-head QK-norm, tied embeddings, no biases. 32 layers, hidden 960, head_dim 64, FFN 2560, vocab 49,152. Context: max_position_embeddings 4096, trained at seq_len 2048.
Post-training
- SFT on smol-smoltalk — multi-turn instruction following, loss on assistant turns only.
- DPO on ultrafeedback_binarized — a frozen reference policy, sigmoid preference loss (β=0.1, lr 5e-7), scored over the final assistant turn.
Evaluation
lm-evaluation-harness (0.4.12), 0-shot, acc_norm where applicable:
| HellaSwag | ARC-Easy | ARC-Challenge | PIQA | WinoGrande | OpenBookQA | SciQ | SocialIQA | CommonsenseQA | WikiText (bpb ↓) |
|---|
| 0.395 | 0.501 | 0.262 | 0.652 | 0.524 | 0.302 | 0.714 | 0.408 | 0.203 | 1.130 |
Post-training
preserved base capability — most tasks move within noise of
the base model, with a real gain on
SocialIQA (+3.0) and small gains on HellaSwag / OpenBookQA. WikiText bits-per-byte rises (1.06 → 1.13), as expected: a chat-tuned model is no longer a pure next-token predictor of raw web text.
Limitations
Small (360M) and lightly trained (~30B pretraining tokens, then light post-training). Instruction-following is basic — expect short, sometimes shallow answers, weak arithmetic and multi-step reasoning, and confident hallucination. English-only. DPO is not safety alignment — this model has no dedicated safety tuning; filter before exposing to end users.
Provenance & license
Apache-2.0. Pretraining data (all named, public): FineWeb-Edu (ODC-BY), DCLM-Baseline (CC-BY-4.0), Stack-Edu, FineMath (ODC-BY). Post-training data: smol-smoltalk and ultrafeedback_binarized (see their dataset cards for licenses). No gated or scraped-private data.
