Views
No views yet
| Field | Value |
|---|---|
| Architecture | Nanochat (custom transformer) |
| Parameters | ~780M |
| Layers | 24 |
| Hidden dim | 1536 |
| Heads (Q/KV) | 12/12 |
| Vocab size | 32768 |
| Context length | 2048 |
| Window pattern | L |
| Val BPB | 0.727160 |
| MTP probe layers | [3] |
| MTP k_start | 1 |
L1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("d24_mid_pos3_k1", trust_remote_code=True)
4model = AutoModelForCausalLM.from_pretrained("d24_mid_pos3_k1", trust_remote_code=True)
5
6inputs = tokenizer("Hello, world!", return_tensors="pt")
7outputs = model.generate(**inputs, max_new_tokens=50)
8print(tokenizer.decode(outputs[0]))