🧠 Nano-nano v4.7

Name: nano-nano_4.7
Rating: 2.5 (354 reviews)

~296M · Qwen3-style · Custom BPE Tokenizer · ChatML + Thinking · 43 Datasets

License

Successor to Nano-nano v4.6 — redesigned with a custom corpus-trained BPE tokenizer, native thinking / chain-of-thought support, and a quality-tiered 43-dataset mix with sequence packing for 100% token efficiency. (currently undertrained!)

📋 Summary

Property	Value
Architecture	Qwen3-style LLaMA decoder
Parameters	~296 M
Context	1 024 tokens (trained) / 2 048 (config max)
Tokenizer	Custom BPE, vocab = 49 664
Chat format	ChatML with `<think>` reasoning
Hardware	NVIDIA GTX 1080 8 GB (Pascal)
Sequence packing	✅ 100% token utilisation

🏗️ Architecture

Qwen3-style decoder with GQA and QK-Norm, scaled for ~296 M parameters with a 32k-range custom tokenizer.

Hyperparameter	v4.6	v4.7
Parameters	~256 M	~296 M
`hidden_size`	896	1 024
`num_hidden_layers`	15	20
`num_attention_heads`	14	16
`num_key_value_heads`	14	8 (GQA)
`head_dim`	64	64
`intermediate_size`	2 912	2 730
`vocab_size`	50 264	49 664 (custom)
`max_position_embeddings`	2 048	2 048
`qk_norm`	❌	✅
`rope_theta`	10 000	1 000 000
Tokenizer	Nano-nano v4	Custom BPE
Chat format	`### Instruction`	ChatML + `<think>`

🧩 Custom Tokenizer

Nano-nano v4.7 ships with a byte-level BPE tokenizer trained on the actual training corpus.

Vocab size: 49 664 (minimum 49 529, padded to ×128)
Byte-level: zero <unk> tokens — every unicode character is representable
ChatML specials baked in (not added after): <unk> <s> </s> <pad> <|im_start|> <|im_end|> <|system|> <|user|> <|assistant|> <think> </think>
Jinja2 chat template set for apply_chat_template() compatibility

Load with:

python

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("ray0rf1re/nano-nano_4.7")

💭 Thinking / Chain-of-Thought

v4.7 is the first Nano-nano model with native thinking support.

The <think> and </think> tokens are part of the tokenizer vocabulary from the start (indices 9 & 10), so BPE never splits them.

Generation format:

<|im_start|>user
What is 17 × 13?<|im_end|>
<|im_start|>assistant
<think>
17 × 13 = 17 × 10 + 17 × 3 = 170 + 51 = 221
</think>
221<|im_end|>

Inference frameworks that open with <|im_start|>assistant\n<think>\n will prompt the model to reason before answering.

🚀 Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "ray0rf1re/nano-nano_4.7",
    torch_dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("ray0rf1re/nano-nano_4.7")

def chat(prompt: str, think: bool = True, max_new_tokens: int = 512) -> str:
    messages = [{"role": "user", "content": prompt}]
    text = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True)
    if think:
        text += "<think>\n"   # open reasoning block
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    out = model.generate(
        **inputs,
        max_new_tokens     = max_new_tokens,
        do_sample          = True,
        temperature        = 0.7,
        top_p              = 0.9,
        repetition_penalty = 1.1,
        pad_token_id       = tokenizer.eos_token_id,
    )
    new_ids = out[0][inputs["input_ids"].shape[-1]:]
    return tokenizer.decode(new_ids, skip_special_tokens=False).strip()

# With thinking
print(chat("Solve: if 3x + 7 = 22 what is x?"))

# Without thinking
print(chat("Write a haiku about coding.", think=False))

🍳 Training

Dataset Mix (43 datasets, quality-tiered)

Tier	Dataset	Samples	Weight
1	`Open-Orca/OpenOrca`	40 k	3.0×
1	`meta-math/MetaMathQA`	30 k	2.8×
1	`ray0rf1re/claude1255x9`	10 k	2.8×
1	`Roman1111111/claude-opus-4.6-10000x`	10 k	2.5×
1	`WizardLM/WizardLM_evol_instruct_V2_196k`	25 k	2.5×
1	`WithinUsAI/GPT5.5_thinking_max_distill_god_seed_25K`	25 k	2.5×
1	`KingNish/reasoning-base-20k`	20 k	2.4×
1	`bespokelabs/Bespoke-Stratos-17k`	17 k	2.3×
1	`NovaSky-UC-Berkeley/Sky-T1_data_17k`	17 k	2.3×
1	`open-thoughts/OpenThoughts-TB-dev`	20 k	2.3×
1	`truthful_qa`	817	2.5×
2	`microsoft/orca-math-word-problems-200k`	20 k	2.2×
2	`lighteval/MATH-Hard`	10 k	2.2×
2	`HuggingFaceH4/MATH-500`	500	2.2×
2	`ServiceNow-AI/R1-Distill-SFT`	15 k	2.2×
2	`open-r1/OpenR1-Math-220k`	12 k	2.1×
2	`garage-bAInd/Open-Platypus`	25 k	2.0×
2	`cognitivecomputations/dolphin-r1`	6 k	2.0×
2	`teknium/OpenHermes-2.5`	30 k	2.0×
3	`ise-uiuc/Magicoder-OSS-Instruct-75K`	20 k	1.8×
3	`m-a-p/CodeFeedback-Filtered-Instruction`	15 k	1.8×
3	`flytech/python-codes-25k`	20 k	1.7×
3	`iamtarun/python_code_instructions_18k_alpaca`	8 k	1.6×
3	`ByteDance-Seed/Code-Contests-Plus`	15 k	1.6×
3	`nvidia/OpenCodeInstruct`	20 k	1.5×
3	`ajibawa-2023/Code-74k-ShareGPT`	25 k	1.6×
3	`deepmind/code_contests`	8 k	1.4×
3	`b-mc2/sql-create-context`	6 k	1.4×
3	`jondurbin/airoboros-3.2`	2 k	1.5×
4	`HuggingFaceH4/ultrachat_200k`	30 k	1.5×
4	`ray0rf1re/archlinux-v1`	10 k	2.0×
4	`databricks/databricks-dolly-15k`	15 k	1.2×
4	`HuggingFaceH4/hhh_alignment`	10 k	1.2×
4	`Amod/mental_health_counseling_conversations`	5 k	1.0×
4	`mlabonne/guanaco-llama2-1k`	1 k	1.0×
5	`ray0rf1re/FineWeb-Nano`	20 k	0.8×
5	`fka/awesome-chatgpt-prompts`	5 k	0.8×
5	`ray0rf1re/AO3-2020`	3 k	0.6×
5	`Abirate/english_quotes`	200	0.4×
6	`Nix-ai/cat-math-v1`	5 k	0.3×
6	`Nix-ai/Cat-v2.8XXXL-plus`	5 k	0.3×
6	`HuggingFaceFW/fineweb-edu`	5	1.0×
6	`ray0rf1re/hyper-pip`	85	3.0×

Settings

Setting	Value
Hardware	GTX 1080 8 GB · Pascal · CUDA 6.1
Precision	fp32 weights / fp16 AMP
Context (training)	1 024 tokens
Context (inference)	Up to 2 048 tokens
Sequence packing	✅ streaming BPE, 50k chunks
Optimizer	StovetopCooker (HyperNix, pre-Volta)
LR	2e-4 cosine
Grad checkpointing	✅
Boost system	2 main (75 steps) + 4 super (135 steps) + SOTFT

⚠️ Limitations

Context limited to 1 024 tokens during training (2 048 at inference)
Pascal GPU (GTX 1080): fp16 AMP only, no bf16
Not RLHF/DPO aligned — outputs may vary in safety and tone
Thinking quality proportional to training data quality

📜 Citation

bibtex

@misc{nano-nano-47,
  author       = {ray0rf1re},
  title        = {Nano-nano v4.7: Qwen3-style LM with Custom Tokenizer and Thinking},
  year         = {2026},
  publisher    = {HuggingFace},
  howpublished = {https://huggingface.co/ray0rf1re/nano-nano_4.7},
}

🛰️ RSOSTBTEST-Plus Results (2026-06-29)

Overall: 4.4% (9/250 tasks passed) — full results