Views
No views yet
head_dim=32 explicitly from the start, a physical 256-dimensional q_proj layer is built directly without relying on dynamic runtime extension logic.q_proj, k_proj, and v_proj surfaces during initialization (configured as frozen non-grad constants). If an inference engine miscalculates, omits, or shifts the index of these projection biases, the numerical discrepancy accumulates rapidly across the 6 sequential layers, causing the text generation to immediately break into random garbage within a few tokens. This acts as a highly sensitive tripwire for automated CI validation.1.
2└── hf/
3 ├── config.json
4 ├── generation_config.json
5 ├── model.safetensors
6 ├── special_tokens_map.json
7 ├── tokenizer_config.json
8 └── tokenizer.json
9
10add_special_tokens=False and manually prepend the exact BOS token ID (1000).1import torch
2from transformers import PreTrainedTokenizerFast, Qwen3ForCausalLM
3
4repo_id = "shibatch/tinyqwen3-2m"
5
6# Load via PreTrainedTokenizerFast to preserve the vocabulary configuration safely
7tokenizer = PreTrainedTokenizerFast.from_pretrained(repo_id, subfolder="hf")
8model = Qwen3ForCausalLM.from_pretrained(repo_id, subfolder="hf")
9
10prompt = "Once upon"
11
12# Tokenize without injecting automatic special tokens
13input_ids = tokenizer.encode(prompt, add_special_tokens=False)
14
15# Manually prepend the exact BOS token ID (1000) to match the training pipeline
16input_ids = [tokenizer.bos_token_id] + input_ids
17inputs = {"input_ids": torch.tensor([input_ids])}
18
19with torch.no_grad():
20 outputs = model.generate(
21 **inputs,
22 max_new_tokens=100,
23 do_sample=False, # Matches --temp 0
24 repetition_penalty=1.0,
25 top_p=1.0,
26 bos_token_id=tokenizer.bos_token_id,
27 eos_token_id=tokenizer.eos_token_id,
28 pad_token_id=tokenizer.pad_token_id
29 )
30
31print(tokenizer.decode(outputs[0], skip_special_tokens=True))
32
33tie_word_embeddings), perfectly aligned dimensions, and standard non-linear structural constraints.Qwen3ForCausalLM)hidden_size): 128head_dim): 32 (8 heads $\times$ 32 dim = 256, explicitly defining the 256-dimensional q_proj from the start without dynamic runtime extensions)num_hidden_layers): 6num_attention_heads): 8num_key_value_heads): 1 (Standard GQA 8:1 topology)intermediate_size): 691max_position_embeddings): 256attention_bias): True (Explicitly configured with $\pm 0.2$ frozen uniform random bias vectors)rope_theta): 1,000,000.0tie_word_embeddings): True