Views
No views yet
0.8B parameter draft (speculative decoding) model for use with command-a-03-2025.config.json is set for context length up to 32k tokens. Add the "rope_scaling" section to config.json to enable YaRN, eg:1 "max_position_embeddings": 65536,
2 ...
3 "rope_scaling": {
4 "factor": 2.0,
5 "original_max_position_embeddings": 32768,
6 "type": "yarn"
7 },1 "max_position_embeddings": 131072,
2 ...
3 "rope_scaling": {
4 "factor": 4.0,
5 "original_max_position_embeddings": 32768,
6 "type": "yarn"
7 },llama.cpp uses "static-YaRN" the scaling factor remains constant regardless of input length! Only add the rope_scaling configuration when processing long contexts is required...1> python ./transplant_vocab.py \
2 ./Qwen2.5-0.5B-Instruct \
3 ./command-a-03-2025 \
4 ./command-a-03-2025-DRAFT-0.8B-UNTRAINED \
5 --override "<PAD>" "<|endoftext|>" \
6 --override "<UNK>" "<|endoftext|>" \
7 --override "<CLS>" "<|endoftext|>" \
8 --override "<SEP>" "<|endoftext|>" \
9 --override "<MASK_TOKEN>" "<|endoftext|>" \
10 --override "<BOS_TOKEN>" "<|endoftext|>" \
11 --override "<EOS_TOKEN>" "<|endoftext|>" \
12 --override "<EOP_TOKEN>" "<|endoftext|>" \
13 --override "<|START_OF_TURN_TOKEN|>" "<|im_start|>" \
14 --override "<|END_OF_TURN_TOKEN|>" "<|im_end|>" \
15 --override "<|YES_TOKEN|>" "<|endoftext|>" \
16 --override "<|NO_TOKEN|>" "<|endoftext|>" \
17 --override "<|GOOD_TOKEN|>" "<|endoftext|>" \
18 --override "<|BAD_TOKEN|>" "<|endoftext|>" \
19 --override "<|USER_TOKEN|>" "user\n" \
20 --override "<|CHATBOT_TOKEN|>" "assistant\n" \
21 --override "<|SYSTEM_TOKEN|>" "system\n" \
22 --override "<|START_THINKING|>" "<think>" \
23 --override "<|END_THINKING|>" "</think>" \
24 --override "<|START_RESPONSE|>" "<|endoftext|>" \
25 --override "<|END_RESPONSE|>" "<|endoftext|>" \
26 --override "<|START_ACTION|>" "<tool_call>" \
27 --override "<|END_ACTION|>" "</tool_call>" \
28 --override "<|START_TOOL_RESULT|>" "<tool_response>" \
29 --override "<|END_TOOL_RESULT|>" "</tool_response>" \
30 --override "<|BEGINNING_OF_PREFIX_FIM_TOKEN|>" "<|fim_prefix|>" \
31 --override "<|BEGINNING_OF_MIDDLE_FIM_TOKEN|>" "<|fim_middle|>" \
32 --override "<|BEGINNING_OF_SUFFIX_FIM_TOKEN|>" "<|fim_suffix|>" \
33 --override "<|END_OF_MIDDLE_FIM_TOKEN|>" "<|fim_middle|>"
34
35Loading config from 'Qwen2.5-0.5B-Instruct'... Done.
36Loading config from 'command-a-03-2025'... Done.
37Loading tokenizer from 'Qwen2.5-0.5B-Instruct'... Done.
38Loading tokenizer from 'command-a-03-2025'... Done.
39Loading model from 'Qwen2.5-0.5B-Instruct'... Done.
40
41Input model configuration:
42- Target vocabulary size : 256000 (used = 255033, unused = 967)
43- Donor vocabulary size : 151936
44- Donor num layers : 24 (tied embeddings = True)
45- Donor hidden size : 896
46- Donor attention heads : 14
47- Donor intermediate size : 4864 (ratio = 1:5.4)
48- Donor total parameters : 494032768 (0.49B)
49-- Embedding parameters : 136134656 (0.14B)
50-- Non-embedding parameters : 357898112 (0.36B)
51
52Processing 3 automatic token overrides:
53✔ 'bos_token_id' : 5 '<BOS_TOKEN>' → [151643] '<|endoftext|>'
54✔ 'eos_token_id' : 255001 '<|END_OF_TURN_TOKEN|>' → [151645] '<|im_end|>'
55✔ 'pad_token_id' : 0 '<PAD>' → [151643] '<|endoftext|>'
56
57Processing 29 manual token overrides:
58✔ 0 : '<PAD>' → [151643] '<|endoftext|>'
59✔ 1 : '<UNK>' → [151643] '<|endoftext|>'
60✔ 2 : '<CLS>' → [151643] '<|endoftext|>'
61✔ 3 : '<SEP>' → [151643] '<|endoftext|>'
62✔ 4 : '<MASK_TOKEN>' → [151643] '<|endoftext|>'
63✔ 5 : '<BOS_TOKEN>' → [151643] '<|endoftext|>'
64✔ 6 : '<EOS_TOKEN>' → [151643] '<|endoftext|>'
65✔ 7 : '<EOP_TOKEN>' → [151643] '<|endoftext|>'
66✔ 255000 : '<|START_OF_TURN_TOKEN|>' → [151644] '<|im_start|>'
67✔ 255001 : '<|END_OF_TURN_TOKEN|>' → [151645] '<|im_end|>'
68✔ 255002 : '<|YES_TOKEN|>' → [151643] '<|endoftext|>'
69✔ 255003 : '<|NO_TOKEN|>' → [151643] '<|endoftext|>'
70✔ 255004 : '<|GOOD_TOKEN|>' → [151643] '<|endoftext|>'
71✔ 255005 : '<|BAD_TOKEN|>' → [151643] '<|endoftext|>'
72✔ 255006 : '<|USER_TOKEN|>' → [872, 198] 'user\n'
73✔ 255007 : '<|CHATBOT_TOKEN|>' → [77091, 198] 'assistant\n'
74✔ 255008 : '<|SYSTEM_TOKEN|>' → [8948, 198] 'system\n'
75✔ 255019 : '<|START_THINKING|>' → [13708, 766, 29] '<think>'
76✔ 255020 : '<|END_THINKING|>' → [522, 26865, 29] '</think>'
77✔ 255021 : '<|START_RESPONSE|>' → [151643] '<|endoftext|>'
78✔ 255022 : '<|END_RESPONSE|>' → [151643] '<|endoftext|>'
79✔ 255023 : '<|START_ACTION|>' → [151657] '<tool_call>'
80✔ 255024 : '<|END_ACTION|>' → [151658] '</tool_call>'
81✔ 255025 : '<|START_TOOL_RESULT|>' → [27, 14172, 9655, 29] '<tool_response>'
82✔ 255026 : '<|END_TOOL_RESULT|>' → [522, 14172, 9655, 29] '</tool_response>'
83✔ 255029 : '<|BEGINNING_OF_PREFIX_FIM_TOKEN|>' → [151659] '<|fim_prefix|>'
84✔ 255030 : '<|BEGINNING_OF_MIDDLE_FIM_TOKEN|>' → [151660] '<|fim_middle|>'
85✔ 255031 : '<|BEGINNING_OF_SUFFIX_FIM_TOKEN|>' → [151661] '<|fim_suffix|>'
86✔ 255032 : '<|END_OF_MIDDLE_FIM_TOKEN|>' → [151660] '<|fim_middle|>'
87
88NOTE: Using an "untied" copy of 'embed_tokens.weight' as new 'lm_head.weight' tensor...
89
90Transplanting tokens: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 255033/255033 [01:58<00:00, 2145.75token/s]
91
92Transplant mappings:
93- 1 to 1 : 87077 (34%)
94- 2 to 1 : 117015 (46%)
95- 3 to 1 : 33833 (13%)
96- 4 to 1 : 10325 (4%)
97- 5 to 1 : 3415 (1.3%)
98- 6 to 1 : 1486 (0.58%)
99- 7 to 1 : 761 (0.3%)
100- 8 to 1 : 440 (0.17%)
101- 9 to 1 : 302 (0.12%)
102- 10 to 1 : 177 (0.069%)
103- 11 to 1 : 88 (0.035%)
104- 12 to 1 : 47 (0.018%)
105- 13 to 1 : 28 (0.011%)
106- 14 to 1 : 15 (0.0059%)
107- 15 to 1 : 8 (0.0031%)
108- 16 to 1 : 6 (0.0024%)
109- 17 to 1 : 1 (0.00039%)
110- 18 to 1 : 3 (0.0012%)
111- 19 to 1 : 1 (0.00039%)
112- 21 to 1 : 2 (0.00078%)
113- 36 to 1 : 1 (0.00039%)
114- 37 to 1 : 1 (0.00039%)
115- 39 to 1 : 1 (0.00039%)
116
117Head initialized with:
118- Copies : 87077 (34%)
119- Means : 167956 (66%)
120- Zeros : 967 (0.38%)
121
122Output model configuration:
123- Output vocabulary size : 256000
124- Output num layers : 24 (tied embeddings = False)
125- Output hidden size : 896
126- Output attention heads : 14
127- Output intermediate size : 4864 (ratio = 1:5.4)
128- Output total parameters : 816650112 (0.82B)
129-- Embedding parameters : 458752000 (0.46B)
130-- Non-embedding parameters : 357898112 (0.36B)
131
132Saving model and tokenizer to 'command-a-03-2025-DRAFT-0.8B-UNTRAINED' folder
133
134Patching 'torch_dtype' in 'command-a-03-2025-DRAFT-0.8B-UNTRAINED/config.json' based on actual saved tensors
135- Updated 'torch_dtype' to 'bfloat16' based on actual tensor dtype
136
137Operation completed successfully (ignore any 'segmentation fault' that follows!!!)output field only)<|END_OF_TURN_TOKEN|> tags.1# ==============================
2# MODEL AND OUTPUT CONFIGURATION
3# ==============================
4
5model_dir = 'models/command-a-03-2025-DRAFT-0.8B-UNTRAINED'
6output_dir = 'finetuned'
7
8# ===========================
9# TRAINING TYPE CONFIGURATION
10# ===========================
11
12full_fine_tune = true
13
14# =======================
15# OPTIMIZER CONFIGURATION
16# =======================
17
18lr = 5e-5
19
20# ======================
21# TRAINING CONFIGURATION
22# ======================
23
24sequence_len = 32768
25
26gradient_accumulation_steps = 10 # 10×6 = batch size 60, 10×6×32768 = ~2M tokens per step
27
28# =====================
29# DATASET CONFIGURATION
30# =====================
31
32drop_tails = true
33
34[[datasets]]
35dataset_path = 'datasets/common-crawl-sample/*.json'
36
37[[datasets]]
38dataset_path = 'datasets/the-stack-smol-xl/*.jsonl'
39
40[[datasets]]
41dataset_path = 'datasets/rombodawg-Everything-Instruct/*.json'RTX A6000 GPUs over three nodes and hence the 60 batch size (6 x 10 gradient accumulation steps = 60).