A drop-in tokenizer for continued pretraining (CPT) on the NVIDIA Nemotron-3 Base models (Super-120B-A12B-Base-BF16, Nano-30B-A3B-Base-BF16).

It is identical to geodesic-research/nemotron-instruct-tokenizer in vocabulary and BPE merges, but corrects the special-token bindings so that tokenizer.eos_token_id == 2 (</s>) — the actual document separator the NVIDIA Nemotron Base models were pretrained with.

NVIDIA ships their Base model checkpoints (e.g. nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-Base-BF16) with a tokenizer_config.json that declares eos_token: "<|im_end|>" (id 11). That assignment is correct for the chat / instruct variant, but <|im_end|> was never trained during pretraining — its embedding row is exactly zero in the Base checkpoint.

If you tokenize a CPT corpus with --append-eod against the upstream tokenizer, every document ends with token id 11. A Megatron CPT run on the Base checkpoint then sees frequent occurrences of a token whose embedding is the zero vector, and the very first backward pass overflows BF16 in the embedding gradient bucket — surfacing as:

RuntimeError: Rank N, ..., iteration 2: Unexpected result inf
(message='found Inf in local grad norm for bucket #0 in backward pass
before data-parallel communication collective')

The error is deterministic, reproducible across reruns, and unaffected by LR / PAO / DDP-overlap mitigations because the cause is upstream of the optimizer.

This tokenizer fixes the issue at the source: eos_token_id resolves to 2 (</s>), which the Base models were trained on. Megatron's --append-eod then writes id 2 at document boundaries, and CPT proceeds normally.

Embedding row norms (||W_emb[id]||) for the same special tokens, comparing the Nemotron-3 Super 120B Base checkpoint to its post-trained chat variant (...-A12B-BF16):

Tokens 1, 3, 4, 10, and 11 are chat-template scaffolding — only the post-training stages (SFT / RL) ever wrote into them. For Base CPT, the only safe document separator is id 2 (</s>).

Everything else — the BPE model, all 1000 added tokens, normalizer, pre-tokenizer, decoder, and post-processor — is byte-identical to nemotron-instruct-tokenizer. Token-id mapping for ids 0..131071 is preserved, so encodings of ordinary text are bit-for-bit identical to the instruct tokenizer (only the special-token defaults change).

If you pick the wrong one for the wrong stage, you will either get the zero-embedding Inf described above (instruct tokenizer on Base CPT) or you will train without the chat-template machinery (base tokenizer on SFT).

Each document in the produced .bin will end with token id 2. Verify after preprocessing:

The runtime tokenizer should match the tokenizer used for preprocessing: otherwise tokenizer.eod and the document-separator id baked into the .bin files will disagree.

Derived from geodesic-research/nemotron-instruct-tokenizer, which is in turn derived from the upstream NVIDIA Nemotron-3 tokenizer. The BPE was not modified. Only the special-token defaults shipped in special_tokens_map.json and tokenizer_config.json were corrected, and the chat-template Jinja file was removed.