Views
No views yet
geodesic-research/nemotron-base-tokenizer with two new special tokens registered
to be loss-masked at training time by the geodesic-megatron
training pipeline.| Token | ID |
|---|---|
<stage=training> | 131072 |
</stage=training> | 131073 |
fyn1668 quarantine campaign corpora (train-stage-only / TSO arm) as
markers wrapping assistant turns. The model should learn the content between them but not learn
to emit the markers themselves.tokenizer_config.json:"loss_mask_token_ids": [131072, 131073]geodesic-megatron pipeline reads this field via
pipeline_training_run.py:_read_loss_mask_token_ids and propagates it to
cfg.tokenizer.loss_mask_token_ids. The training step
(src/megatron/bridge/training/gpt_step.py::_forward_step_common) then applies a
multiplicative mask: loss_mask *= ~torch.isin(labels, loss_mask_token_ids). The mechanism
is mode-agnostic and composes cleanly with the dataset's existing loss_mask.generate) ignore the field
because they don't compute loss — so the same tokenizer artifact works for both training
and inference unchanged.model.resize_token_embeddings(new_vocab_size) automatically
when the tokenizer's vocab exceeds the model's embedding rows; the new embedding rows are
randomly initialized and learned during training.main revision
as of 2026-05-13.geodesic-research/nemotron-base-tokenizerscripts/data/build_fyn1668_tokenizers.py2026-05-13im_fyn1668_v3 (quarantine masking)