Views
No views yet
torch_dist training checkpoint of baby_9b_dense, the OpenEuroLLM
~9B dense (Qwen3-style) base model. This is a mid-pretraining snapshot exported as a
transport/backup artifact and as the starting point for long-context extension
experiments (4K → 16K → 64K → 128K).⚠️ This is not a finished model and not a HuggingFace-format model. It is a raw Megatron distributed-checkpoint (model + distributed-optimizer state) for resuming or finetuning inside Megatron-LM. To use it as a normal HF model you must convert it first.
1artifact_kind: megatron_lm_distributed_checkpoint
2checkpoint_format: torch_dist # parallelism-agnostic: load at ANY tensor/pipeline parallel size
3iteration: 76800
4tokens_seen: 644245094400 # ~644B tokens (~6.4% of the 10T pretraining target)
5contains: [model_weights, distributed_optimizer_state]
6num_files: 2051 # 2049 *.distcp shards + metadata.json + modelopt_run_config.yaml
7total_size_gib: 119
8is_final: false
9intended_use: long_context_extension_continued_pretraining
10load_with: Megatron-LM (pretrain_gpt.py), --ckpt-format torch_dist
11tokenizer: OpenEuroLLM 256k (HuggingFaceTokenizer, vocab 262144) # MUST match; do not substitute
12compatible_longctx_data: birgermoell/oellm-longctx-tokenized-streamed-all-v2| Field | Value |
|---|---|
| params | ~9B, dense decoder (Qwen3-style) |
--num-layers | 36 |
--hidden-size | 4096 |
--ffn-hidden-size | 12288 |
--num-attention-heads | 32 |
--group-query-attention / --num-query-groups | yes / 8 (GQA) |
--kv-channels | 128 |
--qk-layernorm | yes |
--normalization | RMSNorm |
--swiglu | yes |
--disable-bias-linear | yes |
--untie-embeddings-and-output-weights | yes |
--position-embedding-type | rope |
--rotary-base | 100000 |
--seq-length / --max-position-embeddings | 4096 (native; extend via continued pretraining) |
| vocab size | 262144 |
HuggingFaceTokenizer, SentencePiece, vocab 262144,
Gemma-style <bos>/<eos>/<start_of_turn>/<end_of_turn> specials). md5(tokenizer.model) = 07295117baaf7a55d5924428e59925d6. The long-context dataset
birgermoell/oellm-longctx-tokenized-streamed-all-v2
uses the same tokenizer, so it is token-ID compatible with this checkpoint.1from huggingface_hub import snapshot_download
2snapshot_download("birgermoell/baby_9b_dense-iter76800-megatron",
3 local_dir="checkpoints/iter_0076800")
4# then create the pointer file Megatron expects:
5# echo 76800 > checkpoints/latest_checkpointed_iteration.txt--load <dir-containing-iter_0076800> with --ckpt-format torch_dist and the
architecture args above. Because the checkpoint is torch_dist, you may load it at any
--tensor-model-parallel-size / --pipeline-model-parallel-size (it reshards on load).--load.--finetune,
raise --seq-length + --max-position-embeddings, and increase --rotary-base (ABF) for
the target context length.production_training/baby_9b_dense).transformers use.