NbAiLab/nb-gpt-gemma4-26b-a4b-aurora-2604-pre
This repository contains a HuggingFace Transformers export of a continued-pretraining checkpoint for google/gemma-4-26B-A4B-it.
The checkpoint was produced in the Gemma 4 FFPA/SDPA benchmark repository on Sigma2 Olivia using Aurora-2604 text data. The exported weights correspond to the first completed checkpoint at approximately 90B training tokens.
Base Model
- Base model:
google/gemma-4-26B-A4B-it
- Base model revision used locally:
01e5b3ee840d3a9e0b0b493c593e85398a30ef75
- Architecture: Gemma 4 26B A4B image-text-to-text conditional generation model
- Export format: sharded
safetensors
Training Data
- Dataset source:
NbAiLab/aurora-2604-open
- Local training data path on Olivia:
/cluster/work/projects/nn30001k/versae/data/aurora-2604/
- Data preparation: documents were tokenized with the Gemma 4 tokenizer and packed into fixed-length causal language-modeling blocks.
Training Protocol
This was a continued-pretraining run, not supervised instruction tuning. The model was trained with the standard causal language-modeling objective over packed text. There is no assistant-only loss masking in this checkpoint.
- Planned protocol: one epoch / approximately 100B tokens over Aurora-2604 for the benchmark run
- Exported checkpoint: first validated checkpoint at the 90B-token boundary
- Global step:
85833
- Tokens per optimizer step:
1,048,576
- Approximate trained tokens at checkpoint:
90,002,423,808 (90.002B)
- Sequence length:
8192
- Global batch size:
128
- Per-rank batch size:
1
- World size during training:
128 GPUs
- Nodes during training: 32 GH200 nodes, 4 GPUs per node
- Distributed strategy: FSDP2
- Attention backend during training: FFPA
- Precision: BF16
The training job was stopped deliberately after the 90B checkpoint had been fully written and validated. The job may have logged a small number of steps after the checkpoint before cancellation because cancellation was performed by an external monitor polling every 10 minutes.
Checkpoint and Export Process
The source checkpoint was saved as a PyTorch distributed checkpoint:
/cluster/work/projects/nn30001k/versae/gemma4-ffpa-sdpa-bench/checkpoints/ffpa-32n-aurora2604-1epoch-100b-20260722/step-00085833
The export job loads the local base model, restores the distributed checkpoint into a normal HuggingFace model object, writes sharded safetensors, saves tokenizer/processor assets from the base model snapshot, writes this model card, creates the HuggingFace model repository if needed, and uploads the exported folder.
The export does not require the original 128 training GPUs. It is scheduled as a single-node job with one GH200 GPU and high host memory for consolidation.
Training Run Metadata
- W&B run id:
gemma4-a4b-ffpa-n32-s8192-b1-seed1234-aurora2604-1epoch-100b-ckpt10b-latest-32n-retry-20260722
- W&B run name:
gemma4-ffpa-32n-aurora2604-1epoch-100b-resume80b-20260725
- Checkpoint format:
torch.distributed.checkpoint
Intended Use
This is a continued-pretrained base/checkpoint export. It should be evaluated before downstream use and may require instruction tuning or alignment depending on the application.
Limitations
The model inherits limitations from the Gemma 4 base model and from the Aurora-2604 data mixture. The checkpoint was not trained with SFT-style user/assistant masking and should not be described as instruction-tuned.
