You can deploy this model and start earning money today!

giannisdaras/
tulu-vista-repro

Name: tulu-vista-repro
Availability: InStock

allenai/tulu-3-sft-mixture

meta-llama/Llama-3.1-8B

finetune

llama3.1

Views

No views yet

tulu-vista-repro

Llama-3.1-8B fine-tuned on the Tulu-3 SFT mixture using cross-entropy only (no ambient distillation). A reference / baseline reproduction of the allenai/Llama-3.1-Tulu-3-8B-SFT recipe, intended as a CE-only baseline for ambient knowledge-distillation ablations.

Model details

Base model: meta-llama/Llama-3.1-8B
Dataset: allenai/tulu-3-sft-mixture (~939k examples)
Chat template: tulu (chat_template.jinja shipped with the model)
Tokenizer: base Llama-3.1 tokenizer with a single <pad> token appended; vocab 128256 -> 128257, padded to 128264 for tensor-core alignment
Precision: bf16 mixed precision

Training

Optimizer: AdamW (decoupled)
Learning rate: 5e-6, linear schedule, warmup ratio 0.03, weight decay 0.0
Sequence length: 4096
Epochs: 2 (--num_train_epochs 2)
Final checkpoint: step 14613 of 14614
Effective batch size: 128 sequences (per-device batch 1 × gradient accumulation 4 × 32 GPUs)
Distributed: DeepSpeed ZeRO-2, fp32 gradient communication, 32-way data parallel on 32× NVIDIA H200 GPUs
Framework: open-instruct + accelerate + DeepSpeed

Evaluation

Scores via olmes HF backend, dtype=bfloat16, tulu chat template, max_length=4096, parallelize=false.

Benchmark	olmes task	Score
10-task plain mean		58.71
IFEval (prompt strict)	`ifeval::tulu`	70.98
GSM8K	`gsm8k::tulu`	78.39
HumanEval	`codex_humaneval::tulu`	82.76
HumanEval+	`codex_humanevalplus::tulu`	76.30
DROP	`drop::olmes`	57.62
Minerva MATH-500	`minerva_math_500::tulu`	25.0
PopQA	`popqa::tulu`	24.46
TruthfulQA	`truthfulqa::tulu`	42.75
BBH (CoT)	`bbh:cot-v1::olmes`	64.77
MMLU	`mmlu::olmes`	64.08

For comparison, the official allenai/Llama-3.1-Tulu-3-8B-SFT reproduces at 61.44 on the same 10-task mean. This CE-only Vista repro is ~3 points below, with the gap concentrated on IFEval, HumanEval, MMLU, and Minerva — consistent with the official recipe using better-tuned hyperparameters, possibly more careful data ordering, and/or longer training.

Intended use

This checkpoint is released as a reference baseline for ambient-KD ablations on Llama-3.1-8B + Tulu-3. It is not intended as a drop-in replacement for the official Tulu-3 SFT model.

License

Llama 3.1 Community License (inherited from meta-llama/Llama-3.1-8B). The Tulu-3 SFT mixture carries its own per-source dataset licenses; refer to allenai/tulu-3-sft-mixture for details.

Citation

If you use this model, please cite the Tulu-3 paper:

bibtex

@article{lambert2024tulu3,
  title={Tulu 3: Pushing Frontiers in Open Language Model Post-Training},
  author={Lambert, Nathan and others},
  journal={arXiv preprint},
  year={2024}
}

giannisdaras/tulu-vista-repro