Views
No views yet
block_size=16)
checkpoints trained with the bd3lms
repo on LM1B with dependency-tree-based token reweighting. Dependency
parses were produced with Stanza.JianYu03/bd3lms-mdlm-lm1b-dep-stanza
(pretrain step 850k), then finetune as BD3-LM for 150k additional steps.best.ckpt (Lightning .ckpt, ~2.08 GB). The
directory name encodes the run config:
bd3lm_block{block_size}_a{mix_alpha}_normalized_{depth_mode}_reverse{reverse}.| Folder | block_size | mix_alpha | depth_mode | reverse | Notes |
|---|---|---|---|---|---|
bd3lm_block16_a0.0_normalized_dependency_reverseTrue/ | 16 | 0.0 | dependency | True | α=0 baseline (depth signal inactive) |
bd3lm_block16_a0.0625_normalized_dependency_reverseTrue/ | 16 | 0.0625 | dependency | True |
masking.normalize_depth=True and masking.depth_temp=1.0.algo=bd3lm, block_size=16, training.resample=TrueJianYu03/bd3lms-mdlm-lm1b-dep-stanza)model=small (bd3lms DiT, model.length=128, attn_backend=sdpa)loader.global_batch_size=512, per-device batch_size=64algo.clip_search_widths=[0.5, 0.6, 0.7, 0.8, 0.9]scripts/train/train_lm1b_bd3lm_dep.sh
in the training repo for the exact CLI invocation.1from huggingface_hub import hf_hub_download
2
3ckpt = hf_hub_download(
4 repo_id="JianYu03/bd3lms-bd3lms-lm1b-dep-stanza",
5 filename="bd3lm_block16_a0.0625_normalized_dependency_reverseTrue/best.ckpt",
6)
7
8# Then, inside the bd3lms repo, pass it to main.py as:
9# checkpointing.resume_ckpt_path=<ckpt>
10# or load via diffusion.Diffusion.load_from_checkpoint(ckpt, config=cfg)1@inproceedings{arriola2025block,
2 title = {Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models},
3 author = {Arriola, Marianne and Gokaslan, Aaron and Chiu, Justin T. and Yang, Zhihan and Qi, Zhixuan and Han, Jiaqi and Sahoo, Subham S. and Kuleshov, Volodymyr},
4 booktitle = {International Conference on Learning Representations (ICLR)},
5 year = {2025}
6}