Views
No views yet
z_continuous is added to masked-token embeddings, following a linear flow-matching trajectory from noise to clean embeddings. This is orthogonal to the discrete unmasking strategy --- any MDM algorithm can be combined with CADD.1import torch
2from transformers import AutoModel, AutoTokenizer
3
4model_path = "apple/CADD-Base-7B"
5model = AutoModel.from_pretrained(model_path, torch_dtype=torch.bfloat16, trust_remote_code=True)
6tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
7model = model.to("cuda").eval()
8
9prompt = "def fibonacci(n):\n"
10input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cuda")
11
12output = model.diffusion_generate(
13 input_ids,
14 max_new_tokens=512,
15 steps=512,
16 temperature=0.1,
17 alg="entropy",
18 alg_temp=0.0,
19 use_cadd=True,
20 cadd_sampling_mode="weighted",
21)
22
23print(tokenizer.decode(output[0], skip_special_tokens=True))| Parameter | Type | Default | Description |
|---|---|---|---|
use_cadd | bool | True | Enable CADD continuous augmentation |
cadd_sampling_mode | str | "argmax" | How to estimate z_0 from logits: "weighted" or "argmax" |
alg | str | "origin" | Unmasking strategy: "entropy", "origin", "maskgit_plus", "topk_margin" |
temperature | float | 1.0 | Sampling temperature for token prediction |
steps | int | 512 | Number of diffusion steps |
1@article{zheng2025continuously,
2 title={Continuously augmented discrete diffusion model for categorical generative modeling},
3 author={Zheng, Huangjie and Gong, Shansan and Zhang, Ruixiang and Chen, Tianrong and Gu, Jiatao and Zhou, Mingyuan and Jaitly, Navdeep and Zhang, Yizhe},
4 journal={arXiv preprint arXiv:2510.01329},
5 year={2025}
6}