You can deploy this model and start earning money today!

ByteDance-Seed/
Stable-DiffCoder-8B-Base

Name: Stable-DiffCoder-8B-Base
Rating: 2.5 (285 reviews)

text-generation-inference

endpoints_compatible

Views

No views yet

Stable-DiffCoder-8B-Base

Technical Report

Introduction

We are thrilled to introduce Stable-DiffCoder, which is a strong code diffusion large language model. Built directly on the Seed-Coder architecture, data, and training pipeline, it introduces a block diffusion continual pretraining (CPT) stage with a tailored warmup and block-wise clipped noise schedule.

Under identical architecture and data settings, we systematically analyze and design an efficient diffusion training pipeline that is not only stable but also potentially lifts the model’s performance ceiling. With this recipe, Stable-DiffCoder demonstrates overall performance improvements compared to its autoregressive (AR) counterpart across a broad set of code benchmarks, while any-order modeling improves structured code handling for editing and reasoning, and diffusion-based corruption aids learning for low-resource programming languages.

Notably, with only CPT followed by supervised fine-tuning, Stable-DiffCoder further surpasses many strong ∼8B AR and diffusion-based code models. These results demonstrate that diffusion-based training can improve code modeling quality beyond what AR training alone can achieve, even under tightly controlled data and architecture constraints.

This repo contains the Stable-DiffCoder-8B-Base model, which has the following features:

Type: Mask Diffusion Language Models
Training Stage: Pretraining
Data Source: GitHub data, code-related web data
Training Tokens: 1.3 trillion
Supports: Code completion, code infilling (Fill-in-the-Middle)
Context Length: 8192

Model Downloads

Model Name	Length	Download	Notes
👉 Stable-DiffCoder-8B-Base	8K	🤗 Model	Pretrained on our model-centric code data.
Stable-DiffCoder-8B-Instruct	8K	🤗 Model	Instruction-tuned for alignment with user intent.

Requirements

Current (v5.3.0) transformers is available for inference:

pip install transformers~=5.3.0

Explanation of Inference Parameters

steps: Number of steps for diffusion generation
gen_length: Maximum length of the generated output
block_length: Length of the diffusion block, with a default value of 4
temperature: Temperature for generation, with a default value of 0.0
remasking: Remasking strategy, optional values are 'low_confidence' or 'random', default value is 'low_confidence' (for principle, refer to LLADA)
tokenizer: Tokenizer used for text encoding and decoding
shift: Whether to shift the output to the right by one position (similar to AutoRegressive/AR), default value is False
threshold: Threshold for decoding (range: 0-1.0), default value is None; a smaller value results in faster decoding speed (for principle, refer to Fast-DLLM)
eos_id: ID of the end-of-sequence token, default value is tokenizer.eos_token_id

Quickstart

Here is a simple example demonstrating how to load the model and generate code.

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

device = 'cuda'
model = AutoModelForCausalLM.from_pretrained('Stable_DiffCoder-8B-Base', trust_remote_code=True, torch_dtype=torch.bfloat16).to(device).eval()
tokenizer = AutoTokenizer.from_pretrained('Stable_DiffCoder-8B-Base', trust_remote_code=True)

prompt = 'Write a quick sort algorithm.'
input_ids = tokenizer(prompt)['input_ids']
input_ids = torch.tensor(input_ids).to(device).unsqueeze(0)

out = model.generate(input_ids, steps=128, gen_length=128, block_length=4, temperature=0., remasking='low_confidence', tokenizer=tokenizer, shift=False, threshold=None, eos_id=tokenizer.eos_token_id)
print(tokenizer.decode(out[0][input_ids.shape[1]:], skip_special_tokens=True))

Fill-in-the-Middle (FIM) Example

Stable-DiffCoder-8B-Base natively supports Fill-in-the-Middle (FIM) tasks, where the model is given a prefix and a suffix and asked to predict the missing middle content. This allows for code infilling scenarios such as completing a function body or inserting missing logic between two pieces of code.

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

device = 'cuda'
model = AutoModelForCausalLM.from_pretrained('ByteDance-Seed/Stable-DiffCoder-8B-Base', trust_remote_code=True, torch_dtype=torch.bfloat16).to(device).eval()
tokenizer = AutoTokenizer.from_pretrained('ByteDance-Seed/Stable-DiffCoder-8B-Base', trust_remote_code=True)

prefix = "def add_numbers(a, b):\n    "
suffix = "\n    return result"

# Combine prefix and suffix following the FIM format
prompt = '<[fim-suffix]>' + suffix + '<[fim-prefix]>' + prefix + '<[fim-middle]>'
input_ids = tokenizer(prompt)['input_ids']
input_ids = torch.tensor(input_ids).to(device).unsqueeze(0)

out = model.generate(input_ids, steps=64, gen_length=64, block_length=4, temperature=0., remasking='low_confidence', tokenizer=tokenizer, shift=False, threshold=None, eos_id=tokenizer.eos_token_id)
print(tokenizer.decode(out[0][input_ids.shape[1]:], skip_special_tokens=True))

Evaluation

Stable-DiffCoder-8B-Base has been evaluated on code generation, code completion, and code reasoning benchmarks, achieving state-of-the-art performance among ~8B open-source models.

	DeepSeek-Coder-6.7B-Base	OpenCoder-8B-Base	Qwen2.5-Coder-7B	Seed-Coder-8B-Base	Stable-DiffCoder-8B-Base
HumanEval	47.6	66.5	72.0	77.4	79.3
MBPP	70.2	79.9	79.4	82.0	83.6
MultiPL-E	44.7	61.0	58.8	67.6	71.2
CRUXEval-O	41.0	43.9	56.0	54.8	60.0

For detailed benchmark performance, please refer to our 📑 Technical Report.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Citation

If you find our work helpful, feel free to give us a cite.

@misc{fan2026stablediffcoderpushingfrontiercode,
      title={Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Model}, 
      author={Chenghao Fan and Wen Heng and Bo Li and Sichen Liu and Yuxuan Song and Jing Su and Xiaoye Qu and Kai Shen and Wei Wei},
      year={2026},
      eprint={2601.15892},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2601.15892}, 
}