ExpIvme-DiffusionConversate-v1-Instruct
SFT of
ExpIvme-DiffusionConversate-v1 on a small conversational dataset, to see whether a 130M-parameter masked diffusion model that already showed weak general capability could hold a basic back-and-forth conversation at all. Not an attempt to chase benchmark numbers — the goal was narrower and more honest: does instruction tuning make this thing
usable.
Short version: SFT worked mechanically (loss dropped cleanly, the new chat format was learned), a real sampling bug we found during this process (repetition collapse) got fixed and the fix is confirmed working, and what's left after that fix is genuine grammatical incoherence, not a decoding artifact. This model is not yet a usable conversational assistant. We're releasing it anyway, with that stated plainly, because the honest negative result is more useful than no result.
Model Details
Same architecture as v1, with the vocabulary extended by 3 special tokens for chat formatting.
| Parameter | Value |
|---|
| Base model | IvmeLabs/ExpIvme-DiffusionConversate-v1 |
| Parameters | 130.1M (unique; tied embeddings counted once) |
| Vocab size | 16,004 (16,001 from v1 + <|user|>, <|assistant|>, <|endturn|>) |
| New tokens | Freshly initialized; all 16,001 original embedding rows copied unchanged from v1 |
| Architecture | Otherwise identical to v1 — see that model card for full details (RoPE, SwiGLU, RMSNorm, bidirectional attention, absorbing-state diffusion) |
SFT Method
Standard masked-diffusion SFT (matches LLaDA's published approach): concatenate <|user|>{message}<|endturn|><|assistant|>{response}<|endturn|> per turn, mask only assistant-turn content during the forward diffusion process, keep user turns and role markers as fixed, always-visible context. Verified during development that this boundary is never violated — role markers and user tokens cannot be selected for masking under any sampled mask rate, checked across hundreds of random trials including near-maximal masking rates.
Masking-rate range: t ∈ [0.5, 1.0], not the [0, 1] used for v1's pretraining. This is a deliberate, literature-informed choice, not the default: a recent ablation (arXiv 2601.22450) found that for generative tasks specifically, training concentrated on high-noise t outperforms both uniform and mid-range "signal-rich" windows, because low-t training lets the model lean on partial context "hints" that simply aren't present when generation starts from a fully-masked response — which is exactly the condition this model faces at inference. The same paper found the opposite ranking for discriminative multiple-choice tasks, so this choice specifically favors conversational generation over benchmark-style scoring, on purpose.
Data
HuggingFaceTB/smoltalk,
everyday-conversations config: 2,260 short, simple dialogues, 251,507 total assistant-turn tokens. Chosen deliberately small and simple — v1 already showed weak general capability (see its model card's ARC-Easy result), so a large or complex SFT set would likely have been wasted rather than absorbed.
Hyperparameters
| Setting | Value |
|---|
| Optimizer | AdamW |
| Peak LR | 5e-5 |
| LR schedule | Cosine, 50-step warmup, decays to 5e-6 |
| Weight decay | 0.01 |
| Epochs | 3 (285 steps total) |
| Batch size | 24, auto-probed against GPU memory |
| Precision | bfloat16 |
| Hardware | Single NVIDIA L4 (24GB) |
| Training time | Under 4 minutes |
Loss curve
Dropped sharply and cleanly: ~45 at step 0 to ~6-7 by the end of epoch 1, staying in that range through epochs 2-3 with the same per-step noise pattern as v1's pretraining (expected -- it's the same 1/t-reweighted objective). Fast, stable convergence is expected here: SFT is adapting an already-trained model to a narrow, simple task, not learning language from scratch.
What we found
The sampler bug -- found, fixed, confirmed
Early SFT samples (using the same noise-perturbed unmasking already documented as v1's fix) collapsed into literal phrase repetition: "I help you! I help you! I can I help! I can can I help you're!" This is a known, actively-studied failure mode for masked diffusion models specifically -- not unique to this checkpoint. Multiple independent papers confirm masked diffusion models degrade into repetition under exactly these conditions (short outputs, low step counts), in contrast to autoregressive models, which tend to hallucinate rather than repeat under similar stress.
Fix: presence-penalty logit adjustment, adapted from the standard autoregressive-decoding mitigation. At each denoising step, before sampling, we count how many times each candidate token already appears in the response span generated so far and subtract presence_penalty x count from that token's logit -- discouraging the model from repeatedly committing to the same token once it's already been used. Scoped to the response span only, so the model isn't penalized for words the user's own message happens to repeat. presence_penalty=1.2 is the default, matching the empirical sweet spot found in prior work on this exact mechanism.
We verified this fix works -- both on a synthetic model built to reproduce the collapse pattern (repetition dropped monotonically as the penalty increased, with no signs of a new failure mode appearing) and on this checkpoint directly: post-fix samples contain zero verbatim phrase loops, compared to the reliable collapse before the fix.
What's left: genuine incoherence, not a decoding artifact
With the repetition bug fixed, real samples from this model look like this:
User: Hi there, how are you?
Assistant: Hello! How! I! You took you today? I today? I hope you help your mind or support your options if something it is welcome.). This can start with you together, be trying to see, and get up with the previous or and thenup. You want to feel great. I can also improve
Individual word choices are locally plausible ("Hello", "welcome", "help", "improve" -- all reasonable conversational vocabulary), but they don't compose into grammatical sentences. This is not the same failure as the repetition bug, and fixing the repetition did not reveal hidden coherence underneath it -- it exposed that the underlying token-level predictions, once no longer masked by phrase-looping, simply don't assemble into coherent multi-word structure yet.
We think this is most likely a straightforward capacity/data ceiling rather than a new bug: v1's own model card already reports at-chance ARC-Easy performance and a training budget (1.75B tokens for 130M params) well below typical pretraining ratios. SFT on top of a base model that hasn't yet learned much beyond local token-level fluency doesn't manufacture grammatical competence that wasn't there -- it can only elicit and format whatever competence already exists.
Evaluation
ARC-Easy
Re-run after SFT to check whether the vocabulary extension, embedding resize, or fine-tuning itself affected the base model's (lack of) discriminative capability -- this is an independent question from the generative-coherence issue above, since ARC-Easy is scored via masked-span ELBO comparison, not generation.
| Metric | Result |
|---|
| Accuracy (ELBO-ranked, 4-way) | (fill in after running) |
| Random baseline | 25.00% |
(See ExpIvme-DiffusionConversate-v1's model card for the pre-SFT result and full methodology notes -- same ELBO-based scoring approach, unchanged here.)
Inference
1import torch
2import torch.nn.functional as F
3from transformers import AutoModel, AutoTokenizer
4
5model = AutoModel.from_pretrained(
6 "IvmeLabs/ExpIvme-DiffusionConversate-v1-Instruct", trust_remote_code=True,
7).cuda().eval()
8tokenizer = AutoTokenizer.from_pretrained(
9 "IvmeLabs/ExpIvme-DiffusionConversate-v1-Instruct", trust_remote_code=True,
10)
11mask_id = model.config.mask_token_id
12user_id = model.config.user_token_id
13assistant_id = model.config.assistant_token_id
14endturn_id = model.config.endturn_token_id
15
16@torch.no_grad()
17def chat(user_message, max_response_len=64, steps=32, temperature=1.0,
18 gumbel_temp=1.0, presence_penalty=1.2):
19 prefix_ids = [user_id] + tokenizer.encode(user_message) + [endturn_id, assistant_id]
20 input_ids = torch.tensor(
21 [prefix_ids + [mask_id] * max_response_len], dtype=torch.long, device="cuda",
22 )
23 prefix_len = len(prefix_ids)
24
25 for step in range(steps):
26 logits = model(input_ids=input_ids).logits
27
28 if presence_penalty > 0:
29 response_span = input_ids[:, prefix_len:]
30 visible = response_span.masked_fill(response_span == mask_id, -1)
31 counts = torch.zeros(1, model.config.vocab_size, device="cuda")
32 valid = visible[0][visible[0] >= 0]
33 if len(valid) > 0:
34 counts[0].scatter_add_(0, valid, torch.ones_like(valid, dtype=torch.float))
35 logits = logits - presence_penalty * counts.unsqueeze(1)
36
37 probs = F.softmax(logits / temperature, dim=-1)
38 sampled = torch.multinomial(probs.view(-1, probs.size(-1)), 1).view(input_ids.shape)
39
40 is_masked = input_ids == mask_id
41 n_masked = is_masked.sum().item()
42 if n_masked == 0:
43 break
44
45 frac_remaining = 1.0 - (step + 1) / steps
46 denom = max(1 - step / steps, 1e-6)
47 n_to_unmask = min(max(1, int(n_masked * (1 - frac_remaining / denom))), n_masked)
48
49 conf = probs.gather(-1, sampled.unsqueeze(-1)).squeeze(-1)
50 log_conf = torch.log(conf.clamp(min=1e-9))
51 u = torch.rand_like(conf).clamp(min=1e-9, max=1 - 1e-9)
52 gumbel_noise = -torch.log(-torch.log(u))
53 score = (log_conf + gumbel_temp * gumbel_noise).masked_fill(~is_masked, float("-inf"))
54
55 topk = torch.topk(score.view(1, -1), k=n_to_unmask, dim=-1).indices
56 update_mask = torch.zeros_like(is_masked).view(1, -1).scatter_(1, topk, True).view(is_masked.shape)
57 input_ids = torch.where(update_mask, sampled, input_ids)
58
59 response_tokens = input_ids[0, prefix_len:].tolist()
60 if endturn_id in response_tokens:
61 response_tokens = response_tokens[:response_tokens.index(endturn_id)]
62 return tokenizer.decode(response_tokens)
63
64print(chat("Hi there, how are you?"))
trust_remote_code=True is required. No .generate() support -- this sampler is the real inference path.
Limitations
- Output is not reliably grammatical. This is the headline limitation, not a footnote. Expect locally-plausible word choice without coherent sentence structure. See "What we found" above.
- Inherits all of v1's limitations: not expected to have meaningful factual/knowledge capability, English only, 1024 token context, no controlled AR baseline exists for comparison.
- 3 epochs / 2,260 conversations is a light SFT touch. We have not tested whether more data, more epochs, or a larger/more diverse SFT set would meaningfully help, versus the ceiling being set by v1's pretraining rather than SFT scale.
- The
t ∈ [0.5, 1.0] masking range was chosen for generation quality, not benchmark performance -- if you re-purpose this checkpoint for discriminative scoring, that choice works against you, not for you.
- Default
presence_penalty=1.2 fixes verbatim repetition specifically; it does not and cannot fix grammatical incoherence, which is a different failure mode with a different (likely: more/better pretraining) fix.
What's Next
The honest next step, in our view, is not more SFT tricks -- it's revisiting v1's pretraining. We separately identified that v1's pretraining used an unclipped t ~ U(0,1) masking schedule, which recent literature identifies as wasting real training budget on low-signal mask-rate extremes; a corrected schedule plus more training tokens (masked diffusion models tolerate data repetition far better than autoregressive models, per controlled studies showing a data-reuse half-life over 15x longer) is the more promising lever than further SFT-side fixes on top of an undertrained base.
Citation
1@misc{expivme-diffusionconversate-v1-instruct,
2 author = {IvmeLabs},
3 title = {ExpIvme-DiffusionConversate-v1-Instruct},
4 year = {2026},
5 publisher = {Hugging Face},
6 url = {https://huggingface.co/IvmeLabs/ExpIvme-DiffusionConversate-v1-Instruct}
7}
Built by IvmeLabs. This one didn't work yet, and we're saying so.
