A merged, ready-to-serve BF16 checkpoint of NVIDIA's Nemotron 3.5 30B-A3B, tuned with
Antislop and FTPO to suppress the overused
lexical patterns that make LLM prose recognisable as LLM prose.
43.2% of overused patterns suppressed across six domains, 67.8% on creative writing,
with no capability regression on any benchmark measured.
V1 suppressed
66.41% on creative writing prompts and was measured only there. V2 keeps the method and
widens the scope:
V1
V2
Prompt domains
creative writing only
six (creative, real-user, factual QA, explanatory, instruction-following, conversational)
Held-out eval
400 prompts
952 prompts
Human baseline
Reddit writing
3.13B prose chars, 12 registers
Pipeline iterations
2
5
Banlist
4,267 patterns
8,033 patterns
Typography profiling
none
28 features
Preference pairs
~13,000
15,000
V1 and V2 headline numbers are not directly comparable. V1 scored against its own
4,267-pattern banlist; V2 scores against a banlist 1.9x larger. On V1's exact 388 creative
prompt IDs, V2 measures 67.78% — level with V1 while also covering five more domains.
V1's benchmarks were also run with thinking on and V2's with thinking off, so the
absolute MMLU and IFEval figures differ between cards for protocol reasons, not model
reasons.
What this is
Language models overuse a characteristic vocabulary: stock dialogue tags, a fixed set of
atmospheric props, and a long tail of default names. Antislop identifies those patterns
for this specific model by comparing its output against a human-written baseline, then
uses a backtracking sampler to catch the model reaching for one and record what it should
have reached for instead. Those records become preference pairs, and FTPO (Final Token
Preference Optimization) trains the preference into the weights so it persists with the
sampler switched off.
V2 adds a second axis: typography and punctuation. V1's n-gram analysis dropped every
non-letter token before counting, so it was structurally blind to em dashes, ellipses,
curly quotes, emoji, and markdown. V2 profiles 28 such features against human rates and
fires bans proportionally — with probability p = 1 - human_rate / model_rate, so the
residual lands at human parity rather than zero. Suppressing a tic to far below human usage
is its own detectable signature.
Results
952 held-out prose prompts, identical sampling for both models (temp 1.0, top_p 1.0,
top_k 50, min_p 0.01), Antislop sampler off. These numbers reflect what training
changed in the weights, not what a sampler suppresses at inference.
Metric
Baseline
FTPO V2
Δ
Banlist suppression (overall)
0%
43.21%
+43.21
Banlist suppression (creative, n=388)
0%
67.78%
+67.78
Banned-pattern rate per 100k chars
203.28
115.45
−43.2%
MMLU (600 q)
0.7583
0.7567
−0.16pp
GSM8K (250 q)
0.9040
0.9160
+1.20pp
MMLU-Pro (600 q)
0.7000
0.7200
+2.00pp
IFEval prompt-strict (323)
0.7802
0.7709
−0.93pp
IFEval instruction-strict (323)
0.8299
0.8209
−0.90pp
HumanEval+ pass@1 (164)
0.8598
0.8537
−0.61pp
Lexical diversity (index, baseline=100)
100.00
99.98
−0.02
MATTR-500
0.5567
0.5634
+0.0067
No benchmark difference is meaningfully distinguishable from zero at these sample sizes.
HumanEval+ is worth calling out: V2 deliberately removed all code and math prompts from
the training pool, and code ability is unchanged (one problem out of 164).
Suppression by domain
Domain
n
Baseline /100k
V2 /100k
Suppression
creative
388
218.13
70.29
67.78%
real_user
275
199.10
142.56
28.40%
instruction_following
71
49.80
36.00
27.71%
factual_qa
106
230.63
197.41
14.40%
explanatory
82
237.60
213.54
10.13%
conversational
30
249.92
238.30
4.65%
overall
952
203.28
115.45
43.21%
Suppression is strongest where the slop is densest and the register is furthest from the
model's assistant default. Conversational is the weak case at 4.65%.
Typography: a split result
This is the new capability in V2, and it works for some feature classes and fails for
others. Rates per 100k prose chars against the human baseline, code stripped:
Improved:
Feature
Human
Baseline
V2
Before
After
not just X but Y
1.10
2.17
1.12
1.97x
1.02x
curly apostrophe
33.83
122.61
29.58
3.62x
0.87x
curly double quotes
40.19
46.56
14.32
1.16x
0.36x
ellipsis character
2.20
3.68
0.28
1.67x
0.13x
...
15.31
24.63
18.06
1.61x
1.18x
markdown hrule
10.25
12.97
11.10
1.26x
1.08x
Made worse:
Feature
Human
Baseline
V2
Before
After
unspaced em dash
2.07
80.58
108.38
38.9x
52.3x
markdown header
1.31
60.05
61.07
45.9x
46.7x
bullet list item
3.89
140.54
148.83
36.1x
38.3x
emoji
0.14
2.59
2.85
18.6x
20.4x
The split is not random. Features with an exact substitute get fixed; features requiring
a structural choice get worse. A curly apostrophe maps to ', so FTPO's "chosen" token is
well defined and learnable. An em dash's replacement is a comma, a colon, a period, or a
rewritten clause depending on the sentence — the chosen token is effectively arbitrary and
the gradient carries no consistent signal. Markdown headers, bullets and emoji fail the same
way.
If you need em dashes suppressed, use the Antislop sampler at inference. Proportional firing
held them at ~1.0x human during generation; it is the weight-baking step that fails.
Note also that several of the wins overshoot to below human rates. Structural diagnostics
moved toward human on five of six measures (sentence length mean/sd/cv, conjunction openers,
single-sentence paragraphs) and away on one (paragraph length).
What gets banned
The banlist is not a hand-written style guide. It is 8,033 patterns measured as overused in
this model's own output relative to human prose: 4,342 n-grams (2,218 bigrams, 2,124
trigrams) and 3,990 slop phrases.
Em dashes and markdown structure get worse, as above.
The 43.21% figure understates true slop suppression. The banlist contains prompt
echoes that were never slop: character names (elara, kaelen, huck, sansa), code
tokens (pygame, cloudformation), and non-English function words. These inflate the
denominator. A prompt-echo filter is the main outstanding fix.
No writing-quality judge was run. V1 reported an LLM-judge comparison; V2's judge and
τ-bench runs were blocked by an invalid API key and are simply absent, not null results.
Conversational and explanatory registers barely move (4.65% and 10.13%).
Suppression is a reduction, not a hard filter.
Usage
Requires trust_remote_code=True for the nemotron_h architecture.
python
1from transformers import AutoModelForCausalLM, AutoTokenizer
23model_id ="thoughtworks/Nemotron-3.5-30B-A3B-Antislop-FTPO-V2"4tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)5model = AutoModelForCausalLM.from_pretrained(6 model_id, torch_dtype="bfloat16", device_map="auto", trust_remote_code=True7)89messages =[{"role":"user","content":"Write the opening of a story about a lighthouse keeper."}]10text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)11out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=512)12print(tok.decode(out[0], skip_special_tokens=True))
Note: on transformers 5.14.x, apply_chat_template(..., tokenize=True) under the
TokenizersBackend tokenizer class returns a truncated sequence. Render with
tokenize=False and tokenize the resulting string, as shown above. This affects the
upstream base model identically and is not specific to this checkpoint.
The chat template defaults to thinking on, emitting an open <think> block. Pass
enable_thinking=False to apply_chat_template (or
chat_template_kwargs={"enable_thinking": false} over an OpenAI-compatible API) for direct
prose. All prose evaluation above was run with thinking off.
vLLM
Stable vLLM 0.27.1 and later support NemotronHForCausalLM natively.
The entire fine-tune lives in lm_head as a perturbation of mean 3.3e-05. A default
Q4_K_M puts the output tensor at Q6_K, whose quantization noise is large enough to swamp a
delta that small and quietly undo the training. Expect ~24.9 GB at 6.29 BPW rather than the
usual ~19 GB, partly for this reason and partly because 134 of 401 tensors fall back to
q5_0 (moe_intermediate_size 1856 is not a multiple of the Q4_K block size).
Training
Method
Antislop, then FTPO (Final Token Preference Optimization)
Target modules
lm_head only
LoRA rank / alpha / dropout
256 / 256 / 0.05
Steps / epochs
434 / 2
Learning rate
5.45e-05 (auto-scaled)
Final train loss
2.065
Pipeline iterations
5
Generation
4,511 prompts per iteration
Preference pairs
15,000 (12,000 lexical, 3,000 typography)
Banlist size
8,033 patterns + 28 typography features
Hardware
1x H200, about 6 h for the FTPO stage
Training-set filtering removed 11,900 pairs of rare content vocabulary (Zipf < 2.5, e.g.
proper nouns mistaken for slop), 5,315 non-Latin-script pairs, and 2,089 displacement-only
pairs where the model had merely rerouted one suppressed tic into another.
Model architecture
Inherited unchanged from the base model. A hybrid Latent Mixture-of-Experts design with
interleaved Mamba-2 and MoE layers plus select attention layers, and Multi-Token Prediction
(MTP) layers for speculative decoding. 52 layers (23 Mamba-2, 23 MoE, 6 attention), 128
experts with 6 active plus 1 shared, 30B total parameters, 3B active. Context length up to
262,144.
License
Released under OpenMDW-1.1, matching the license NVIDIA
applies to the public Nemotron 3.5 Lightning releases. The Antislop framework itself is
MIT-licensed.
Citation
The method was published at ICLR 2026:
bibtex
1@inproceedings{paech2026antislop,
2 title = {Antislop: A Comprehensive Framework for Identifying and Eliminating
3 Repetitive Patterns in Language Models},
4 author = {Paech, Samuel and Roush, Allen and Goldfeder, Judah and Shwartz-Ziv, Ravid},
5 booktitle = {The Fourteenth International Conference on Learning Representations},
6 year = {2026},
7 url = {https://openreview.net/forum?id=gLcyM1khyp},
8 eprint = {2510.15061},
9 archivePrefix = {arXiv},
10 primaryClass = {cs.CL}
11}
