Synthetic training data is a plague.
So, there's a lot of shitty RP slop out there. Like, a lot. Basically every model is infected with it, and all the tunes and loras and datasets everyone's using to "enhance" base models is infested. Benchmarking, which is a garbage practice as it's done, results in convergence on roughly the same wannabe OpenAI text in a frenzy of measuring the wrong things for the wrong reasons. There's a shit-ton of bad amateur smut written by 19-year-olds and that keeps coming up. This trash was in Llama2 and it's still in whatever these new huge models are. Claudeisms out the yang. OpenAIisms left and right. Generally repetitive "barely above a whisper" type shit.
Well, these days most inference systems support long lists of token sequences to ban. Well, at least KCPP and LCPP do. I have no idea what you ExLlama clowns have because you build software for people who own 4 GPUs. Anyway, the point being what you want is to be able to paste in a big honkin pile of banned strings so that the model stops slopping at you and stops sounding like it has LimaRP stuck between its teeth like rotting spinach.
If you use kcpp, you have to bump the max banned tokens accepted, but it's just in the python script right at the top. You don't even have to recompile. I use 8192 because, fuck it, it's my hardware. The performance is my problem. LostRuins doesn't like a list that big because it'll slow down some mysterious "public koboldcpp instances" that must be, like, 0.0001% of all kcpp users. But it works fine. Just bump the number.
Then paste this whole damned thing into ST. Use the global banlist so you don't have to re-paste it into every sampler config you own.
After that? Enjoy half as much LLM slop!
Design Principles
Slop output comes in two forms. One is simple token spam (audible pop, Seraphina, sense of) from both attractors in the training corpus and OAI/Anthropic-tainted injections to hit benchmarks. In general these come from people using synthetic training data while fine-tuning and model training or training from an internet corpus. It's a dead internet now, too much of the text is cheap OpenAI crud.
The other form of slop is harder to kill. It's the tendency of models to tell stories and write text in certain ways. Rule of three and overuse of em-dash are major examples. Over focusing on scent (what's with the fuckin' ozone and sweat?) is a personal bugbear. Hands on chests drive me nuts. Two-phrase sentences in particular are one of the worst: "I try to kill this cadence, failing more than succeeding." A lot of this is fundamental. Other antisoc-qa-assoc work is making progress on it since mere sequence banning won't put a dent in it.
A subset of that insidious phrasing slop is cliched sequences of events or plots as well as structure in general. Smirks quirk and then heads tilt. Proximity causes smelling things all the time. Those particular smells are always two things and that's a glorious example of what OAI taint really looks like. The problem here is that just banning "that scent of" doesn't make the output not about two scents. Event proximity and sequences are hard as fuck to ban with pure token lists.
The entire upper section of the banlist (up to "ZZZMARKERZZZ") is hand-curated as thoughtfully as possible. Any time a sequence is added to it, prior model output is reviewed to see if similar non-identical squences are being spammed. That is to say a human who knows both how language works in general and also who has taste chooses sequences to ban. I can smell slop a mile away and I'm doing my best here.
The lower section is taken from public slop listings. To keep total token count down, a lot of shit from the slop listings is left out: vampire fic shit and male-presenting role play characters in particular. Some related to settings that are very specific to fic scenarios I'm not going to run into. It may benefit from more cleanup.
The design of this list is such that if possible a single token or word isn't being banned outright. This leaves the model free to use words in non-cliche or overwrought ways while quashing repetitive OAI/Lima/Llama-isms that have been baked into every LLM. In some cases, I've banned words outright because there's no better way to avoid it setting off related sequences. In others, it's because the word itself is a shitty one I never want to see.
There are many entries where a bunch of sequences being banned starts with all the same word. All of them are deliberate choices about permitting words without having to read obnoxious overused sequences. When at all possible the shortest sequence is being banned to allow models to express concepts with less cliched phrasing.
Biases
There are not only inherent biases in this list of tokens, but there are selected biases: the vast majority of the banned strings are cliches that come up when the LLM is role-playing (or ERP'ing) and/or writing as one or more cis females. There are some strings in there which aren't chosen around that criterion. In particular those are strings likely to have come up in ao3 and/or RP forum datasets.
That is to say: if you use these strings for banning tokens, but you'll probably be disappointed if the LLM is writing or using male characters or if you really fucking love vampire romance. There are heaps upon heaps of human-generated and synthetic slop that won't have come up in my life at all and I'd never even know it was there. At that point you're relying on the banned phrase list at the bottom.
This bias is intentional. I will spend zero time seeking out that kind of slop in order to kill it. We can all wish someone else, someone affected by it, will come along and add to it. Ideally at the bottom or as an include so I can truncate it when I use this data. Pull requests are accepted. Please don't trash the actual list I use as a daily driver.
Credits
- Me. The top half of the file is shit I banned by hand over the course of months because I was tired of seeing it.
- AntiSlop-Sampler's pile of common strings and phrases, and it occupies the remainder of the file. It's largely just pasted in, so there's a bunch of stuff that's statistically likely to come up only if you read a lot of bad ao3 vampire fic. I've never seen, like, two thirds of it.
- antisoc-qa-assoc for rummaging around in latent spaces and for doing actual engineering-based "prompt engineering" by engineers who know what engineering is. And who have actually read a book that doesn't suck, who have written books, who know what writing is.
Afterword
Hey LostRuins! Hi! I love your work. Also, my arbitrarily-chosen 4096 ban tokens wasn't even close to the right limit. Your default of 768 isn't enough. Slop is that bad out there. I was right, am right about the need for massive sequence bans, and you were wrong. I love you anyway.
License
None: Dedicated to the public domain. All rights waived. Any indication of other licensing in this document is purely because HF license metadata is fucked; only the dedication to public domain applies. The person who provides this data doesn't live anywhere that authoritarian concepts like "moral rights" or un-waivable rights exist. Therefore they do not apply.
We're not done, we're just slow.
