N.B. To use this model with a convenient interface
check us out on Github!
FELA-Moderation: CPU native content moderation and PII detection
FELA-Moderation reads a piece of text and does two things with it. It scores how toxic the text
is across a set of moderation categories, and it points out the spans that look like personal
information: names, emails, phone numbers, credit cards, and the like. It is small enough to run
on a plain CPU, so that whole moderation and PII pass can live inside your own service or app
instead of going out to a moderation API and billing you per call.
It reads text as raw bytes rather than words. That has a nice side effect: there is no
vocabulary file to ship, and the model is not locked to any one language.
The model covers a 19 category moderation taxonomy, the 6 native Jigsaw labels, byte level PII over
56 entity types, and four extra classifier heads for spam, jailbreak, NSFW severity, and targeted
identity. It is about 18.9M parameters and ships as model.safetensors (fp32) plus an int8 export
of roughly 19 MB for on device and WASM serving. We ship weights only and never redistribute the
source text.
What goes in, what comes out
- Input: any UTF-8 text string. It is byte tokenized to
[CLS] b0 b1 ... (each byte is its own
token, ids 0 to 255, plus 256=PAD, 257=CLS, 258=SEP) and truncated to 512 tokens. The
encode_text and pad_batch helpers in modeling.py do this for you.
- Output 1 (taxonomy): 19 independent probabilities over the moderation taxonomy (the 11 OpenAI
aligned categories plus identity_attack, insult, profanity_obscene, threat, sexual_explicit,
severe_toxicity, dehumanize, incitement_violence). Full list in
config.json. Sigmoid, multi
label.
- Output 2 (jigsaw): 6 independent probabilities over the native Jigsaw labels (toxic,
severe_toxic, obscene, threat, insult, identity_hate). Sigmoid, multi label.
- Output 3 (pii): per token BIO tags over 56 entity types (113 tags: O plus B- and I- per entity).
- Output 4 (safety heads): spam / scam / phishing, jailbreak / prompt injection, NSFW severity, and
targeted identity.
The head is chosen with task= in the forward call (taxonomy, jigsaw, pii, spam,
jailbreak, nsfw, identity, or both).
Why we built it this way
The model is built on the same Fourier Neural Operator the rest of the FELA family uses: a
filter that mixes the whole sequence at once in the frequency domain, with no attention matrix
that balloons as the text gets longer.
Alongside it rides a gated linear attention layer that picks up the content dependent details, and
its working memory stays a fixed size. Because none of this leans on the usual all pairs attention,
a moderation plus PII pass stays fast on an ordinary CPU. That speed is the whole cost argument
against paying a per call API.
It is deliberately small, about 18.9M parameters. One shared backbone feeds several light heads
(taxonomy, Jigsaw, PII, spam, jailbreak, NSFW, identity), and the byte vocabulary is only 259
symbols, which quantizes to int8 well.
Performance
Format and footprint. The fp32 weights ship as model.safetensors (about 75 MB). INT8 dynamic
quantization of the Linear layers shrinks the model to roughly 19 MB while the FFT and GLA einsum
paths stay in float; the int8 export holds accuracy on the held out slices (see below). Because the
encoder carries no KV cache and a fixed size state, working memory does not grow with input length.
Accuracy
Held out numbers are on permissively licensed splits: Jigsaw on a fixed leak free slice of the
Jigsaw training data, Civil Comments validation for taxonomy, and nemotron plus synthetic for PII.
Headline held out:
| head | AUROC / acc | note |
|---|
| PII (byte token acc) | 0.96 | |
| Jigsaw (mean AUROC) | 0.94 | |
| taxonomy: hate / harassment | 0.84 / 0.77 | |
| taxonomy: sexual | 0.67 | data limited, explicit data not in the training set |
| spam / jailbreak / nsfw | 0.94 / 0.88 / 0.82 | |
| identity | 0.64 | weaker, use as a soft signal only |
The int8 export holds these numbers. Final training loss 0.69. The model's advantage is privacy on
local hardware, speed, and clean provenance at roughly comparable quality, not topping a specialist
leaderboard.
The one honest gap is the sexual category (0.67). It does not close with more training: it is a data
limitation. Explicit sexual data was deliberately kept out of the training set, so that category is
supplied only by synthetic examples. We report it rather than hide it.
How to run it
1import torch
2from modeling import load_model, encode_text, pad_batch
3
4m = load_model("model.safetensors")
5ids, _ = encode_text("You are awful. Email me at jane.doe@example.com", m.cfg.max_len)
6input_ids, mask = pad_batch([ids], m.cfg.max_len)
7out = m(input_ids, mask, task="both")
8
9tax = torch.sigmoid(out["taxonomy"][0]) # 19 taxonomy probabilities
10jig = torch.sigmoid(out["jigsaw"][0]) # 6 Jigsaw probabilities
11pii = out["pii"][0].argmax(-1) # per token BIO tag id
Loading with standard tooling
The repo ships config.json (architecture hyperparameters plus the full label and tag maps and the
per category toxicity thresholds) and a self contained modeling.py with a load_model /
from_pretrained entry point:
1from modeling import load_model
2m = load_model("/path/to/weights_dir") # OR
3m = load_model("lowdown-labs/fela-moderator")
The fp32 weights are shipped as model.safetensors.
Serving artifacts
model.safetensors plus config.json for the safetensors load path (fp32).tier_full_int8.safetensors plus tier_full_scales.json for the roughly 19 MB int8 export.manifest.json and NOTICE.txt describe the tiers and the training source attributions.verify.py runs a fixed sample input and checks the output shapes and a verification value.
For serving at scale, use the separate CPU native FELA server
(https://github.com/Lowdown-Labs/fela_server). It runs this model on CPU with no GPU required.
Training data
We train on the sources below and distribute model weights only. No source text or derived dataset
is redistributed, so the CC-BY-SA-3.0 status of Wikipedia derived comment text (under Jigsaw and
Civil Comments) never triggers share alike. data/licenses.py carries the verified license of every
source, and NOTICE.txt records the attributions.
- PII:
nvidia/nemotron-pii (CC-BY-4.0, 55+ types, character offset spans) plus
gretelai/synthetic_pii_finance_multilingual (Apache-2.0) plus synthetic generation via
Faker (MIT) and the Presidio generator (MIT, driven from Faker).
- Toxicity and taxonomy:
google/civil_comments (CC0-1.0), google/jigsaw_toxicity_pred (CC0-1.0
annotations), ucberkeley-dlab/measuring-hate-speech (CC-BY-4.0),
allenai/real-toxicity-prompts (Apache-2.0).
- Spam:
redasers/difraud (MIT) and ucirvine/sms_spam (CC-BY-4.0).
- Jailbreak and prompt injection:
cyberec/llm-prompt-injection-attacks (Apache-2.0),
deepset/prompt-injections (Apache-2.0), jackhhao/jailbreak-classification (Apache-2.0),
Lakera/gandalf_ignore_instructions (MIT).
- NSFW:
google/civil_comments sexual_explicit signal, with
mmathys/openai-moderation-api-evaluation (MIT) used for evaluation only.
- Targeted identity:
ucberkeley-dlab/measuring-hate-speech (CC-BY-4.0).
Full split definitions, size assertions, and per source license verdicts are reproduced in
train.py and data/licenses.py.
Expanded taxonomy (19). Beyond the 11 OpenAI aligned categories the model adds identity_attack,
insult, profanity_obscene, threat, sexual_explicit, severe_toxicity, dehumanize, and
incitement_violence, each sourced from a permissive dataset. Growing the head does not change the
byte level backbone, so the int8 size is unchanged. Five categories (self_harm, self_harm_intent,
self_harm_instructions, sexual_minors, violence_graphic) have no permissive source: they are masked
from real data and supplied by synthetic and reviewed examples, and reported separately so coverage
is never overclaimed.
Byte level PII BIO with exact spans. Character offsets from the real sources and byte offsets
from the synthetic generator are aligned to byte tokens in data/bio.py (verified 100% correct on
1,251 spans). The head is trained across context lengths (128 to 8192 bytes) with PII needles buried
in benign carrier text, so it finds a single email or one toxic span inside a long document.
How it was trained
The model was trained CPU only via DiLoCo (data parallel with infrequent parameter averaging
coordinated through S3), on commodity CPU instances with no GPU. Weights checkpoint to S3
periodically, so a spot interruption resumes rather than restarts. The taxonomy and Jigsaw heads
train every step; the spam, jailbreak, NSFW, and identity heads rotate. Run
python train.py --smoke for an offline end to end check that trains a few steps on synthetic data
and writes the full artifact set.
The model took about two hours on a single CPU instance, on the order of 0.05 kg CO2e - a few phone
charges of energy, and millions of times less than training a frontier language model.
Intended use, limitations, and safety
What it is for: the moderation and PII inference core inside an on premises or on device text
pipeline, where sending user text to a third party API is a cost or privacy problem. It flags toxic
content and personal information for a downstream policy layer.
What it is not for: this is a small model and a decision support signal, not a final arbiter. Do not
use it as the sole gate on user safety without human review and per category threshold tuning
(sensible starting thresholds ship in config.json).
Known limitations:
- The taxonomy head is scored on validation, not a held out competition test; treat those numbers as
in distribution. The sexual category is data limited (0.67), and the targeted identity head is
weaker (0.64) and should be used as a soft signal only.
- English centric supervision: the byte tokenizer is multilingual by construction, but the training
labels are predominantly English, so non English accuracy is not established.
- The PII head reflects the nemotron and Presidio aligned entity catalog and its distribution;
entity types and formats outside that corpus may be missed.
How to cite
1@misc{lowdownlabs_felamoderation,
2 title = {FELA-Moderation: CPU native byte level content moderation and PII detection},
3 author = {Lowdown Labs},
4 year = {2026},
5 note = {Model card}
6}
You should also cite the datasets and baselines used (Jigsaw / Conversation AI, nemotron,
civil_comments, measuring-hate-speech, real-toxicity-prompts, and Detoxify / Presidio); full
references and licenses are in data/licenses.py and the references below.
Acknowledgements and references
- Fourier Neural Operator: Li, Z., et al. (2021). Fourier Neural Operator for Parametric Partial
Differential Equations. ICLR. https://arxiv.org/abs/2010.08895
- Gated Linear Attention: Yang, S., et al. (2024). Gated Linear Attention Transformers with
Hardware-Efficient Training. https://arxiv.org/abs/2312.06635
- Byte level modeling: Xue, L., et al. (2022). ByT5: Towards a Token-Free Future with Pre-trained
Byte-to-Byte Models. TACL. https://arxiv.org/abs/2105.13626
- PyTorch: Paszke, A., et al. (2019). NeurIPS. https://arxiv.org/abs/1912.01703
Model family
This is part of the FELA family from Lowdown Labs: one Fourier Neural Operator architecture across
many modalities, all CPU native and subquadratic. This repo is pushed as lowdown-labs/fela-moderator.
Sibling repos share no weights, so none carries a base_model link.
License
Apache-2.0. The model weights and the code are both under Apache-2.0 (see LICENSE and NOTICE.txt): no
separate model license and no non commercial restriction. Every training source is permissively
licensed and commercial safe. We distribute weights only, never the source text.
