This repository publishes a headless, multilingual-input, classification-ready four-layer Qwen3.5 text backbone. The root model has no task labels and no classification head. It is a compact starting point for user-trained single-label or multilabel classifiers, not a chat model or a ready-made universal classifier.

The repository follows one integrated 1-1-1 layout:

Repository root: task-agnostic four-layer base trained on unlabeled general text.
models/semeval-propaganda/: ready-to-use SemEval-derived propaganda classifier preserved as a separate task-specific lineage.
distillation/: reusable 24L→8L→6L→4L platform for user-owned data.

Model at a glance

Item	Root model
Upstream	`Qwen/Qwen3.5-0.8B`
Runtime class	`Qwen3_5TextModel` in Transformers 5.13.0
Text layers	4: `linear_attention → full_attention → linear_attention → full_attention`
Hidden / FFN	1,024 / 3,584
Vocabulary	trimmed Qwen3.5 tokenizer, 128,000 entries
Root parameters	210,889,024
Root BF16 weights	421,783,648 bytes (402.29 MiB)
Output	hidden states; no LM head, task head, labels, or thresholds
Task-free KD data	4,096 unlabeled WikiText-103 raw rows
Downstream evidence	one English 14-label SemEval-derived transfer probe

Load the headless base

python

from transformers import AutoModel, AutoTokenizer

model_id = "mp-juuuns/qwen35-standalone4l-classification-base"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)
backbone = AutoModel.from_pretrained(model_id, trust_remote_code=False)

The published root weight file contains no score.weight. A fresh task head and labeled fine-tuning are required before outputs have class meaning.

Make your own classifier

bash

git clone https://huggingface.co/mp-juuuns/qwen35-standalone4l-classification-base
cd qwen35-standalone4l-classification-base/distillation
python -m pip install -e .

qwen35-distill finetune \
  --checkpoint .. \
  --train examples/multilabel_train.jsonl \
  --labels examples/labels.json \
  --mode multilabel \
  --output my-classifier-4l

For a full task-specific compression run, fine-tune a 24-layer teacher, then repeat materialize-classifier and distill-classifier for 24to8, 8to6, and 6to4. Exact JSONL formats, split rules, manifests, and commands are in distillation/README.md.

How the root weights were made

The text backbone was retained from the upstream-derived 4L model. The 128k tokenizer was selected from the verified English-focused classifier lineage, and embedding rows were remapped by token string rather than by row position. This release is a vocabulary-remapped artifact, not a new training run. Students were trained sequentially through 24→8→6→4 using hidden-boundary and final-representation matching on unlabeled text.

The task-free training set was exactly 4,096 deterministically shuffled, non-empty rows from the first locally cached WikiText-103 raw training shard. No SemEval examples, labels, logits, thresholds, classification heads, or evidence/span objectives were read while training the root weights.

KD stage	Selected source layers	Mean total loss	Mean interface loss	Mean final loss
24→8	`0,4,6,11,13,16,20,23`	0.176227	0.215628	0.166264
8→6	`0,1,3,4,6,7`	0.033821	0.044861	0.031147
6→4	`0,2,3,5`	0.047020	0.070484	0.043120

These are training-objective values, not downstream quality scores. Exact parameter counts, BF16 file sizes, hashes, and stage values are in benchmark/compression_ladder.csv and provenance/task_agnostic_stages/.

The chart reports the actual unsharded safetensors parameter counts and BF16 file sizes. It does not imply that downstream quality scales monotonically with layer count.

Why 128k vocabulary trimming?

The original root used 248,320 input-token rows. This update retains 128,000 token IDs and remaps each retained token to the corresponding row of the original embedding. The goal is to reduce edge storage and memory for the English-focused classification deployment path; it is not a claim that the model remains equally suitable for every language represented by the upstream tokenizer.

Quantity	248k root	128k root	Change
Vocabulary entries	248,320	128,000	−48.4%
Parameters	334,096,704	210,889,024	−36.8%
BF16 weight bytes	668,198,976	421,783,648	−36.9%

The remapping manifest is v128k_remap_manifest.json. It records the source hashes, token-string mapping hash, output hash, and parameter count.

Quality and efficiency evidence

The new root is headless, so its downstream quality is NR until a new task head is attached and evaluated. Existing evidence is shown for context:

248k task-free root transfer probe: Macro-F1 mean 0.587476 across three seeds on the previously opened SemEval-derived test.
128k specialized classifier lineage: Macro-F1 mean 0.587568 across three seeds on the same opened task.
These two values are not a controlled vocabulary ablation: the models differ in task-specific head/training lineage. They must not be interpreted as proof that vocabulary trimming preserves quality.

The direct efficiency result of this release is the storage/parameter reduction above. Existing CUDA resource measurements also show the 128k specialized lineage at lower peak allocation than the 248k task-free lineage, but that comparison is confounded by the separate classifier lineage and is therefore descriptive only. Raw inputs and limitations are in benchmark/figures/v128k_quality_efficiency.csv and benchmark/figures/manifest.json.

Downstream benchmark contract

Because the root is headless, it cannot be scored as a classifier without adaptation. Its task-appropriate benchmark is therefore a controlled transfer probe:

attach a fresh randomly initialized 14-label classification head;
train all parameters for five epochs on the same SemEval-derived train split;
select one global threshold on the epoch-5 calibration split;
evaluate once per seed on the already-opened 55-article test split;
compare against the same four-layer structural copy without task-free KD.

This derived task measures article-level presence of 14 propaganda techniques. It is not the official SemEval span- or fragment-level task. Results are exploratory because the public test split had already been opened.

Frozen protocol

Setting	Value
Seeds	41, 42, 43
Train / calibration / test articles	260 / 56 / 55
Test windows	434
Epochs	5, no early stopping
Input windows	max length 256, overlap stride 128
Article aggregation	label-wise maximum probability
Batch	micro 1, gradient accumulation 32, effective 32
Optimizer	AdamW, LR 2e-5, weight decay 0.01, gradient clip 1.0
Precision / device	BF16 / NVIDIA GeForce RTX 5070 Ti
Threshold selection	epoch-5 calibration Macro-F1, then Micro-F1

Overall results by seed

Initialization	Seed	Threshold	Macro-F1	Micro-F1	Exact match	Predicted positive rate
Task-agnostic KD base	41	0.15	0.58831	0.64681	0.05455	0.57143
Task-agnostic KD base	42	0.30	0.60284	0.66552	0.03636	0.40909
Task-agnostic KD base	43	0.10	0.57128	0.64653	0.00000	0.51558
Structural copy, no task-free KD	41	0.05	0.51694	0.57740	0.00000	0.71299
Structural copy, no task-free KD	42	0.20	0.53494	0.60834	0.00000	0.62078
Structural copy, no task-free KD	43	0.20	0.50238	0.58564	0.00000	0.59610

Three-seed summary

Values are mean ± sample SD across the three training seeds. The SD is not a confidence interval.

Initialization	Macro-F1	Micro-F1	Exact match	Predicted positive rate
Task-agnostic KD base	0.58748 ± 0.01579	0.65295 ± 0.01088	0.03030 ± 0.02777	0.49870 ± 0.08248
Structural copy, no task-free KD	0.51809 ± 0.01631	0.59046 ± 0.01603	0.00000 ± 0.00000	0.64329 ± 0.06161
Existing specialized 4L, separate lineage	0.58757 ± 0.00717	see nested reports	see nested reports	see nested reports

The task-agnostic KD initialization improved same-seed mean Macro-F1 by +0.06939 over the structural-copy control. This is an observed comparison under this protocol, not a significance or universal-superiority claim.

Three-seed Macro-F1 and Micro-F1 comparison

Bars are three-seed means and whiskers are sample SD. The specialized 4L point is a separate historical lineage rather than a fine-tuned child of the root base.

Task-agnostic base: per-label test results

Precision, recall, and F1 are three-seed means. Support is the fixed number of positive test articles, not a three-seed sum.

Label	Support	Precision	Recall	F1
`Appeal_to_Authority`	11	0.3959	0.5758	0.4657
`Appeal_to_fear-prejudice`	24	0.5540	0.7361	0.6278
`Bandwagon,Reductio_ad_hitlerum`	7	0.4852	0.8095	0.6007
`Black-and-White_Fallacy`	12	0.3668	0.4722	0.4054
`Causal_Oversimplification`	18	0.5429	0.7963	0.6447
`Doubt`	22	0.5224	0.9848	0.6802
`Exaggeration,Minimisation`	22	0.4621	0.7727	0.5727
`Flag-Waving`	17	0.5643	0.8235	0.6465
`Loaded_Language`	45	0.8649	0.9852	0.9209
`Name_Calling,Labeling`	33	0.7381	0.9091	0.8125
`Repetition`	23	0.5197	0.7536	0.6148
`Slogans`	14	0.5161	0.7381	0.5835
`Thought-terminating_Cliches`	7	0.3492	0.3333	0.3190
`Whataboutism,Straw_Men,Red_Herring`	10	0.2762	0.5000	0.3304

Per-seed values and sample SDs for all labels and all arms are available in benchmark/per_label_metrics.csv and benchmark/per_label_summary.csv.

Labels are ordered by their three-seed mean F1. Support remains visible because several classes have only 7–14 positive test articles.

Efficiency and resource measurements

These numbers belong to the fresh-head transfer checkpoints, not a standalone headless-root latency benchmark.

Measure	Task-agnostic transfer model
Parameters with 14-label head	334,111,040
Test inference	0.08131 ± 0.00243 s/article
Test batch	55 articles / 434 windows
Peak CUDA allocated	3,380,975,616 bytes (3.15 GiB)
Total train-and-evaluate wall time	1,591.69 s mean (26.53 min)

The timing is specific to the recorded RTX 5070 Ti, BF16, batch-1, sliding-window protocol and includes article aggregation. It is not a claim about CPU, mobile, UNO Q, or other deployment performance. Full seed-level resource fields are in benchmark/resource_metrics.csv.

These plots compare the recorded fresh-head transfer checkpoints only. Apparent differences must not be generalized to the headless root on untested hardware.

Machine-readable benchmark bundle

benchmark/BENCHMARK_CARD.md: generated benchmark report and interpretation
benchmark/summary.json: complete aggregate
benchmark/seed_metrics.csv: overall metrics per arm and seed
benchmark/arm_summary.csv: three-seed means and sample SDs
benchmark/per_label_metrics.csv: per-label values per arm and seed
benchmark/per_label_summary.csv: per-label means and sample SDs
benchmark/resource_metrics.csv: parameters, CUDA memory, wall time, and test timing
benchmark/reports/: normalized full frozen reports, including article-level probability vectors
benchmark/figures/: Matplotlib PNG/PDF figures, per-figure source CSVs, and a hash manifest

Multilingual scope

The root retains the upstream Qwen3.5 tokenizer and multilingual architecture. The official Qwen3.5-0.8B model card states expanded support for 201 languages and dialects and reports multilingual upstream benchmarks such as MMMLU, MMLU-ProX, NOVA-63, INCLUDE, Global PIQA, PolyMATH, WMT24++, and MAXIFE. This is why this repository carries the Hugging Face multilingual language and model tag.

However, this four-layer root was distilled on English WikiText and its only downstream transfer probe is English SemEval-derived data. Tokenizer coverage and inherited architecture support do not establish retained classification quality in all 201 languages. No multilingual downstream score is reported for this root release. Users should fine-tune and evaluate on each intended language and domain before making performance claims.

Separate SemEval model

The ready-to-use propaganda classifier is under models/semeval-propaganda/. It preserves the published seed-41/42/43 checkpoints and links to the original repository at immutable revision e7e0ff1.

That specialized model is not claimed to descend from this task-free base. It has its own SemEval task-specific shrink/distillation and fine-tuning history and a reduced 128k vocabulary. The updated root now uses the token-string- remapped 128k tokenizer documented above. Its complete reports remain under models/semeval-propaganda/benchmark/.

Limitations

A classification head and labeled training are required before root-model predictions have task meaning.
Task-free KD used only 4,096 rows from one English WikiText shard.
Only one downstream task and one hardware/software setting were measured.
The 55-article test split was previously opened; results are exploratory.
Per-label support ranges from 7 to 45 articles and rare-label estimates are unstable.
Three seeds describe run-to-run variation; they are not a confidence interval.
Multilingual input support has not been validated as multilingual downstream classification quality for this four-layer root.
Default layer maps are documented structural choices, not universal optima.
Long-context, calibration, robustness, fairness, and production safety have not been established.
Do not use either model as a fact checker, safety oracle, or autonomous decision maker.

License and attribution

The released code and model artifacts use Apache-2.0, subject to upstream model and data terms. Qwen3.5 is attributed to Qwen. Transformers, PyTorch, Hugging Face Hub, WikiText, and SemEval remain the work of their respective authors. No WikiText or SemEval source records are redistributed.

The WikiText page currently has a license wording discrepancy: metadata lists CC BY-SA 3.0 and GFDL, while prose says CC BY-SA 4.0. Users should inspect Salesforce/wikitext directly.