Japanese Pretraining Data Filter Classifier
This repository contains a 4-class Japanese document quality classifier for filtering pretraining data for a Japanese small language model focused on business decision support, keyword/tag extraction, causal reasoning, scenario analysis, and insight generation.
The classifier was trained from LLM-filtered Common Crawl Japanese samples. The main supervision field is filter_result.d.
Label Mapping
Original LLM labels were mapped into 4 classifier classes:
Class ID Class Source filter_result.d 0 rejectX, R1 low_valueKL, D2 keepK3 high_valueKH
Original label meaning:
dMeaning KHkeep high value Kkeep KLkeep low weight Ddeduplicate Rneeds review Xreject
The LLM field w was used as sample weight. Because X and R examples have w=0.0, training used sample_weight = max(w, 0.2) so the reject class could still be learned.
Input Construction
Each training example is built from:
URL, if available
script statistics, if available: kana_ratio, cjk_ratio, latin_ratio, chars
document text
For long documents, the text is truncated as:
first 10,000 characters
[TAIL]last 2,000 characters
The resulting input is tokenized with max_length=2048.
Example structure:
1 [URL]
2 https://example.jp/article
3 [SCRIPT_STATS]
4 kana_ratio=... cjk_ratio=... latin_ratio=... chars=...
5 [TEXT]
6 ...
7 [TAIL]
8 ...
Data
Source paths used locally:
LLM-filtered raw labeled data: /mnt/data/commoncrawl_ja_2025_present/data/filtered_random_10k
Raw sample file: /mnt/data/commoncrawl_ja_2025_present/data/ja_sample.jsonl
Prepared classifier split:
Split Rows Train 178,986 Test 19,888 Total 198,874
Split method:
stratified train/test split
test size: 0.1
seed: 20260604
Original filter_result.d counts:
Label Count D137,652 X30,408 KL17,014 K12,082 R1,374 KH344
Mapped 4-class counts:
Class Total Train Test reject31,782 28,604 3,178 low_value154,666 139,199 15,467 keep12,082 10,874 1,208 high_value344 309 35
Model
Base model:
sbintuitions/modernbert-ja-130mReason for choosing this model:
Japanese-focused encoder model
efficient 130M parameter size
supports long-context classification better than typical BERT-style 512-token encoders
tokenizer/model loaded cleanly in the local training environment
The trained model is stored in:
outputs_4class_modernbert/full_best/best_model/Final training checkpoints are also included:
1 outputs_4class_modernbert/full_best/checkpoint-11000/
2 outputs_4class_modernbert/full_best/checkpoint-11188/
The best checkpoint selected by macro_f1 was checkpoint-11000; best_model/ contains the loadable best model and tokenizer.
Hyperparameter Sweep
Sweep was run on a subset only:
train subset: 60,000 samples
eval subset: 12,000 samples
epochs: 1
Run Max Length LR Batch/GPU Macro F1 Weighted F1 Accuracy Reject F1 Keep+ F1 sweep_len2048_lr2e-52048 2e-516 0.5995 0.8395 0.8322 0.6796 0.6519 sweep_len1024_lr2e-51024 2e-532 0.4925 0.7593 0.7383 0.5853 0.5895 sweep_len4096_lr1e-54096 1e-58 0.4314 0.7559 0.7705 0.3577 0.5249
The best sweep setting was max_length=2048, learning_rate=2e-5.
Final Training Configuration
Final training used the full train split, not the 60k sweep subset.
1 CUDA_VISIBLE_DEVICES = 0,1 TOKENIZERS_PARALLELISM = false \
2 torchrun --standalone --nproc_per_node = 2 \
3 code/train_4class_classifier.py \
4 --model-name sbintuitions/modernbert-ja-130m \
5 --data-dir data_4class \
6 --output-dir outputs_4class_modernbert/full_best \
7 --max-length 2048 \
8 --learning-rate 2e-5 \
9 --weight-decay 0.01 \
10 --epochs 2 \
11 --batch-size 16 \
12 --gradient-accumulation-steps 1 \
13 --warmup-ratio 0.06 \
14 --class-weight sqrt_balanced \
15 --eval-steps 1000 \
16 --save-steps 1000 \
17 --num-proc 8
Important settings:
Parameter Value GPUs 2, GPU 0 and 1 Max length 2048 Epochs 2 Per-device train batch size 16 Effective train batch size 32 Learning rate 2e-5Weight decay 0.01Warmup ratio 0.06Class weighting sqrt_balancedBF16 enabled Seed 20260604
Class weights:
Class Weight reject0.3150 low_value0.1428 keep0.5110 high_value3.0312
Final Results
Final evaluation was run on the held-out test split of 19,888 examples.
Metric Value Accuracy 0.8702 Macro F1 0.7050 Weighted F1 0.8736 Reject F1 0.7355 Keep+ F1 0.7181 Eval loss 0.1485 Train runtime 3,112.9 sec Train samples/sec 115.0
Per-class metrics:
Class Precision Recall F1 Support reject0.7099 0.7631 0.7355 3,178 low_value0.9372 0.8959 0.9161 15,467 keep0.6074 0.8377 0.7042 1,208 high_value0.6190 0.3714 0.4643 35
Confusion matrix, rows=true labels and columns=predicted labels in order [reject, low_value, keep, high_value]:
1 [
2 [2425, 739, 14, 0],
3 [ 989, 13857, 620, 1],
4 [ 2, 187, 1012, 7],
5 [ 0, 2, 20, 13]
6 ]
Notes:
low_value is the strongest class because it dominates the dataset.keep has high recall but moderate precision, which is acceptable for permissive pretraining filtering.high_value is weak and unstable because the test split has only 35 examples.For production use, treat high_value as a useful ranking signal rather than a fully reliable hard class until more KH examples are labeled.
Reproducing the Data Split
1 python code/prepare_4class_dataset.py \
2 --input-dir raw/filtered_random_10k \
3 --output-dir data_4class \
4 --test-size 0.1 \
5 --seed 20260604 \
6 --head-chars 10000 \
7 --tail-chars 2000 \
8 --weight-floor 0.2
Inference Example
1 from transformers import AutoModelForSequenceClassification , AutoTokenizer
2 import torch
3
4 model_dir = "model"
5 tokenizer = AutoTokenizer . from_pretrained ( model_dir )
6 model = AutoModelForSequenceClassification . from_pretrained ( model_dir )
7 model . eval ( )
8
9 text = "[TEXT]\n日本語の文書本文..."
10 inputs = tokenizer ( text , return_tensors = "pt" , truncation = True , max_length = 2048 )
11
12 with torch . no_grad ( ) :
13 logits = model ( ** inputs ) . logits
14 pred_id = int ( logits . argmax ( dim = - 1 ) )
15
16 print ( model . config . id2label [ pred_id ] )
Repository Layout
1 README.md
2 prepare_4class_dataset.py
3 train_4class_classifier.py
4 run_modernbert_4class_sweep.sh
5 data_4class/
6 outputs_4class_modernbert/
7 raw/
outputs_4class_modernbert/full_best/best_model/: best finetuned classifieroutputs_4class_modernbert/full_best/checkpoint-*: final training checkpointsreports/sweep_*: sweep final reports and logs, without sweep checkpointsprepare_4class_dataset.py, train_4class_classifier.py: dataset preparation and training scriptsdata_4class/: prepared train/test JSONL splitraw/filtered_random_10k/: LLM-filtered raw labeled data used to create the splitraw/ja_sample.jsonl: requested raw sample fileoutputs_4class_modernbert/full_best/final_report.json: final training reportreports/sweep_summary.json: sweep summary 
