R-ViHSD — ViSoBERT + TF-IDF/LinearSVM Stacking
A Vietnamese hate-speech and text-noise classification pipeline combining:
ViSoBERT for semantic representation.TF-IDF character + word n-grams for robust lexical features.LinearSVC models for hate-speech and noise classification.Logistic Regression stacking for the final hate-speech prediction.
The pipeline is trained using Out-of-Fold (OOF) stacking to reduce data leakage. After OOF training, the final base models are retrained on the full labeled dataset for inference on new data.
1. Task Definition
For each Vietnamese text sample, the system predicts two outputs.
Hate-Speech Label
Noise Type
1 ORIGINAL
2 NO_DIACRITICS
3 TEENCODE
4 CHAR_REPEAT
5 PUNCT_NOISE
6 OBFUSCATION
7 MIXED
The minimum inference input format is:
1 id , text
2 0001 , "sample text"
3 0002 , "another sample"
The id column is preserved in the output but is not used as a model feature.
2. Overall Architecture
1 ┌─────────────────────────┐
2 │ TEXT │
3 └────────────┬────────────┘
4 │
5 ┌──────────────────────┼──────────────────────┐
6 │ │ │
7 ▼ ▼ ▼
8 ┌────────────────┐ ┌──────────────────┐ ┌─────────────────┐
9 │ ViSoBERT │ │ TF-IDF char+word│ │ Surface Features│
10 └───────┬────────┘ └────────┬─────────┘ └────────┬────────┘
11 │ │ │
12 3 class probabilities ┌─────┴─────┐ │
13 │ │ │
14 ▼ ▼ │
15 Hate LinearSVC Noise LinearSVC │
16 │ │ │
17 3 decision 7 decision │
18 scores scores │
19 │ │ │
20 │ softmax │
21 │ │ │
22 └─────┬─────┘ │
23 │ │
24 ┌──────────────────────┴──────────────────────┘
25 │
26 ▼
27 ┌──────────────────────┐
28 │ 18 Stacking Features │
29 └──────────┬───────────┘
30 │
31 ▼
32 ┌──────────────────────┐
33 │ Logistic Regression │
34 │ Meta-Model │
35 └──────────┬───────────┘
36 │
37 ▼
38 CLEAN / OFFENSIVE / HATE
The final noise_type prediction does not pass through the meta-model. It is obtained directly from:
TF-IDF → Noise LinearSVC → argmax3. Saved Model Structure
After training, the main working directory is expected to contain:
1 rvihsd_stacking_work/
2 │
3 ├── visobert_full/
4 │ ├── config.json
5 │ ├── model.safetensors / pytorch_model.bin
6 │ ├── tokenizer_config.json
7 │ ├── tokenizer.json
8 │ └── ...
9 │
10 ├── full_vectorizer.joblib
11 ├── full_hate_svm.joblib
12 ├── full_noise_svm.joblib
13 ├── meta_model.joblib
14 │
15 ├── visobert_oof.npy
16 ├── svm_oof.npy
17 ├── noise_oof_scores.npy
18 ├── visobert_fold*_pred.npy
19 │
20 └── submissions/
Required Files for Inference
Only the following files are required for inference:
1 visobert_full/
2 full_vectorizer.joblib
3 full_hate_svm.joblib
4 full_noise_svm.joblib
5 meta_model.joblib
OOF files are used during training and evaluation but are not required for deployment.
4. ViSoBERT Model
Backbone:
ViSoBERT is fine-tuned for three-class hate-speech classification:
1 0 → CLEAN
2 1 → OFFENSIVE
3 2 → HATE
Main training configuration:
Parameter Value Backbone uitnlp/visobertMax sequence length 128Epochs 3Learning rate 2e-5Weight decay 0.01Warmup ratio 0.08Label smoothing 0.05Train batch size 128Evaluation batch size 64Gradient accumulation 2Seed 42
During training, ViSoBERT uses weighted cross-entropy with sqrt-balanced class weights to reduce the effect of class imbalance.
The model produces three probabilities:
1 P(CLEAN)
2 P(OFFENSIVE)
3 P(HATE)
These three values are passed to the stacking meta-model.
5. TF-IDF Features
The pipeline combines two TF-IDF vectorizers.
Character TF-IDF
1 analyzer = "char"
2 ngram_range = ( 3 , 5 )
3 max_features = 150_000
4 min_df = 2
5 sublinear_tf = True
Word TF-IDF
1 analyzer = "word"
2 ngram_range = ( 1 , 2 )
3 max_features = 80_000
4 min_df = 2
5 sublinear_tf = True
The two sparse matrices are concatenated:
1 Character TF-IDF + Word TF-IDF
2 │
3 ▼
4 Sparse Feature Matrix
Character n-grams are particularly useful for noisy social-media text, including:
teencode,
misspellings,
missing Vietnamese diacritics,
repeated characters,
obfuscation,
punctuation noise.
6. LinearSVC Models
Two independent LinearSVC models are used.
Hate-Speech SVM
Classes:
Configuration:
1 LinearSVC (
2 C = 2.0 ,
3 class_weight = "balanced" ,
4 random_state = 42
5 )
The stacking model uses the three decision_function scores, not calibrated probabilities.
Noise SVM
Classes:
1 ORIGINAL
2 NO_DIACRITICS
3 TEENCODE
4 CHAR_REPEAT
5 PUNCT_NOISE
6 OBFUSCATION
7 MIXED
Configuration:
1 LinearSVC (
2 C = 2.0 ,
3 class_weight = "balanced" ,
4 random_state = 42
5 )
For stacking, the seven decision scores are transformed using:
softmax(noise_score, axis=1)These values are only used as stacking features and should not be interpreted as calibrated probabilities.
7. Surface Features
The pipeline also extracts five handcrafted features:
1 1. log(1 + text length)
2 2. punctuation ratio
3 3. Vietnamese-diacritic ratio
4 4. repeated-character score
5 5. obfuscation score
The final meta-model input consists of:
1 ViSoBERT probabilities 3
2 Hate SVM decision scores 3
3 Noise SVM softmax scores 7
4 Surface features 5
5 -------------------------------
6 Total 18
The feature order must remain exactly the same during inference.
8. Meta-Model
The final hate-speech meta-model is:
1 StandardScaler
2 ↓
3 LogisticRegression
Configuration:
1 Pipeline ( [
2 ( "scale" , StandardScaler ( ) ) ,
3 ( "lr" , LogisticRegression (
4 C = 1.0 ,
5 max_iter = 3000 ,
6 class_weight = "balanced" ,
7 random_state = 42
8 ) )
9 ] )
Input:
Output:
The model is saved as:
9. Out-of-Fold Stacking
The meta-model should not be trained on predictions produced by base models that have already seen the same samples.
The training pipeline therefore uses:
1 StratifiedGroupKFold (
2 n_splits = 5 ,
3 shuffle = True ,
4 random_state = 42
5 )
Workflow:
1 Fold 1 → train base models on other folds → predict Fold 1
2 Fold 2 → train base models on other folds → predict Fold 2
3 ...
4 Fold 5 → train base models on other folds → predict Fold 5
These predictions form the Out-of-Fold feature matrix.
The meta-model is then trained on:
1 ViSoBERT OOF predictions
2 +
3 Hate SVM OOF scores
4 +
5 Noise SVM OOF scores
6 +
7 Surface features
8 ↓
9 Logistic Regression Meta-Model
Grouping based on normalized text can also be used to reduce duplicate or augmentation leakage across folds.
10. Final Training
After OOF features are generated and the meta-model is trained:
The TF-IDF vectorizer is fitted again on all labeled data.
The hate-speech LinearSVC is trained on all labeled data.
The noise LinearSVC is trained on all labeled data.
ViSoBERT is fine-tuned on all labeled data.
All final models are saved for inference.
If:
USE_VALIDATION_FOR_FINAL_TRAIN = Truethe final models use:
training_set + validation_set11. Environment Requirements
Recommended installation:
1 pip install -U \
2 "transformers>=4.46" \
3 "accelerate>=1.0" \
4 "scikit-learn>=1.4" \
5 sentencepiece \
6 joblib \
7 scipy \
8 pandas \
9 numpy \
10 torch
Main dependencies:
1 Python
2 PyTorch
3 Transformers
4 scikit-learn
5 SciPy
6 NumPy
7 Pandas
8 Joblib
9 SentencePiece
CUDA GPU support is recommended for ViSoBERT inference but is not required.
12. Loading the Models
Example:
1 from pathlib import Path
2 import joblib
3
4 from transformers import (
5 AutoTokenizer ,
6 AutoModelForSequenceClassification ,
7 )
8
9 WORK_DIR = Path (
10 "/content/drive/MyDrive/rvihsd_stacking_work"
11 )
12
13 tokenizer = AutoTokenizer . from_pretrained (
14 WORK_DIR / "visobert_full"
15 )
16
17 visobert = AutoModelForSequenceClassification . from_pretrained (
18 WORK_DIR / "visobert_full"
19 )
20
21 vectorizer = joblib . load (
22 WORK_DIR / "full_vectorizer.joblib"
23 )
24
25 hate_svm = joblib . load (
26 WORK_DIR / "full_hate_svm.joblib"
27 )
28
29 noise_svm = joblib . load (
30 WORK_DIR / "full_noise_svm.joblib"
31 )
32
33 meta_model = joblib . load (
34 WORK_DIR / "meta_model.joblib"
35 )
full_vectorizer.joblib contains a custom DualTfidf class. The inference environment must define a compatible DualTfidf class before loading the file with joblib.
13. Inference Flow
For each new text sample:
1 text
2 │
3 ├── ViSoBERT
4 │ └── 3 class probabilities
5 │
6 ├── TF-IDF
7 │ ├── Hate LinearSVC
8 │ │ └── 3 decision scores
9 │ │
10 │ └── Noise LinearSVC
11 │ ├── 7 decision scores
12 │ └── softmax → 7 stacking features
13 │
14 └── Surface features
15 └── 5 features
The final stacking input is:
1 meta_X = np . hstack ( [
2 visobert_prob , # 3
3 hate_svm_score , # 3
4 noise_soft , # 7
5 surface_features , # 5
6 ] )
The feature dimension must satisfy:
assert meta_X.shape[1] == 18Final hate-speech prediction:
hate_pred = meta_model.predict(meta_X)Final noise prediction:
noise_pred = noise_score.argmax(axis=1)14. Input Format
A new CSV file should contain at least:
1 id , text
2 1 , "first sentence"
3 2 , "second sentence"
4 3 , "third sentence"
Additional columns may exist, but inference should only depend on:
This prevents accidental use of labels or unrelated metadata.
15. Output Format
Recommended output:
1 id , pred_label , pred_noise_type
2 1 , CLEAN , ORIGINAL
3 2 , OFFENSIVE , TEENCODE
4 3 , HATE , NO_DIACRITICS
Columns:
Column Description idOriginal sample ID pred_labelCLEAN, OFFENSIVE, or HATEpred_noise_typeOne of the seven supported noise classes
16. Label Mapping
Hate-Speech Labels
1 LABELS = [
2 "CLEAN" ,
3 "OFFENSIVE" ,
4 "HATE" ,
5 ]
Mapping:
1 0 → CLEAN
2 1 → OFFENSIVE
3 2 → HATE
Noise Labels
1 NOISE_LABELS = [
2 "ORIGINAL" ,
3 "NO_DIACRITICS" ,
4 "TEENCODE" ,
5 "CHAR_REPEAT" ,
6 "PUNCT_NOISE" ,
7 "OBFUSCATION" ,
8 "MIXED" ,
9 ]
Mapping:
1 0 → ORIGINAL
2 1 → NO_DIACRITICS
3 2 → TEENCODE
4 3 → CHAR_REPEAT
5 4 → PUNCT_NOISE
6 5 → OBFUSCATION
7 6 → MIXED
Do not change the class order when using the already-trained models.
17. Recommended Inference Checks
Useful safety checks:
1 assert len ( output ) == len ( test_df )
2 assert output [ "id" ] . is_unique
3
4 assert set (
5 output [ "pred_label" ]
6 ) . issubset ( LABELS )
7
8 assert set (
9 output [ "pred_noise_type" ]
10 ) . issubset ( NOISE_LABELS )
11
12 assert meta_X . shape [ 1 ] == 18
If the stacking matrix does not contain exactly 18 features, the inference feature construction no longer matches training.
18. Components That Must Stay Consistent
When using the existing trained models, keep the following unchanged:
hate-speech label order,
noise label order,
MAX_LENGTH = 128,tokenizer saved in visobert_full,
meta_surface_features implementation,DualTfidf implementation,18-feature stacking order,
noise-score softmax transformation before the meta-model.
The stacking order must remain:
1 [ViSoBERT: 3]
2 +
3 [Hate SVM: 3]
4 +
5 [Noise SVM: 7]
6 +
7 [Surface Features: 5]
19. Do New Test Sets Require Retraining?
No.
If the following trained artifacts are available:
1 visobert_full/
2 full_vectorizer.joblib
3 full_hate_svm.joblib
4 full_noise_svm.joblib
5 meta_model.joblib
a new dataset only requires:
1 LOAD MODELS
2 ↓
3 LOAD NEW CSV
4 ↓
5 TF-IDF + SVM INFERENCE
6 ↓
7 ViSoBERT INFERENCE
8 ↓
9 SURFACE FEATURE EXTRACTION
10 ↓
11 STACKING
12 ↓
13 SAVE PREDICTIONS
There is no need to rerun:
1 5-fold OOF training
2 TF-IDF fitting
3 SVM training
4 ViSoBERT fine-tuning
5 Meta-model training
20. Training Cache
The notebook may use:
Typical cache files include:
1 svm_oof.npy
2 noise_oof_scores.npy
3 visobert_oof.npy
4 visobert_fold*_pred.npy
These files are useful for resuming training or reusing OOF predictions.
They are not required for deployment.
21. Main Training Hyperparameters
1 N_FOLDS = 5
2 SEED = 42
3
4 MODEL_NAME = "uitnlp/visobert"
5 MAX_LENGTH = 128
6 EPOCHS = 3
7
8 TRAIN_BATCH_SIZE = 128
9 EVAL_BATCH_SIZE = 64
10 GRAD_ACCUM_STEPS = 2
11
12 LEARNING_RATE = 2e-5
13 WEIGHT_DECAY = 0.01
14 WARMUP_RATIO = 0.08
15 LABEL_SMOOTHING = 0.05
16
17 CHAR_NGRAM = ( 3 , 5 )
18 WORD_NGRAM = ( 1 , 2 )
19
20 CHAR_MAX_FEATURES = 150_000
21 WORD_MAX_FEATURES = 80_000
22
23 MIN_DF = 2
24
25 SVM_C_HATE = 2.0
26 SVM_C_NOISE = 2.0
27
28 META_C = 1.0
22. Evaluation Metric
Both tasks are evaluated using Macro-F1.
Hate-speech classification:
1 f1_score (
2 y_hate ,
3 hate_pred ,
4 average = "macro"
5 )
Noise classification:
1 f1_score (
2 y_noise ,
3 noise_pred ,
4 average = "macro"
5 )
The notebook may also compute a combined score:
1 0.85 × Hate Macro-F1
2 +
3 0.15 × Noise Macro-F1
This README intentionally does not report a fixed F1 score because the actual metric depends on the specific training run and cached predictions.
23. Minimal Deployment Package
For deployment on another machine, the project can be organized as:
1 model/
2 ├── visobert_full/
3 ├── full_vectorizer.joblib
4 ├── full_hate_svm.joblib
5 ├── full_noise_svm.joblib
6 ├── meta_model.joblib
7 ├── inference.py
8 └── README.md
A command-line inference interface may look like:
1 python inference.py \
2 --input new_test.csv \
3 --output predictions.csv \
4 --model-dir model
24. Notes
The meta-model predicts only the hate-speech label.
The noise label is produced directly by the noise LinearSVC.
LinearSVC.decision_function() values are not probabilities.The softmax applied to noise scores is used as a stacking transformation rather than probability calibration.
Model compatibility depends on preserving the preprocessing and feature-ordering logic used during training.
When transferring joblib artifacts between environments, compatible versions of Python and scikit-learn are recommended.
25. Summary
The final inference system is:
1 ViSoBERT
2 +
3 TF-IDF Character/Word Features
4 +
5 Hate LinearSVC
6 +
7 Noise LinearSVC
8 +
9 Surface Features
10 ↓
11 Logistic Regression Stacking
12 ↓
13 Final Hate-Speech Prediction
14
15 Noise LinearSVC
16 ↓
17 Final Noise-Type Prediction
This architecture combines transformer-based semantic information with sparse lexical features that are robust to noisy Vietnamese social-media text.
