Views
No views yet
| Feature | Module Profanity (CharCNN) | Module Toxic Detection (KcELECTRA) |
|---|---|---|
| Primary Goal | Detects explicit profanity, slang, and intentional typos (leetspeak). | Detects context-aware toxicity, hate speech, and offensive tones. |
| Architecture | Character-level CNN (CharCNN) | Transformer (Fine-tuned KcELECTRA-base) |
| Model Size | ~12MB (Ultra-lightweight) | ~400MB+ (Large) |
| Inference Speed | Very Fast (Real-time, CPU friendly) | Moderate (GPU recommended) |
| Input Method | Character/Jamo-level (Max len: 150) | WordPiece Tokenizer (Max len: 128) |
| Strengths | Highly robust against variations. | Detects subtle insults and toxicity without explicit swear words. |
module_profanity/1import torch
2from module_profanity.src.tokenizer import CharTokenizer
3from module_profanity.src.model import CharCNN
4
5# Load model and tokenizer
6device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
7tokenizer = CharTokenizer.load("./module_profanity/models/tokenizer.json")
8model = CharCNN(vocab_size=len(tokenizer), num_classes=2).to(device)
9model.load_state_dict(torch.load("./module_profanity/models/char_cnn_model.pth", map_location=device))
10model.eval()
11
12# Predict
13text = "게임 참 좆같이도 하시네요"
14encoded = tokenizer.encode(text)
15input_tensor = torch.tensor([encoded]).to(device)
16
17with torch.no_grad():
18 output = model(input_tensor)
19 prob = torch.nn.functional.softmax(output, dim=1)[0][1].item() # Class 1: Profanity
20
21print(f"Profanity Probability: {prob * 100:.2f}%")module_toxicdetection/beomi/KcELECTRA-base model, which is pre-trained on Korean online comments and colloquialisms. It excels at understanding the semantic meaning and nuance of a sentence. It can identify toxic behavior even when no explicit profanity is present, making it suitable for community moderation and safety filtering.beomi/KcELECTRA-base1import torch
2from transformers import AutoTokenizer, AutoModelForSequenceClassification
3
4# Load model from checkpoint
5model_path = "./module_toxicdetection/models/final_model"
6tokenizer = AutoTokenizer.from_pretrained(model_path)
7model = AutoModelForSequenceClassification.from_pretrained(model_path)
8model.eval()
9
10# Predict
11text = "글을 참 개같이도 잘쓰시네요 ㅎㅎ" # Toxic tone without explicit swear words
12inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
13
14with torch.no_grad():
15 outputs = model(**inputs)
16 probs = torch.nn.functional.softmax(outputs.logits, dim=-1)
17 toxic_score = probs[0][1].item()
18
19print(f"Toxic Score: {toxic_score:.4f}").
├── module_profanity/ # CharCNN-based Profanity Detection
│ ├── models/ # Trained weights (.pth) and tokenizer.json
│ ├── src/ # Source code (model definition, tokenizer, etc.)
│ ├── train.py # Training script
│ └── predict.py # Inference test script
│
├── module_toxicdetection/ # KcELECTRA-based Toxicity Detection
│ ├── models/ # HuggingFace model directory
│ ├── src/ # Dataset loaders and utility scripts
│ ├── train.py # Training script using Trainer API
│ └── src/predict.py # Inference test script
│
└── README_NEW.md # Documentation (this file)