Scam Sentinel — Fine-tuned Gemma 4 E2B for Multimodal Scam Risk Detection

This is not a final forensic deepfake detector. It is a multimodal scam risk assistant that combines phone call transcript analysis, conversation patterns, and verification workflows.

Headline Results (300-sample real evaluation, apples-to-apples)

1. Same-size apples-to-apples (E2B base → E2B + QLoRA): F1 jumps +28.1 pt (58.0 → 86.1), FPR collapses 88× (97.7% → 1.1%), Precision more than doubles (41.4% → 98.0%).
2. Untuned Gemma 4 base is unusable for this task: both base models flag the vast majority of normal messages as suspicious (FPR 78.9% and 97.7%). The instruction-tuned base has no domain prior for scam vs. normal text.
3. Fine-tuning beats raw scale: the fine-tuned 5B model outperforms the larger 8B base by +22.7 F1 points (63.4 → 86.1).
4. Recall trade-off is intentional: 96.8% (E2B base) → 76.8% (fine-tuned). See "Design rationale" below — the production cascade's Stage 1 retains high recall.

Model Details

• Developed by: Alice0914 (Gemma 4 Good Hackathon submission)
• Base model: unsloth/gemma-4-E2B-it (~5B params, MatFormer architecture)
• Adapter type: LoRA (PEFT) — 28.7M trainable params (0.56% of base)
• Training framework: Unsloth + TRL SFTTrainer
• Quantization at training: 4-bit NF4 (QLoRA)
• License: Apache 2.0
• Language: English
• Project: Scam Sentinel GitHub repo

Intended Use

Direct Use

• risk_level: safe / low / medium / high / critical
• patterns: Detected scam patterns (urgency, impersonation, secrecy, etc.)
• user_message: Plain-language explanation answering "Is this a scam? Why? What to do? How to verify?"
• tool_calls: Function calls into 12 protective tools (notify family, suggest callback, block payment, etc.)

Out-of-Scope Use

• Voice authenticity / deepfake audio detection (use a dedicated audio model)
• Languages other than English
• Real-time telephony interception (requires phone-system integration)
• Replacement for human judgment in financial decisions

How to Use

Training Details

- Training Data

• 3,100 chat-formatted samples (system + user + assistant)
• Generated from 80 hand-written seeds + 571 real UCI SMS Spam samples × Gemma-4 paraphrased variants
• 8 categories: family_impersonation, prosecutor_scam, bec_scam, romance_scam, package_scam, bank_phishing, phishing_link, normal
• Assistant responses follow a 5-step Chain-of-Thought (IDENTIFY → ASSESS → EXPLAIN → DECIDE TOOLS → ANSWER) + JSON output format
• Train / dev split: 3,100 / 771 (stratified by category)

- Training Hyperparameters

• Method: QLoRA (4-bit NF4 base + LoRA r=16)
• LoRA: r=16, alpha=32, dropout=0.05, target_modules="all-linear"
• Batch: 1 × grad_accum 8 (effective batch 8)
• Epochs: 2 (~775 steps)
• Learning rate: 2e-4, cosine schedule, warmup_ratio=0.03
• Optimizer: paged_adamw_8bit
• Precision: bf16 (compute) / NF4 (base weights)
• Max sequence length: 1024
• Random seed: 3407

- Hardware

• GPU: Google Colab Pro L4 (22.5 GB VRAM)
• Training time: ~50 minutes for 2 epochs
• Framework versions: PEFT 0.19.1, transformers ≥4.50, trl ≥1.4, Unsloth (latest from GitHub)

Evaluation

- Testing Data

• Held-out set of 300 hand-labeled real samples
• Distribution: 175 safe / 7 low / 79 medium / 26 high / 13 critical
• Sources: FTC consumer-fraud reports, UCI SMS Spam Collection (training-disjoint subset), custom edge cases
• The evaluation set is disjoint from training via the seeds_real.jsonl filter — verified by hash check

- Metrics (this adapter)

• Strict 5-class match: 69.0% (model occasionally over-classifies within the dangerous range, e.g., medium → high — the correct failure mode for a safety-critical app)

- What Fine-tuning Changed (E2B base → E2B + QLoRA)

• The base instruction-tuned model has no in-domain prior for "what does a normal message look like?" — it flags 97.7% of safe messages as suspicious
• Fine-tuning re-calibrates the decision boundary using 3,100 in-domain examples
• The recall reduction is a deliberate trade-off favoring user trust over raw catch rate

- Fine-tuning vs Raw Scale (E4B base → E2B + QLoRA)

• A fine-tuned smaller model decisively outperforms a larger base model on this task
• Demonstrates that domain adaptation dominates scale for safety-critical classification with limited training compute
• Total cost: one Colab L4 session, ~50 minutes

Note on comparison fairness: All three setups use the same 300-sample test set and identical v3 system prompt; no RAG. Base models use Ollama Q4_K_M quantization; the fine-tune uses Unsloth NF4 (4-bit). Both are 4-bit; quantization differences contribute marginally — the +28.1 F1 / 88× FPR delta is dominated by the adapter, not quantization or size.

- Design Rationale: Precision over Recall

• Stage 1 — a fast classifier (e.g., gemma3:4b) ensures every potentially dangerous message is escalated (recall 99%+)
• Stage 2 — this fine-tuned adapter provides high-confidence reasoning and tool calls only when action is warranted (precision 98%)
• Stage 1 handles "catch everything"
• Stage 2's job is to justify action — blocking payments, alerting family, demanding callback verification
• With 1.1% FPR, when this model flags a message, downstream actions are trusted by users
• Higher recall at this stage would re-introduce the user-trust collapse seen in the base model (FPR 97.7%), making the product unusable in real deployment regardless of recall

Bias, Risks, and Limitations

- Language

• English-only: Trained on English text; performance on other languages is not validated

- Classification Behavior

• Over-classification bias within the dangerous range: The model leans toward "more dangerous" classifications (e.g., medium → high)
• This is intentional — once false positives on safe messages are eliminated, the safer error mode within non-safe messages is to over-classify
• Downstream tools (wait timer, callback verification) make over-classification cheap to recover from

- Recall Trade-off

• Some borderline messages may be missed
• Recommended deployment pairs this adapter with a high-recall first-pass classifier (cascade Stage 1)

- Training Data Provenance

• Synthetic origin: 80% of training data was Gemma-paraphrased from hand-written seeds and real UCI SMS spam
• Evaluation uses real held-out data only to detect any overfit to synthetic style

- Tool Calls Are Advisory

• The 12 protective tools are recommended actions
• Downstream systems must enforce safety policies independently
• The model does not execute actions — it returns structured intent

Citation