Fine-tuned Gemma-3-4b-it on a bilingual (English/Russian) scam-risk analysis
instruction dataset for local-first scam defense for non-technical people,
built for the AutoScientist Challenge — Personal-Finance category.
Model Details
Why Personal Finance
Scams are a direct, first-party financial-loss problem: wire fraud, gift-card
and crypto payments, and stolen banking credentials move money out of a
victim's account in minutes, and are rarely recoverable afterward. The
highest-risk targets — older adults, non-native speakers, and anyone
unfamiliar with a scam's exact playbook — are also the least equipped to
catch the manipulation in the moment. This model reads one suspicious message
and, before any money or credentials move, tells the person in plain language
whether it's dangerous, what the attacker is trying to extract, and the
single safest next step. That is a personal-finance protection tool in the
most literal sense: it exists to stop money from leaving a real person's
account into a scammer's.
Model Demo
Click to image below or press here - Demo Video.
Try the Model Demo to see the model analyze potential scam messages and explain how to avoid them.
Scambreaker-Gemma-3-4B is a LoRA fine-tuned version of Gemma-3-4b-it,
adapted to analyze a single suspicious message (SMS, email, DM) and return a
validated JSON safety card: a risk level, the scam type, the manipulation
tactics used, a compact "scam DNA" breakdown, the single safest next action, a
message to forward to a trusted person, and a short plain-language summary.
The base Gemma-3-4B model was chosen for its multilinguality and suitability
for a small fine-tuning dataset. No additional data was used during
fine-tuning; the goal was solely to teach the model the JSON output format
and the scam-analysis task itself, not to expand its knowledge.
Developed by: Sultanov Danial
Model type: Causal LM, LoRA fine-tuned (text-decoder only)
Dataset: dokster/adaption-scam-dataset-ru-en-v1
Language(s): English, Russian
License: Gemma Terms of Use
Finetuned from: google/gemma-3-4b-it
Training platform: Adaption Labs
Training Winrates
Uses
Direct Use
Given one suspicious message (English or Russian), it returns JSON scam-analysis:
scam_type: fixed-vocabulary label, e.g. credential_theft, job_scam, tech_support
tactics: up to 3 short manipulation-tactic strings
scam_dna: impersonates / pressure / ask / risk — short descriptive phrases, in the message's language
safest_action, trusted_person_message, summary: plain-language guidance, in the message's language
Intended as decision support for a non-technical person deciding whether
to click, reply, or call — not an autonomous blocker.
Out-of-Scope Use
Not legal, financial, or cybersecurity advice
Not a spam/scam filter for production email or SMS pipelines without
additional evaluation on real-world traffic (training data is synthetic —
see Bias, Risks, and Limitations)
Not evaluated for languages other than English and Russian
Should never be wired to auto-block, auto-delete, or auto-reply without a
human in the loop
Bias, Risks, and Limitations
Synthetic training data. Messages are template-generated, not collected
real-world scams; phrasing and scenario mix may not match real traffic.
US-centric scenarios, including in the Russian-language examples (USPS,
IRS, DMV, Zelle) — the dataset does not reflect scam patterns specific to
Russian-speaking regions.
Recommendations
Always present output as guidance to slow down and verify through an official
channel — never as a definitive verdict, and never surface an action that
tells the user to click the suspicious link or call the number in the message
itself.
How to Get Started with the Model
python
1from transformers import AutoModelForImageTextToText, AutoTokenizer
2from peft import PeftModel
3import torch
45base_model = AutoModelForImageTextToText.from_pretrained(6"google/gemma-3-4b-it",7 dtype=torch.bfloat16,8 device_map="auto",9)10model = PeftModel.from_pretrained(base_model,"dokster/gemma-3-4b-scambreaker-adapter")11tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-4b-it")1213SYSTEM_PROMPT ="..."# full JSON schema + safety-rules prompt (see repo)14message ="USPS: Your package is held. Verify account details now: https://... Ref 1000."1516prompt = tokenizer.apply_chat_template(17[{"role":"system","content": SYSTEM_PROMPT},18{"role":"user","content":f"/no_think\n\nAnalyze this message for scam risk:\n\n{message}\n\nReturn only the JSON object."}],19 tokenize=False, add_generation_prompt=True,20)21inputs = tokenizer(prompt, return_tensors="pt").to(model.device)22output = model.generate(**inputs, max_new_tokens=512, do_sample=False)23print(tokenizer.decode(output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
Note:google/gemma-3-4b-it is a gated model. Request access before loading
Training Details
Training Data
2,324 examples (1,200 English + 1,124 Russian), derived from the synthetic scam-message generator.
English examples are the deterministic generator output.
Russian examples were produced by translating only the natural-language fields (never the fixed risk_level/scam_type/tactics labels) with an LLM under strict brand-name and format-preservation rules,
then mapped back across the full row set. See dokster/adaption-scam-dataset-ru-en-v1 for full dataset documentation.
Training Procedure
Training Hyperparameters
Training regime: bf16 mixed precision
Algorithm: LoRA (Supervised Fine-Tuning)
LoRA rank: 16
LoRA alpha: 32
LoRA dropout: 0.0
Target modules:all-linear
Learning rate: 1e-5
LR scheduler: cosine, warmup ratio 0.05
Epochs: 2
Weight decay: 0.0
Gradient clipping: 1.0
Held-out validation: 5% random split, best checkpoint selected by eval loss
Evaluation
Testing Data, Factors & Metrics
Testing Data
This model achieves 94% winrate over base model. To approve this number I created special evaluation dataset.
Evaluated on a "hard" synthetic eval suite — 632 English cases
spanning boundary/calibration scenarios (credential theft, family
impersonation, tech support, job scams, marketplace fraud, "wrong number"
investment scams, and matched safe/benign controls), plus a 316 EN + 316 RU
bilingual subset for a per-language read. This is a held-out synthetic suite,
not real-world traffic — see Bias, Risks, and Limitations.
Metrics
risk_level accuracy: exact match against the gold risk label
dangerous → safe / needs_check: count of dangerous cases under-called —
the safety-critical failure mode
safe → over-flagged: count of benign messages incorrectly flagged, a
usability failure mode
Zero safety-critical misses on the full, reliable English suite — no
dangerous case was ever mislabeled safe or needs_check, and no benign
message was ever over-flagged.
Baseline Comparison: Fine-tuned vs. Base Model
To isolate what fine-tuning actually contributes, the base google/gemma-3-4b-it
model (zero-shot, no LoRA adapter, same system prompt/schema) was evaluated on
the identical full English hard suite (632 cases):
Metric
Base gemma-3-4b-it (no adapter)
Fine-tuned (this card)
risk_level accuracy
65.2% (412/632)
95.3% (602/632)
scam_type accuracy
2.8%
94.5%
mean tactic recall
23.1%
93.1%
dangerous → safe
0
0
dangerous → needs_check
0
0
safe → over-flagged
13
0
Invalid JSON / model errors
0 / 0
0 / 0
The base model already produces valid JSON and never misses a dangerous case
outright — the system prompt alone gets it that far. What fine-tuning adds is
accuracy and calibration: +30 points on risk classification, a ~34x jump
in scam_type accuracy, and elimination of false alarms on benign messages
(13 → 0). This is the clearest evidence that the LoRA training taught the
model the task itself, not just the output format.
Citation
bibtex
1@misc{scam_detection_ru_en_2026,
2 title = {Scam-Risk Analysis: A Bilingual Structured-Output Dataset for Scam Defense},
3 author = {Sultanov Danial},
4 year = {2026},
5 publisher = {HuggingFace},
6 howpublished = {\url{https://huggingface.co/dokster/gemma-3-4b-scambreaker-adapter}},
7 note = {AutoScientist Challenge 2026 — Personal-Finance Category}
8}
