SFT-trained variants of Qwen3-1.7B that produce structured <safety_check> XML analysis before reasoning and responding. Trained on the SupritiVijay/hbrt dataset (14,506 examples, 57% harmful / 43% benign).

The model performs a multi-stakeholder harm-benefit analysis in XML, computes a harm-benefit ratio, and decides whether to answer or refuse — all before entering <think> reasoning.

Example output flow:

<safety_check>
  <stakeholder>...</stakeholder>
  <harms>...</harms>
  <harm_score>0.3447</harm_score>
  <benefits>...</benefits>
  <benefit_score>0</benefit_score>
</safety_check>
<think>
  Based on the safety check, the harm–benefit ratio is 0.994, which exceeds the 0.4 threshold...
</think>
This query is harmful, so I must refuse...

What's in This Repo

This repo contains all models from a systematic 16-experiment study across 5 trials, exploring what matters most when fine-tuning small LLMs for structured safety output.

models/
├── baseline/          # Trial 1: LR sweep (5e-6, 2e-5, 5e-5, 1e-4)
├── xml_format/        # Trial 2: XML representation (verbose, inline, flat, json)
├── optimizer/         # Trial 3: Optimizer/scheduler (linear_decay, adafactor, constant_lr)
├── special_tokens/    # Trial 4: Special token count (0, 2, 12, 30)
└── combined/          # Trial 5: Combined best configuration

Each experiment directory contains both format variants (convergent_final_sft_format/ and final_sft_format/) with epoch-1/ and epoch-2/ checkpoints.

Recommended Model

Path: models/combined/experiment_best_combo/convergent_final_sft_format/epoch-2

Setting	Value
Base Model	Qwen3-1.7B
Format	Convergent (verbose XML)
Learning Rate	5e-5
Optimizer	AdamW + Linear Decay
Special Tokens	0 (none added)
Epochs	2
Effective Batch Size	128 (4 per-device × 8 GPUs × 4 grad accum)
Val Loss	0.1776
Val Accuracy	94.5%

Trial Results

Trial 1: Learning Rate Sweep

Finding: LR=5e-5 is optimal. Below = underfitting, above = marginal gains not worth instability.

LR	Val Loss (conv)	Val Loss (orig)	Val Acc (conv)
5e-6	0.2388	0.2541	93.0%
2e-5	0.1838	0.2001	94.3%
5e-5	0.1779	0.1933	94.4%
1e-4	0.1783	0.1940	94.5%

Trial 2: XML Format

Finding: Verbose XML wins by a landslide. Compressed formats save tokens but learn 2-3x worse. Token efficiency ≠ learnability — verbose XML provides structural redundancy that helps gradient signal.

Format	Val Loss (conv)	Val Loss (orig)	Val Acc (conv)	Train Time
Verbose	0.1780	0.1936	94.4%	52 min
Inline	0.4180	0.4403	87.3%	9 min
JSON	0.5060	0.5274	84.7%	8 min
Flat	0.5090	0.5330	84.6%	8 min

Trial 3: Optimizer / Scheduler

Finding: Barely matters. All within 0.003 val_loss once LR is in the right range. Linear decay wins marginally.

Config	Val Loss (conv)	Val Loss (orig)	Val Acc (conv)
Linear Decay	0.1775	0.1931	94.5%
Cosine (default)	0.1779	0.1933	94.4%
Constant + Warmup	0.1802	0.1953	94.3%
Adafactor	0.1804	0.1957	94.4%

Trial 4: Special Tokens

Finding: Don't add special tokens. Perfect monotonic degradation — more tokens = worse. Adding 30 special tokens replaces learned subword representations with random embeddings that 2 epochs cannot recover.

Tokens Added	Val Loss (conv)	Val Loss (orig)	Val Acc (conv)
0	0.1775	0.1931	94.5%
2	0.1778	0.1935	94.4%
12	0.1931	0.2101	94.0%
30	0.2590	0.2811	92.1%

Trial 5: Combined Best

Convergent format only, combining all winning settings. Confirms the performance ceiling.

Metric	Value
Val Loss	0.1776
Val Accuracy	94.5%

Convergent vs Original Format

Across all 16 experiments, the convergent format consistently outperforms original by ~0.016 val_loss. The convergent format uses > delimiters for hierarchy (e.g., Content Safety Risks > Hate/Toxicity > Bullying) while the original uses ## and # markers. Convergent is also ~10% shorter in token count.

Inference Behavior

Tested on 10 val examples (5 harmful + 5 benign) comparing the best fine-tuned model against base Qwen3-1.7B:

Metric	Fine-tuned	Base Qwen3-1.7B
Starts `<safety_check>`	10/10 (100%)	0/10
Completes full flow	5/10 (50%)	0/10
Generation time (10 ex)	270s	42s

The 50% completion rate is a known limitation of the 1.7B model size — the model sometimes gets stuck in repetitive XML generation loops instead of closing the safety check tag. The 5 that complete show correct harm classification and appropriate refusal/answering behavior.

Training Setup

Hardware: 8× NVIDIA H100 80GB HBM3 (FSDP)
Framework: TRL v0.29.1 + Transformers v4.57.6
Loss: assistant_only_loss=True (loss only on assistant tokens)
Sequence length: Up to 8K tokens, padded to multiple of 64
Dataset split: 90/10 train/val (13,055 / 1,451)

SupritiVijay/hbrt-safety-tuned-models-SFT

HBRT Safety-Tuned Models (SFT)