decensor-env
Unified RL reward environment for decensoring/anti-refusal training.
Expects datasets with a "prompt" column (plain string).
All judges are toggleable via TOML args.
Reward formula:
final = base_reward * word_mult * coherency_mult * md_presence_mult * md_correctness_mult * reasoning_mult * slop_mult
Where base_reward = 1.0 - (refusal_score / 10.0)
Feature toggles (all in [env.args]):
enable_system_prompts - prepend style/length system prompts (default: true)
enable_adherence - adherence judge gate (default: true)
enable_word_count - word count multiplier (default: true)
enable_structured_markers - XML/bracket/YAML gate (default: true)
enable_markdown_judges - markdown presence + correctness judges (default: true)
enable_reasoning_trace - penalize ethics-based refusal in (default: true)
enable_slop_guard - slop-guard heuristic multiplier (default: false)
markdown_exempt_probability - % of rollouts exempt from markdown penalty (default: 0.25)
Judges:
refusal - rates refusal severity 1-9, always on
coherency - coherent + complete + no roleplay narration, always on
adherence - style prompt adherence, early exit gate
markdown - presence (no unnecessary md) + correctness (right list types)
reasoning - checks trace for ethics-based refusal reasoning
slop-guard - local heuristic, no LLM call, scores 0-100
Example TOML:
[[env]]
id = "mangymango/decensor-env"
[env.args]
dataset_names = ["NewEden/RL-Seed-Mix-Iter-3"]
dataset_ratios = [1.0]
num_train_examples = 19000
judge_model = "Qwen/Qwen3-VL-32B-Instruct-FP8"
judge_base_url = "
http://72.46.85.157:31974/v1"
enable_system_prompts = false
enable_adherence = false
enable_word_count = false
enable_slop_guard = true
