You can deploy this model and start earning money today!

Disha-Singh-Singraur/
rulebreaker-training

Name: rulebreaker-training
Availability: InStock

Views

No views yet

🏛️ Shadow Docket — RuleBreaker Tax Exploitation Simulator

A multi-agent reinforcement learning environment where an AI Lawyer exploits tax loopholes while an AI Overseer dynamically patches them.

🎯 Motivation

Tax systems are adversarial by nature: one side optimizes to minimize payment, the other adapts rules to prevent exploitation. RuleBreaker simulates this dynamic as a multi-agent RL problem where two LLMs compete in a turn-based financial environment.

This project explores:

Multi-Agent Interactions — Two agents with opposing objectives must develop theory-of-mind about each other's strategies.
World Modeling — The environment simulates a realistic financial system with accounts, transfers, tax thresholds, aggregation windows, and solvency constraints.
Reward Hacking Prevention — Multiple independent reward signals, strict validation, and pre-transfer solvency checks make it hard for the LLM to find cheap shortcuts.

🏗️ Architecture

Shadow Docket/
├── models.py                    # Pydantic data models (Action, Observation)
├── prompts.py                   # LLM prompt templates for Lawyer & Overseer
├── train.py                     # TRL/Unsloth GRPO training script
├── client.py                    # OpenEnv WebSocket client
├── __init__.py                  # Package exports
├── README.md
├── server/
│   ├── __init__.py
│   ├── app.py                   # FastAPI server (OpenEnv HTTP interface)
│   └── MyEnv_environment.py     # Core environment logic
└── training_logs/               # Generated during training
    ├── baselines.json
    └── training_prompts.json

🎮 How It Works

The Game

Two agents alternate turns over a 10-turn episode:

Turn	Agent	Goal
Odd	Lawyer	Move money between accounts while minimizing tax
Even	Overseer	Update tax rules to close the Lawyer's loopholes

If the Lawyer achieves an exploit rate > 30%, an emergency override forces the Overseer to act immediately.

Lawyer Actions

Action	Description
`transfer`	Move money between two accounts
`split_transfer`	Split money across multiple destinations (primary exploit strategy)
`create_account`	Open a shell account (max 10)
`wait`	Do nothing

Overseer Controls

Parameter	Default	Range	Effect
`tax_threshold`	$10,000	$1K–$50K	Transfers above this are taxed
`tax_rate`	20%	1%–99%	Tax percentage
`aggregation_window_hours`	0	0–168h	Lookback window for aggregating transactions
`source_aggregation`	Off	bool	Aggregate all transfers from same source

Reward Signals

format_reward: +1.0 for valid <reasoning> + <action> XML tags with parseable JSON.
lawyer_reward: Proportional to tax_avoided / baseline_tax. Invalid actions get −1.0.
overseer_reward: Based on how much the exploit rate decreased. Bonus if exploit rate drops below 10%.

🛡️ Anti-Hacking Measures

Protection	Description
Pre-transfer solvency check	`amount + expected_tax` must not exceed balance
Account cap	Maximum 10 accounts to prevent state explosion
Rule clamping	Overseer values are bounded and auto-clamped
Format enforcement	Invalid XML/JSON → instant −1.0 penalty
Single tax source of truth	`compute_tax()` is the only function that calculates tax

🚀 Quick Start

Install Dependencies

pip install openenv-core[core] pydantic uvicorn

Run the Environment Server

python -m server.app

Run Baseline Evaluation Only

python train.py --eval-only

Run Full RL Training

bash

# Install training dependencies
pip install trl unsloth datasets

# Start training
python train.py --model unsloth/Llama-3.2-1B-Instruct --max-steps 500

📊 Baseline Results

Two baseline agents are included for comparison:

Agent	Strategy	Avg Exploit Rate	Avg Reward
`RandomLawyerAgent`	Random valid actions	~0.15	~0.05
`GreedyLawyerAgent`	Always splits below threshold	~0.85	~0.80

The RL-trained agent should discover strategies that approach or exceed the GreedyLawyerAgent's performance.

📁 Key Files Explained

`server/MyEnv_environment.py`

The core simulation. Contains:

compute_tax() — Single source of truth for all tax calculations
RuleBreakerEnvironment — The OpenEnv-compatible environment class
RandomLawyerAgent / GreedyLawyerAgent — Baseline agents

`prompts.py`

Converts raw observation dicts into structured chat prompts that teach the LLM the rules and required output format.

`train.py`

End-to-end GRPO training script:

Evaluates baselines (Random vs. Greedy)
Generates training prompts from environment episodes
Trains the LLM with TRL's GRPOTrainer
Saves the fine-tuned model

🔗 Links

Environment (HF Spaces): Coming soon
Training Notebook (Colab): Coming soon
Demo Video: Coming soon

📄 License

BSD License. See LICENSE for details.

Disha-Singh-Singraur/rulebreaker-training